Sunday, December 30, 2012

go(ing) under

Lots of people like go and this year I began to find it unavoidable - I'm even ending the year with go being the main language that helps pay my bills!

One of the things that go lets you do is pass functions around.
I started the year talking about under from the j language. Which lets you 'bookend' some function with another function (a verb) and a function which undoes that function (obverse).
A great example is Pythagoras' theorem:
  1. You perform some 'pre-processing' on each value (squaring)
  2. You add all the values up
  3. You square root the result
I started the year talking about under in clojure. Here it is implemented in go. Complete with functions being passed around, closures, and functions returning functions. Happy new year!

Monday, October 22, 2012

Running juju locally on 12.04 (or getting over agent-status pending)

I've just tried following the instructions here to get juju running locally on my brand new Ubuntu 12.04 (desktop) system.
No matter what I tried my wordpress and mysql charms seemed stuck in agent-status pending and ip address null. Even waiting a couple of hours.

I needed to take to a couple of extra steps to get it working that didn't seem to be documented in one place. So I thought I'd put them both here.

All it took was to add a rule to ufw:
sudo ufw allow from to any
Hope this helps someone who was stuck in the same way I was.

Saturday, September 15, 2012

Clojure, Vim and the delay sending to screen

So, long story short: I'm use vim for writing clojure (and everything else for that matter)

If you're following the setup How I develop Clojure with Vim on the :wq blog then you might encounter a problem where it takes a few seconds for code you send from vim to appear in your screen session. Here's the solution.

Add the following lines to you ~/.screenrc

msgwait 0
msgminwait 0

And there you have it, instant code from vim to your REPL!

Tuesday, August 14, 2012

Another clojure macro tutorial (that no one should follow)

Disclaimer: This post shows you something you can do with macros. Not something you should do.

I like python.

You define functions

def something(x, y):
    return x + y

And you can document functions
def something(x, y):
"""Adds two things together"""
    return x + y

I also like clojure.

You can define functions
(defn something [x y]
  (+ x y))

And you can document functions
(defn something 
  "Adds to things together"
  [x y]
  (+ x y))

Documentation before the arguments? Despicable! If only there was a way of putting them in the right order.

Well, for the sake of argument let's try

Remember that in clojure code is data. A function is just a list, and we want to be able to define functions with some of the items of the list in a different order. At the moment a function definition list looks like this:

(function-name doc-string args function-body)

and we want to be able to make a function using the argument order

(function-name args doc-string function-body)

The first rule of matco club is "Don't write macros". So lets try:

First, how do we want our function (let's call it defndoc) to work? We want it to behave just like a normal function definition but with the docstring after the args.

(defndoc something [x y]
  "Adds to things together"
  (+ x y))

Now let's try to write it. We want to call our defndoc function and have that call defn with the arguments in the correct order.

(defn defndoc [fname args docs & body]
  (defn fname docs args body))

But this isn't going to work as our arguments are going to get evaluated. But this isn't what we want, looks like we will have to write a macro. This is how it looks

(defmacro defndoc [fname args docs & body]
    `(defn ~fname ~docs [~@args] ~@body))

Let's discuss the differences between this and our non-macro attempt.

First we use a syntax-quote (`). This is going to allow us to choose which bits of our list our evaluated and which are not. For example, the defn we want to be evaluated but the other parts we don't.

The next symbol is unquote (~) which tells clojure to evaluate these symbols.

The last symbol is unquote-split (~@) which tells clojure that there is a list of things here that needs to be expanded in place.

now if you do call macroexpand-1 using our defndoc macro on our function with the docstring following the arguments you will get the following

(clojure.core/defn something "Adds two things together" [x y] (+ x y))

Perfect, now we can sprinkle defndoc all over our code and have the docstring in the place we want it, but also keep clojure happy.

Now don't let me ever catch you doing this!

Saturday, June 30, 2012

Synchronized (promise) queues in Clojure

I've mentioned to a few people at work recently that my favourite built in library in python is the queue. The reasons being that the idea is simple, the interface is simple, and you can do all sorts of amazing things with it. We use it as the basis for turning our asynchronous automation interface into a synchronous one and it essentially provides the entire basis for our regression test suite.

In fact, I like it so much I've decided to write it in clojure


There's instructions there about how to use it in your own applications

The idea started back when I was reading Clojure Programming and discovered promises that came in with clojure 1.3.0.

Promises are essentially empty boxes for you to put stuff in. When you want to read from a promise you will block until the promise has a value

(def p (promise))

@p ;; This will block until we have a value

(deliver p :hello)


(realized? p) ;; has the promise got a value? 

From this point on that promise isn't available for delivering to as it already has a value.

This makes it very easy to implement python's queue class. You just need something that behaves like a queue and fill it with promises.

creating a new queue becomes creating a queue with one promise in it.

putting an item onto the queue becomes delivering to the item at the front of the queue then adding a new promise to the end.

getting is just as simple as getting the item at the front of the queue.

The downside is you need special functions to get the size of the queue and emptiness as you only count promises that have been realised

The first version took a couple of hours and was fun, it will be great to know if anyone makes use of it.

Friday, June 22, 2012

NoClassDefFoundError: scala/ScalaObject

I thought I'd start the weekend by trying to call scala code from a java project in eclipse.

I wanted to try it without having to compile the scala code to a jar first. The whole exercise was very painless.

Define some class in scala:

class Cell(val row: Int, val col: Int, val slot: Int) {


Then call it in your java project:

import cell.Cell;

public class JCell {
 public static void main(String[] args) {
  Cell c1 = new Cell(2, 4, 6);
  System.out.println(c1.row() + " " + c1.col() + " " + c1.slot());

The only sticking point was my build path class order needed to have the scala stuff first - which makes perfect sense when you think about it.

Sunday, April 1, 2012

Williams Compression

Now April is with us I thought It's about time I shared an idea for a new compression scheme I've been thinking about. In short, it converts all files into  ~16 bytes of data. Here's how it works:

for bit in file:
    if bit is ON:
         on += 1
         off += 1

output "{on},{off}".

Here's how it looks:

Led Zeppelin Stairway to Heaven: 37810200, 39509240

Stephen Merchant Standup: 283554505, 314875207

Genesis More Fool Me: 14985253, 15456251

In fact. Here's the whole album:

Selling England by the pound: 119882024, 316167032

It's quite lossy, but it allows you to send most files in a single tweet. You could even send the extended version of the Lord of the Rings.

Thursday, February 23, 2012

NeoBlog - My Neo4j Challenge Entry

It's been an interesting start to the year. Towards the end of January I purchased a copy of Seven Databases in Seven weeks to give myself a boost in the world of nosql databases. Within a week I had discovered the neo4j challenge. It seemed too good an opportunity to miss, so I embarked on writing an application for the competition. This is my write up of how it went.
Some Design

I decided early on that my focus was.
  1. Learning about neo4j
  2. Learning about writing web apps in python
  3. Submitting an entry

Getting Started

With this in mind I spent a week playing around with neo4j via Nigel Small's excellent py2neo library. I started off with modelling the london underground in neo4j, and playing around with finding routes around. Here's a tweet I made with a photo of part of the network. This was a great learning activity, I found a bug in the pyneo library which I fixed and Nigel was good enough to pull into his repo. You can see the commit here This was my first real contribution to an open source project, which I was pretty pleased with.

The Blog Idea

Despite having lots of fun playing I couldn't get it quite working the way I wanted, so I decided to keep it on the back burner, and try out another idea I had for the competition. One I knew wasn't going to take much. The idea was a for each node to be a post. Instead of using tags to connect similar posts you would just connect them with edges. It seemed simple enough - so off I went.

The Doing

I'd not done any real web applications in python before - I few toy django applications, but django (and rails for that matter) always feel a bit heavyweight for my liking (that's a topic for another post) so I was looking forward to using flask. The application took shape quite quickly. I spent a few hours adding users and admin pages - but I felt this began to detract from the aim. My intention from the start was to keep the application simple, I felt that an application that would be shared for other people to clone from should be as small as possible. I wanted others to be able to understand the application in under ten minutes, by removing admin pages and users I managed to get rid of about half the code until I was down to an application that could do 3 things:

  • Add a post
  • Link a post to another one
  • View all posts
Not especially ground breaking or shippable - but ok I believe as an example.

What would I do different next time?

When I started, I wasn't totally sure how everything was going to end up, so I decided to play safe and use a language I was familiar with. Looking back at it, I wish I had taken the chance and written it in clojure, I think this would have been an ideal opportunity to play more with clojure.

Something that didn't occur to me until after I deployed and I'm considering adding (I really should do at some point) is that when two people link a post two edges are created. I think instead an edge should have a weighting, and each user that creates a connection adds to the weight. You could then display similar posts in order of similarity. This idea is playing into some other work I'm doing - but I should really add to this one.


I had lots of fun doing this. I learned a bit more about writing web apps. I learned a bit more about git. and I learned how to use neo4j, all in all, not bad for a few days work.


Neo4j Challenge

Neo Blog Entry

Neo Blog Source Code

Wednesday, January 4, 2012

Under: A new Idiom from the J language

Thanks to this post I've started the year discovering a language I never knew existed - and a cool little feature in it.

Imagine you have a function called g

g(5) #returns 10

Now imagine you have another function which undoes whatever happened in g.
undo-g(10) #returns 5

Not too impressive on the face of it. You could guess that g just multiplies by 2 and undo-g divides by 2. The J language comes with some of these built in. Which it calls obverse functions.
   4 + 4
   4 +^:_1 (4)
In this function +^:_1 effectively means apply the + function -1 times. You could do it twice:
   4 +^:_2 (4)
_4 (In J _4 means -4)
Seem crazy? Stay with it...

How many times do you see this sort of pattern in your code?

Look familiar? Well, because J has the idea of obverse functions you get a lovely little syntax that J calls Under which covers this pattern. In J it looks like this
f&.g x
Which means apply g to x. Then apply f. Then apply the inverse of g.
    obverse(func(verb(x))) #J calls functions verbs
The J documentation lists loads of cool definitions you can build using under.
Here's the idea in clojure using the under pattern to construct a new definition of multiplication and addition. It's a cool idea to start the year with. I wonder how many places I'll start seeing this pattern? For more information about J check out the excellent J for C Programmers (Rich 2007)

Tuesday, January 3, 2012

2011: A Retrospective

What goals did I set myself last year?

1) Publish a blog entry or video that explains monads, teach someone at
work how to use them.
I've not done this, I think I understand monads, but I'm looking for someone who knows more than me to confirm I've got it right.

2) Contribute to an open source project related to the arduino.

I didn't do this, I spent the first 3 months playing with the arduino before I moved on to other things. I designed a simple messaging system in google app engine, and then my interest in the arduino tailed off. I could open the source for this, might be an interesting idea.

3) Finish one of my articles and submit it to some publishers for
consideration to publish.

Not really sure what I meant by this. But I spoke at two conferences this year.

What else did I achieve in 2011?

Touch Typing

I commited myself to learning to touch type during the year. Thanks to a lot of support from others and from a wide range of freely available tools I'm now typing comfortably above 50WPM - and hoping this will increase as I continue practising - All my blog posts are now proudly touch typed!

Test Driven Development

I started work on a new product this year at work, and from the offset everyone on the team was encouraged to do TDD. We're ending the year with the product deployed with 80% test coverage - not perfect but not bad. We've also written our own automated acceptance test suite, which tests all of the things that are above the level of our unit test. We don't yet have a way of measuring test coverage here but it seems reasonable that including the automated suite will push our actual coverage above 80%.

Functional Programming

I've played with functional programming at various point in the year. The end of the year has seen me focus more on clojure, and all that lisp languages offer, but Haskell is still there in the background. I need to get some experience building medium sized applications in Clojure or Haskell to increase my confidence

Emacs & Vim

This year I added Emacs and Vim to my list of editors I am comfortable with. I think I'm more on the side of sticking with emacs. But time will tell.


I think it's clear that I deviated from my goals in some pretty dramatic ways. Looks like I need to examine my priorities more often.

Plans for 2012

Programming in Schools (Codemanship Teacher-Practitioner Exchange)

Help teach Ryan enough stuff so he can teach a class in programming.

Back to Basics: Algorithms and Data Strucutres

It's become clear this year that this is an area of my knowledge that needs some attention. I've signed up to Tim Roughgarden's Design and Analysis of Algorithms Course By the end of the year I need to have blogged at least once about and algorithm and once about a data structure.

Functional Programming in Clojure

By the end of the year I need to be comfortable enough to do a project in clojure.


I need to have written a dsl and use it for something.

Review the Retrospective

I should review this post half way throught the year and update if needed.