Why I'm happy to live without generics in go (for now)

I was at GoSF last month (9th May) when Andrew Gerrand gave his talk on go 1.1. At the end of the talk a couple of questions came up on the subject of generics. You can see them on this video at 1:09:20 and 1:14:40. (The questions aren't audible, but the answers are)

If you don't want to watch the video (you should) the summary (and I'm paraphrasing alot here) was essentially that it's not that anyone is trying to keep generics out of go, but rather no one has found a nice way of putting them in. Andrew went on to mention that he didn't miss them at the moment as copying and pasting a few for loops and if statements was an ok alternative.

I nodded in agreement at the time, but last week I proved it to myself in code, so I thought I'd blog about it. I found myself wanting to add up the value of some things in a list, here's a very trivial substitute for the actual code:
It's nothing ground breaking, you could say I'm folding (or reducing) over the list, you could also just say I'm adding up the price of all the things. It would be nice to have a generic fold that I could use to explicitly say that I'm folding over the list, but what advantage would this give me?

I probably wouldn't just call fold by iteself, i'd probably still put fold inside the TotalPriceOfBasket function so that I can state the purpose more explicitly, and the reader would be left with a much simpler looking function, but - and here's my point. fold, map, filter should be recognisable as patterns. You should be able to look at a function and say "Of yeah, a fold, ok"

I'm not making any statements here about how useful generics are or aren't. or how useful a fold/map/filter function would be in go, all I'm saying is that I can live without them for now. Especially as the reason they're not currently in the language is that the core contributors haven't found a good way of implementing them. There's a lot to be said for that approach.

Deploying a development machine with juju

I've just been reading Stavros' blog post about provisioning and deploying virtual machines using ansible: An example of provisioning and deployment with Ansible.
It's worth a read, I've recently come round to the same way of thinking - keeping my development machine as clean as possible and using virtual box machines to install dependencies on, but there's another side to it. Sometimes I want a machine without having to divide my laptop's precious resources, using the cloud that type of thing is easy. But using ubuntu's juju it's even easier.

juju.ubuntu.com is ubuntu's answer to service orchestration. There's a good video of it in use at OSCON 2012. It's great at the large stuff but it's also great at small stuff - which is where this post comes in. I find it helpful to just think of juju as a way of installing charms on a cloud machine, and charms are just scripts. That way it sounds less scary.

I've got some very simple charms for you to get started here. The one I'm going to cover in this post is called devenv. The purpose of devenv is to setup a simple development environment in the cloud. Sometimes I need to make use of mysql, mongo or postgres, so it covers those as well

How to deploy devenv

After you've followed the intructions on http://juju.ubuntu.com/get-started to get setup with your cloud provider it's as simple as running the deploy_devenv.sh script in my charm_collection repo. Here's what's going on under the covers:

juju deploy --repository=charms --constraints "mem=8G" local:precise/devenv

With juju deploy we're expecting to see our charm in the charms/precise/devenv folder below the current directory. Juju then zips the charm up and deploys it on a machine of at least 8GB memory, which we specified with out mem=8G constraint.
Deploying our charm means running a couple of hooks. install and start. Hooks are just scripts which get run on the deployed machine.
Install looks like this:

#!/bin/sh
apt-get -y install vim tmux git bzr mercurial
wget -O /home/ubuntu/.tmux.conf https://raw.github.com/mattyw/dotfiles/master/tmux.conf
wget -O /home/ubuntu/.vimrc https://raw.github.com/mattyw/dotfiles/master/vimrc
echo "set editing-mode vi" > /home/ubuntu/.inputrc

There's nothing much going on here, make sure some packages are installed, copy some of my configuration files from github to the right places and set vi editing-mode in bash. I could, and probably will add to this by getting my install hook to clone the right repositories, install a few more languages and maybe copy around some of my ssh keys, but this is a good start.
Because all I want to do is install some applications and not run any services the start hook just echos that I'm now up and running.

#!/bin/sh
echo "Running"

The interesting part is the relationships with databases, sometimes I need to be able to use a database for my development work. This can be done by making use of juju's relationships. For convenience I've got 3 hooks that point to the same python file:

mongo-relation-changed -> relation_hooks.py
mysql-relation-changed -> relation_hooks.py
pq-relation-changed -> relation_hooks.py

for this relationship all I want to do is write a config file to my home directory, so that I can make use of the db. The database charms set a number of key/value pair using the relation-set command. all I need to do is call the relation-get command when I want to get a particular value. Here's how it's done for mongo and mysql:

def mysql_relation_changed():
    host = os.popen('relation-get host').read().strip()
    user = os.popen('relation-get user').read().strip()
    database = os.popen('relation-get database').read().strip()
    password = os.popen('relation-get password').read().strip()
    slave = os.popen('relation-get slave').read().strip()
    with open('/home/ubuntu/mysql.conf', 'w') as mfile:
        mfile.write('host=%s\n' % host)
        mfile.write('user=%s\n' % user)
        mfile.write('database=%s\n' % database)
        mfile.write('password=%s\n' % password)
        mfile.write('slave=%s\n' % slave)

    print "Done!"


def mongo_relation_changed():
    host = os.popen('relation-get hostname').read().strip()
    port = os.popen('relation-get port').read().strip()
    with open('/home/ubuntu/mongo.conf', 'w') as mfile:
        mfile.write('host=%s\n' % host)
        mfile.write('port=%s\n' % port)

    print "Done!"

To start create a relationship you just need to tell juju to make one:

juju add-relation devenv mongodb

This will add a relation between my devenv machine and a machine running mongo (which I would have deployed previously) From here, I have a cloud machine up and running with all the tools I need, and a mongo db ready to use.

Some useful development patterns via the raspberry pi & minecraft

Programming minecraft on the raspberry pi is not just fun, it's also a great chance to learn some of the cool things that can be done from the linux command line.
We're going to look at two problems:

1. How can I edit files on my raspberry pi using an editor on my pc?
2. How can I run minecraft on a raspberry pi with a screen connected but without a mouse and keyboard connected?
3. Minecraft fails to start: something about a problem loading shared library libGLESv2.so

The Setup

• 1 raspberry pi connected to a network and a monitor
• 1 PC (I'm using linux, it should work equally well on osx, not sure about windows.)

How can I edit files on my raspberry pi using an editor on my pc?

Create an empty folder on your PC:

mkdir ~/pi

Then use sshfs to mount it

sshfs pi@192.168.1.82: ~/pi/

Where pi is my user name and 192.168.1.82 is the ip address of the pi.
Now, on my pc if I cd to ~/pi I can see the files in the home directory of my pi, which means I can open them on my pc using my editor of choice

How can I run minecraft on a raspberry pi with a screen connected but without a mouse and keyboard connected?

From your pc ssh into the pi specifying -X

ssh -X pi@192.168.1.82

This turns on X11 forwarding - you don't need to know what that means for now, just start minecraft from inside ssh. You should see an empty minecraft window appear
Now take a look at your monitor, you should see minecraft there and be able to use your laptop's mouse and keyboard to move yourself around
As a bonus, you should be able to drag the window around on your pc, and have the window move around on your pi as well

Minecraft fails to start: something about a problem loading shared library libGLESv2.so

When I run minecraft I get this:

./minecraft-pi
error while loading shared libraries: libGLESv2.so: cannot open shared object file: No such file or directory

Shared libraries are essentially bits of code that are shared amongst many programs. You can see what shared objects a program needs by typing

ldd ./minecraft-pi

You might see some stuff marked "not found" at the top

        /usr/lib/arm-linux-gnueabihf/libcofi_rpi.so (0x4022c000)
        libGLESv2.so => not found
        libEGL.so => not found
        libbcm_host.so => not found
        libpng12.so.0 => /lib/arm-linux-gnueabihf/libpng12.so.0 (0x40182000)
        libSDL-1.2.so.0 => /usr/lib/arm-linux-gnueabihf/libSDL-1.2.so.0 (0x40235000)
        libstdc++.so.6 => /usr/lib/arm-linux-gnueabihf/libstdc++.so.6 (0x402c8000)
        libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0x401aa000)

To help linux find some of these shared object you can specify an option on the command line before you run your command:

LD_LIBRARY_PATH=/opt/vc/lib ./minecraft-pi

Using go to unmarshal json lists with multiple types

Everyday I seem to be writing go code to parse a json string, and this problem seems to come up often enough for me to write about it. Thanks to adg and asoko on #go-nuts for their suggestions.

The Problem

Given a list of json objects of different types (lets say People and Places). You want to Unmarshal them into two lists. A list of all the people and a list of all the places.

A bit more definition

Let's use this json string

{
    "things": [
        {
            "name": "Alice",
            "age": 37
        },
        {
            "city": "Ipoh",
            "country": "Malaysia"
        },
        {
            "name": "Bob",
            "age": 36
        },
        {
            "city": "Northampton",
            "country": "England"
        }
    ]
}

To help us write some code, let's give ourselves a function, which should be self explanatory:

func solution(jsonString []byte) ([]Person []Place) {}

And some structures

type Person struct {
 Name string
 Age  int
}

type Place struct {
 City    string
 Country string
}

I've got two solutions to this problem. I would love to know of better ways.

SolutionA: map and type assert

If we tell json to unmarshal into a map we can get it to deal with the parts we know about, and the rest of it will go into an interface{}. As we loop over the json structures we use what we do know about the structures to pass the interface{} to some helper functions what will create one of our structs and add it to our list. Because the map we take in is a map[string]interface{} we will need to type assert our values

func solutionA(jsonStr []byte) ([]Person, []Place) {
 persons := []Person{}
 places := []Place{}
 var data map[string][]map[string]interface{}
 err := json.Unmarshal(jsonStr, &data)
 if err != nil {
  fmt.Println(err)
  return persons, places
 }

 for i := range data["things"] {
  item := data["things"][i]
  if item["name"] != nil {
   persons = addPerson(persons, item)
  } else {
   places = addPlace(places, item)
  }

 }
 return persons, places
}

func addPerson(persons []Person, item map[string]interface{}) []Person {
 name, _ := item["name"].(string)
 age, _ := item["age"].(int)
 person := Person{name, age}
 persons = append(persons, person)
 return persons
}

func addPlace(places []Place, item map[string]interface{}) []Place {
 city, _ := item["city"].(string)
 country, _ := item["city"].(string)
 place := Place{city, country}
 places = append(places, place)
 return places
}

SolutionB: Mixed Type struct

This solution involves creating an interim struct which can be used to represent either a person or a place

type Mixed struct {
 Name    string `json:"name"`
 Age     int    `json:"age"`
 City    string `json:"city"`
 Country string `json:"country"`
}

With this struct we can then unmarshal our json string into a list of these mixed types. As we loop over our Mixed structs we just need to examine each one to work out which type it represents, and then build the right struct from it

func solutionB(jsonStr []byte) ([]Person, []Place) {
 persons := []Person{}
 places := []Place{}
 var data map[string][]Mixed
 err := json.Unmarshal(jsonStr, &data)
 if err != nil {
  fmt.Println(err)
  return persons, places
 }

 for i := range data["things"] {
  item := data["things"][i]
  if item.Name != "" {
   persons = append(persons, Person{item.Name, item.Age})
  } else {
   places = append(places, Place{item.City, item.Country})
  }

 }
 return persons, places
}

These are just two ways I've used to solve these problems, I'd love to know how others have done it.

SolutionC: json.RawMessage (Updated 18Jan13)

Thanks to Jordan's comment and zemo on reddit there is another solution. Using the json.RawMessage structure in the json package we can delay unmarshalling the json structures in the list. We can then go through our list and unmarshal each of them into the correct type

func solutionC(jsonStr []byte) ([]Person, []Place) {
 people := []Person{}
 places := []Place{}
 var data map[string][]json.RawMessage
 err := json.Unmarshal(jsonStr, &data)
 if err != nil {
  fmt.Println(err)
  return people, places
 }
 for _, thing := range data["things"] {
            people = addPersonC(thing, people)
            places = addPlaceC(thing, places)
        }
 return people, places
}

func addPersonC(thing json.RawMessage, people []Person) []Person {
    person := Person{}
    if err := json.Unmarshal(thing, &person); err != nil {
        fmt.Println(err)
    } else {
        if person != *new(Person) {
            people = append(people, person)
        }
    }

    return people
}

func addPlaceC(thing json.RawMessage, places []Place) []Place {
    place := Place{}
    if err := json.Unmarshal(thing, &place); err != nil {
        fmt.Println(err)
    } else {
        if place != *new(Place) {
            places = append(places, place)
        }
    }

    return places
}

Here's the full gist:

Using vim's path to speed up your Go project

If you're working on a project in Go, and you're using vim, then you really need to check out path:

:help path

Here's a very quick tour.

Add this to you .vimrc:

set path +=/your/projects/gopath/src/**

** tells path to include all subdirectories. By adding your projects GOPATH entry it opens up access to two rather cool features that are useful in navigating your project.

:find

You can use :find to open a file instead of using :e. But :find will look in all of your project's directories to find the file. So rather than having to do:

:e /package/v10/another-dir/god.go

You just need:

:find god.go

You also get tab-completion for free - just in case you can't remember the file's name.

gf

gf is the useful little command that goes to the file under the cursor. Well providing you've set your path correcly you can navigate your project's imports with ease:

package main

import (
 "code.google.com/p/go.net/websocket"
)

By "gf"ing over the import vim will open the directory, giving you the list of files in that package.

Just don't dare navigate the directory using the arrow keys!

Update: 8th Jan 2012

On the subject of vim and Go it's worth remembering this little gem:

au BufWritePost *.go !gofmt -w %

Whenever a Go file is saved gofmt will be run against it.

go(ing) under

Lots of people like go and this year I began to find it unavoidable - I'm even ending the year with go being the main language that helps pay my bills!

One of the things that go lets you do is pass functions around.
I started the year talking about under from the j language. Which lets you 'bookend' some function with another function (a verb) and a function which undoes that function (obverse).
A great example is Pythagoras' theorem:

1. You perform some 'pre-processing' on each value (squaring)
2. You add all the values up
3. You square root the result

I started the year talking about under in clojure. Here it is implemented in go. Complete with functions being passed around, closures, and functions returning functions. Happy new year!

Running juju locally on 12.04 (or getting over agent-status pending)

I've just tried following the instructions here to get juju running locally on my brand new Ubuntu 12.04 (desktop) system.
No matter what I tried my wordpress and mysql charms seemed stuck in agent-status pending and ip address null. Even waiting a couple of hours.

I needed to take to a couple of extra steps to get it working that didn't seem to be documented in one place. So I thought I'd put them both here.

All it took was to add a rule to ufw:

sudo ufw allow from 192.168.122.0/24 to any

Hope this helps someone who was stuck in the same way I was.

Clojure, Vim and the delay sending to screen

So, long story short: I'm use vim for writing clojure (and everything else for that matter)

If you're following the setup How I develop Clojure with Vim on the :wq blog then you might encounter a problem where it takes a few seconds for code you send from vim to appear in your screen session. Here's the solution.

Add the following lines to you ~/.screenrc

msgwait 0
msgminwait 0

And there you have it, instant code from vim to your REPL!

Another clojure macro tutorial (that no one should follow)

Disclaimer: This post shows you something you can do with macros. Not something you should do.

I like python.

You define functions

def something(x, y):
    return x + y

And you can document functions

def something(x, y):
"""Adds two things together"""
    return x + y

I also like clojure.

You can define functions

(defn something [x y]
  (+ x y))

And you can document functions

(defn something 
  "Adds to things together"
  [x y]
  (+ x y))

Documentation before the arguments? Despicable! If only there was a way of putting them in the right order.

Well, for the sake of argument let's try

Remember that in clojure code is data. A function is just a list, and we want to be able to define functions with some of the items of the list in a different order. At the moment a function definition list looks like this:

(function-name doc-string args function-body)

and we want to be able to make a function using the argument order

(function-name args doc-string function-body)

The first rule of matco club is "Don't write macros". So lets try:

First, how do we want our function (let's call it defndoc) to work? We want it to behave just like a normal function definition but with the docstring after the args.

(defndoc something [x y]
  "Adds to things together"
  (+ x y))

Now let's try to write it. We want to call our defndoc function and have that call defn with the arguments in the correct order.

(defn defndoc [fname args docs & body]
  (defn fname docs args body))

But this isn't going to work as our arguments are going to get evaluated. But this isn't what we want, looks like we will have to write a macro. This is how it looks

(defmacro defndoc [fname args docs & body]
    `(defn ~fname ~docs [~@args] ~@body))

Let's discuss the differences between this and our non-macro attempt.

First we use a syntax-quote (`). This is going to allow us to choose which bits of our list our evaluated and which are not. For example, the defn we want to be evaluated but the other parts we don't.

The next symbol is unquote (~) which tells clojure to evaluate these symbols.

The last symbol is unquote-split (~@) which tells clojure that there is a list of things here that needs to be expanded in place.

now if you do call macroexpand-1 using our defndoc macro on our function with the docstring following the arguments you will get the following

(clojure.core/defn something "Adds two things together" [x y] (+ x y))

Perfect, now we can sprinkle defndoc all over our code and have the docstring in the place we want it, but also keep clojure happy.

Now don't let me ever catch you doing this!

Synchronized (promise) queues in Clojure

I've mentioned to a few people at work recently that my favourite built in library in python is the queue. The reasons being that the idea is simple, the interface is simple, and you can do all sorts of amazing things with it. We use it as the basis for turning our asynchronous automation interface into a synchronous one and it essentially provides the entire basis for our regression test suite.

In fact, I like it so much I've decided to write it in clojure

sync-q

There's instructions there about how to use it in your own applications

The idea started back when I was reading Clojure Programming and discovered promises that came in with clojure 1.3.0.

Promises are essentially empty boxes for you to put stuff in. When you want to read from a promise you will block until the promise has a value

(def p (promise))

@p ;; This will block until we have a value

(deliver p :hello)

@p
:hello

(realized? p) ;; has the promise got a value? 
true 

From this point on that promise isn't available for delivering to as it already has a value.

This makes it very easy to implement python's queue class. You just need something that behaves like a queue and fill it with promises.

creating a new queue becomes creating a queue with one promise in it.

putting an item onto the queue becomes delivering to the item at the front of the queue then adding a new promise to the end.

getting is just as simple as getting the item at the front of the queue.

The downside is you need special functions to get the size of the queue and emptiness as you only count promises that have been realised

The first version took a couple of hours and was fun, it will be great to know if anyone makes use of it.

NoClassDefFoundError: scala/ScalaObject

I thought I'd start the weekend by trying to call scala code from a java project in eclipse.

I wanted to try it without having to compile the scala code to a jar first. The whole exercise was very painless.

Define some class in scala:

class Cell(val row: Int, val col: Int, val slot: Int) {

}

Then call it in your java project:

import cell.Cell;

public class JCell {
 public static void main(String[] args) {
  Cell c1 = new Cell(2, 4, 6);
  System.out.println(c1.row() + " " + c1.col() + " " + c1.slot());
}
}

The only sticking point was my build path class order needed to have the scala stuff first - which makes perfect sense when you think about it.

Williams Compression

Now April is with us I thought It's about time I shared an idea for a new compression scheme I've been thinking about. In short, it converts all files into  ~16 bytes of data. Here's how it works:

for bit in file:
    if bit is ON:
         on += 1
    else:
         off += 1

output "{on},{off}".

Here's how it looks:

Led Zeppelin Stairway to Heaven: 37810200, 39509240

Stephen Merchant Standup: 283554505, 314875207

Genesis More Fool Me: 14985253, 15456251

In fact. Here's the whole album:

Selling England by the pound: 119882024, 316167032

It's quite lossy, but it allows you to send most files in a single tweet. You could even send the extended version of the Lord of the Rings.

NeoBlog - My Neo4j Challenge Entry

It's been an interesting start to the year. Towards the end of January I purchased a copy of Seven Databases in Seven weeks to give myself a boost in the world of nosql databases. Within a week I had discovered the neo4j challenge. It seemed too good an opportunity to miss, so I embarked on writing an application for the competition. This is my write up of how it went.
Some Design

I decided early on that my focus was.

1. Learning about neo4j
2. Learning about writing web apps in python
3. Submitting an entry

Getting Started

With this in mind I spent a week playing around with neo4j via Nigel Small's excellent py2neo library. I started off with modelling the london underground in neo4j, and playing around with finding routes around. Here's a tweet I made with a photo of part of the network. This was a great learning activity, I found a bug in the pyneo library which I fixed and Nigel was good enough to pull into his repo. You can see the commit here This was my first real contribution to an open source project, which I was pretty pleased with.


The Blog Idea

Despite having lots of fun playing I couldn't get it quite working the way I wanted, so I decided to keep it on the back burner, and try out another idea I had for the competition. One I knew wasn't going to take much. The idea was a for each node to be a post. Instead of using tags to connect similar posts you would just connect them with edges. It seemed simple enough - so off I went.


The Doing

I'd not done any real web applications in python before - I few toy django applications, but django (and rails for that matter) always feel a bit heavyweight for my liking (that's a topic for another post) so I was looking forward to using flask. The application took shape quite quickly. I spent a few hours adding users and admin pages - but I felt this began to detract from the aim. My intention from the start was to keep the application simple, I felt that an application that would be shared for other people to clone from should be as small as possible. I wanted others to be able to understand the application in under ten minutes, by removing admin pages and users I managed to get rid of about half the code until I was down to an application that could do 3 things:

• Add a post
• Link a post to another one
• View all posts

Not especially ground breaking or shippable - but ok I believe as an example.

What would I do different next time?

When I started, I wasn't totally sure how everything was going to end up, so I decided to play safe and use a language I was familiar with. Looking back at it, I wish I had taken the chance and written it in clojure, I think this would have been an ideal opportunity to play more with clojure.

Something that didn't occur to me until after I deployed and I'm considering adding (I really should do at some point) is that when two people link a post two edges are created. I think instead an edge should have a weighting, and each user that creates a connection adds to the weight. You could then display similar posts in order of similarity. This idea is playing into some other work I'm doing - but I should really add to this one.

Summary

I had lots of fun doing this. I learned a bit more about writing web apps. I learned a bit more about git. and I learned how to use neo4j, all in all, not bad for a few days work.

References:

Neo4j Challenge

Neo Blog Entry

Neo Blog Source Code

Under: A new Idiom from the J language

Thanks to this post I've started the year discovering a language I never knew existed - and a cool little feature in it.

Imagine you have a function called g

g(5) #returns 10

Now imagine you have another function which undoes whatever happened in g.

undo-g(10) #returns 5

Not too impressive on the face of it. You could guess that g just multiplies by 2 and undo-g divides by 2. The J language comes with some of these built in. Which it calls obverse functions.

   4 + 4
8
   4 +^:_1 (4)
0

In this function +^:_1 effectively means apply the + function -1 times. You could do it twice:

   4 +^:_2 (4)
_4 (In J _4 means -4)

Seem crazy? Stay with it...

How many times do you see this sort of pattern in your code?

OpenFile
    ReadData
CloseFile

OpenSocket
    SendData
CloseSocket

Look familiar? Well, because J has the idea of obverse functions you get a lovely little syntax that J calls Under which covers this pattern. In J it looks like this

f&.g x

Which means apply g to x. Then apply f. Then apply the inverse of g.

    obverse(func(verb(x))) #J calls functions verbs

The J documentation lists loads of cool definitions you can build using under.
Here's the idea in clojure using the under pattern to construct a new definition of multiplication and addition. It's a cool idea to start the year with. I wonder how many places I'll start seeing this pattern? For more information about J check out the excellent J for C Programmers (Rich 2007)

2011: A Retrospective

What goals did I set myself last year?

1) Publish a blog entry or video that explains monads, teach someone at
work how to use them.
I've not done this, I think I understand monads, but I'm looking for someone who knows more than me to confirm I've got it right.

2) Contribute to an open source project related to the arduino.

I didn't do this, I spent the first 3 months playing with the arduino before I moved on to other things. I designed a simple messaging system in google app engine, and then my interest in the arduino tailed off. I could open the source for this, might be an interesting idea.

3) Finish one of my articles and submit it to some publishers for
consideration to publish.

Not really sure what I meant by this. But I spoke at two conferences this year.

What else did I achieve in 2011?

Touch Typing

I commited myself to learning to touch type during the year. Thanks to a lot of support from others and from a wide range of freely available tools I'm now typing comfortably above 50WPM - and hoping this will increase as I continue practising - All my blog posts are now proudly touch typed!

Test Driven Development

I started work on a new product this year at work, and from the offset everyone on the team was encouraged to do TDD. We're ending the year with the product deployed with 80% test coverage - not perfect but not bad. We've also written our own automated acceptance test suite, which tests all of the things that are above the level of our unit test. We don't yet have a way of measuring test coverage here but it seems reasonable that including the automated suite will push our actual coverage above 80%.

Functional Programming

I've played with functional programming at various point in the year. The end of the year has seen me focus more on clojure, and all that lisp languages offer, but Haskell is still there in the background. I need to get some experience building medium sized applications in Clojure or Haskell to increase my confidence

Emacs & Vim

This year I added Emacs and Vim to my list of editors I am comfortable with. I think I'm more on the side of sticking with emacs. But time will tell.

Summary

I think it's clear that I deviated from my goals in some pretty dramatic ways. Looks like I need to examine my priorities more often.

Plans for 2012

Programming in Schools (Codemanship Teacher-Practitioner Exchange)

Help teach Ryan enough stuff so he can teach a class in programming.

Back to Basics: Algorithms and Data Strucutres

It's become clear this year that this is an area of my knowledge that needs some attention. I've signed up to Tim Roughgarden's Design and Analysis of Algorithms Course By the end of the year I need to have blogged at least once about and algorithm and once about a data structure.

Functional Programming in Clojure

By the end of the year I need to be comfortable enough to do a project in clojure.

DSLs

I need to have written a dsl and use it for something.

Review the Retrospective

I should review this post half way throught the year and update if needed.

String equality, identity and interning in Python

In a list of things I should have already known comes this. The difference between using 'is' and == on strings in Python.

Let's look at two strings. One unicode (u"unicode string") and one not "not unicode string".

Python 2.7.2+ (default, Oct  4 2011, 20:03:08) 
>>> type("foo")
type 'str'
>>> type(u"foo")
type "unicode"
>>> u"foo" == "foo"
True
>>> u"foo" is "foo"
False

So using == shows the two strings as equal, and 'is' doesn't. What's going on here?

Python interns its strings. Which means only one copy of each distinct string is stored. You can see this by using the built-in function id() to see the identity of our strings.

>>> a = "foo"
>>> b = "foo"
>>> c = u"foo"
>>> print id(a)
3074129864
>>> print id(b)
3074129864
>>> print id(c)
3074128400

You can see our normal strings have the same id because they are the same object. Our unicode string has a different id to our two 'normal' strings. Using the == operator asks python to compare equality of our two strings. Using 'is' compares the identity. As our unicode and normal string are different objects, comparing with 'is' returns false.

I wonder how many of us are guilty of misusing 'is' on strings?

Heroku and Ubuntu

So, I've just spent an hour or two trying to make a new heroku app on my ubuntu machine, and I was getting nowhere.

$git push heroku master
Agent admitted failure to sign using the key.
Permission denied (publickey).
fatal: The remote end hung up unexpectedly

I'd followed all the usual help on stackoverflow and heroku's excellent help section but was making no progress. Until I discovered this. The link to the bug report didn't work for me, but it's clear that I needed to set this SSH_AUTH_SOCK=0 environment variable first:

$export SSH_AUTH_SOCK=0
$git push heroku master

And now we're off!

The Sieve of Eratosthenes in Python

Whilst working on the 10 Io one liners to impress your friends post I felt I needed to turn to python to complete number 10 - the Sieve of Eratosthenes. My intention was to understand it in python as best I could, then simplify the python code until I had one line I could try to translate into Io. This post is about that attempt.
We start off by trying to translate the description in the wikipedia article into python line by line. Which gives us the following.

def esieve(n):
    primes = range(2, n+1)
    p = 2
    while p < n:
        for i in range(p, n+1):
            if p*i in primes:
                primes.remove(p*i)
        p += 1
    return primes

9 Lines isn't bad for a start, but that if statement can be cleaned up. What if instead of looking for items in our list one at a time. We make a new list of items to be removed, and remove them all at the end? We could use a set to hold our lists. This lets us use the minus (-) operator to give us a new set of items not in our marked set.

def shorter_esieve(n):
    marked = set()
    p = 2
    while p < n:
        for i in range(p, n+1):
            marked.add(p*i)
        p += 1
    return sorted(set(range(2, n+1)) - marked)

We only removed one line in that last attempt. Not great. But it looks like we're using a while loop and incrementing each step. Why don't we just do a for?

def shorter_esieve(n):
    marked = set()
    for p in range(2, n+1):
        for i in range(p, n+1):
            marked.add(p*i)
    return sorted(set(range(2, n+1)) - marked)

6 lines, getting better. Now here is the magic. We're using two for loops to generate a set of values. So we can just use a list comprehension to build our list, which we then use to make our marked set.

def shorter_esieve(n):
    marked = set([p* i for p in range(2, n+1) for i in range(p, n+1)])
    return sorted(set(range(2, n+1)) - marked)

And moving the assignment inline

def much_shorter_esieve(n):
    return sorted(set(range(2, n+1)) - set([p*i for p in range(2, n+1) for i in range(p, n+1)]))

And there we have it. The Sieve of Eratosthenes in one line of python. If you'd rather watch the refactoring happening step by step. Here's a video, set to suitable music.

10 Io one liners to impress your friends

It's been ages since I've done anything with the Io language. But after seeing this spate of 10 [language] one liners to impress your friends in Davide Varvello's post I thought I would spend the evening trying it with Io. Here's the gist: According to Davide's post the others in this category so far are:
Ruby
Scala
CoffeeScript
Haskell
Clojure
Python
Groovy
I'd love to see more. It's a great little activity to get started with a language.

Chicken Scheme on OS X 10.7 (Lion)

I'm making another attempt at going through SICP doing all of the exercises. I've decided that chicken scheme is going to be the way I will do it. But to get it going on Lion there are some hoops to jump through thanks largely to a new version of gcc.

The method I have found that works is.

1) Download the source from the chicken scheme website.

2) Following the advice here you should pass new compiler options to make:

make C_COMPILER=gcc-4.2 PLATFORM=macosx

3) To install the additional eggs you will need to pass new options to csc like so:

CSC_OPTIONS='-cc gcc-4.2' chicken-install