Why I'm happy to live without generics in go (for now)

I was at GoSF last month (9th May) when Andrew Gerrand gave his talk on go 1.1. At the end of the talk a couple of questions came up on the subject of generics. You can see them on this video at 1:09:20 and 1:14:40. (The questions aren't audible, but the answers are)

If you don't want to watch the video (you should) the summary (and I'm paraphrasing alot here) was essentially that it's not that anyone is trying to keep generics out of go, but rather no one has found a nice way of putting them in. Andrew went on to mention that he didn't miss them at the moment as copying and pasting a few for loops and if statements was an ok alternative.

I nodded in agreement at the time, but last week I proved it to myself in code, so I thought I'd blog about it. I found myself wanting to add up the value of some things in a list, here's a very trivial substitute for the actual code:
It's nothing ground breaking, you could say I'm folding (or reducing) over the list, you could also just say I'm adding up the price of all the things. It would be nice to have a generic fold that I could use to explicitly say that I'm folding over the list, but what advantage would this give me?

I probably wouldn't just call fold by iteself, i'd probably still put fold inside the TotalPriceOfBasket function so that I can state the purpose more explicitly, and the reader would be left with a much simpler looking function, but - and here's my point. fold, map, filter should be recognisable as patterns. You should be able to look at a function and say "Of yeah, a fold, ok"

I'm not making any statements here about how useful generics are or aren't. or how useful a fold/map/filter function would be in go, all I'm saying is that I can live without them for now. Especially as the reason they're not currently in the language is that the core contributors haven't found a good way of implementing them. There's a lot to be said for that approach.

Deploying a development machine with juju

I've just been reading Stavros' blog post about provisioning and deploying virtual machines using ansible: An example of provisioning and deployment with Ansible.
It's worth a read, I've recently come round to the same way of thinking - keeping my development machine as clean as possible and using virtual box machines to install dependencies on, but there's another side to it. Sometimes I want a machine without having to divide my laptop's precious resources, using the cloud that type of thing is easy. But using ubuntu's juju it's even easier.

juju.ubuntu.com is ubuntu's answer to service orchestration. There's a good video of it in use at OSCON 2012. It's great at the large stuff but it's also great at small stuff - which is where this post comes in. I find it helpful to just think of juju as a way of installing charms on a cloud machine, and charms are just scripts. That way it sounds less scary.

I've got some very simple charms for you to get started here. The one I'm going to cover in this post is called devenv. The purpose of devenv is to setup a simple development environment in the cloud. Sometimes I need to make use of mysql, mongo or postgres, so it covers those as well

How to deploy devenv

After you've followed the intructions on http://juju.ubuntu.com/get-started to get setup with your cloud provider it's as simple as running the deploy_devenv.sh script in my charm_collection repo. Here's what's going on under the covers:

juju deploy --repository=charms --constraints "mem=8G" local:precise/devenv

With juju deploy we're expecting to see our charm in the charms/precise/devenv folder below the current directory. Juju then zips the charm up and deploys it on a machine of at least 8GB memory, which we specified with out mem=8G constraint.
Deploying our charm means running a couple of hooks. install and start. Hooks are just scripts which get run on the deployed machine.
Install looks like this:

#!/bin/sh
apt-get -y install vim tmux git bzr mercurial
wget -O /home/ubuntu/.tmux.conf https://raw.github.com/mattyw/dotfiles/master/tmux.conf
wget -O /home/ubuntu/.vimrc https://raw.github.com/mattyw/dotfiles/master/vimrc
echo "set editing-mode vi" > /home/ubuntu/.inputrc

There's nothing much going on here, make sure some packages are installed, copy some of my configuration files from github to the right places and set vi editing-mode in bash. I could, and probably will add to this by getting my install hook to clone the right repositories, install a few more languages and maybe copy around some of my ssh keys, but this is a good start.
Because all I want to do is install some applications and not run any services the start hook just echos that I'm now up and running.

#!/bin/sh
echo "Running"

The interesting part is the relationships with databases, sometimes I need to be able to use a database for my development work. This can be done by making use of juju's relationships. For convenience I've got 3 hooks that point to the same python file:

mongo-relation-changed -> relation_hooks.py
mysql-relation-changed -> relation_hooks.py
pq-relation-changed -> relation_hooks.py

for this relationship all I want to do is write a config file to my home directory, so that I can make use of the db. The database charms set a number of key/value pair using the relation-set command. all I need to do is call the relation-get command when I want to get a particular value. Here's how it's done for mongo and mysql:

def mysql_relation_changed():
    host = os.popen('relation-get host').read().strip()
    user = os.popen('relation-get user').read().strip()
    database = os.popen('relation-get database').read().strip()
    password = os.popen('relation-get password').read().strip()
    slave = os.popen('relation-get slave').read().strip()
    with open('/home/ubuntu/mysql.conf', 'w') as mfile:
        mfile.write('host=%s\n' % host)
        mfile.write('user=%s\n' % user)
        mfile.write('database=%s\n' % database)
        mfile.write('password=%s\n' % password)
        mfile.write('slave=%s\n' % slave)

    print "Done!"


def mongo_relation_changed():
    host = os.popen('relation-get hostname').read().strip()
    port = os.popen('relation-get port').read().strip()
    with open('/home/ubuntu/mongo.conf', 'w') as mfile:
        mfile.write('host=%s\n' % host)
        mfile.write('port=%s\n' % port)

    print "Done!"

To start create a relationship you just need to tell juju to make one:

juju add-relation devenv mongodb

This will add a relation between my devenv machine and a machine running mongo (which I would have deployed previously) From here, I have a cloud machine up and running with all the tools I need, and a mongo db ready to use.

Some useful development patterns via the raspberry pi & minecraft

Programming minecraft on the raspberry pi is not just fun, it's also a great chance to learn some of the cool things that can be done from the linux command line.
We're going to look at two problems:

1. How can I edit files on my raspberry pi using an editor on my pc?
2. How can I run minecraft on a raspberry pi with a screen connected but without a mouse and keyboard connected?
3. Minecraft fails to start: something about a problem loading shared library libGLESv2.so

The Setup

• 1 raspberry pi connected to a network and a monitor
• 1 PC (I'm using linux, it should work equally well on osx, not sure about windows.)

How can I edit files on my raspberry pi using an editor on my pc?

Create an empty folder on your PC:

mkdir ~/pi

Then use sshfs to mount it

sshfs pi@192.168.1.82: ~/pi/

Where pi is my user name and 192.168.1.82 is the ip address of the pi.
Now, on my pc if I cd to ~/pi I can see the files in the home directory of my pi, which means I can open them on my pc using my editor of choice

How can I run minecraft on a raspberry pi with a screen connected but without a mouse and keyboard connected?

From your pc ssh into the pi specifying -X

ssh -X pi@192.168.1.82

This turns on X11 forwarding - you don't need to know what that means for now, just start minecraft from inside ssh. You should see an empty minecraft window appear
Now take a look at your monitor, you should see minecraft there and be able to use your laptop's mouse and keyboard to move yourself around
As a bonus, you should be able to drag the window around on your pc, and have the window move around on your pi as well

Minecraft fails to start: something about a problem loading shared library libGLESv2.so

When I run minecraft I get this:

./minecraft-pi
error while loading shared libraries: libGLESv2.so: cannot open shared object file: No such file or directory

Shared libraries are essentially bits of code that are shared amongst many programs. You can see what shared objects a program needs by typing

ldd ./minecraft-pi

You might see some stuff marked "not found" at the top

        /usr/lib/arm-linux-gnueabihf/libcofi_rpi.so (0x4022c000)
        libGLESv2.so => not found
        libEGL.so => not found
        libbcm_host.so => not found
        libpng12.so.0 => /lib/arm-linux-gnueabihf/libpng12.so.0 (0x40182000)
        libSDL-1.2.so.0 => /usr/lib/arm-linux-gnueabihf/libSDL-1.2.so.0 (0x40235000)
        libstdc++.so.6 => /usr/lib/arm-linux-gnueabihf/libstdc++.so.6 (0x402c8000)
        libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0x401aa000)

To help linux find some of these shared object you can specify an option on the command line before you run your command:

LD_LIBRARY_PATH=/opt/vc/lib ./minecraft-pi

Using go to unmarshal json lists with multiple types

Everyday I seem to be writing go code to parse a json string, and this problem seems to come up often enough for me to write about it. Thanks to adg and asoko on #go-nuts for their suggestions.

The Problem

Given a list of json objects of different types (lets say People and Places). You want to Unmarshal them into two lists. A list of all the people and a list of all the places.

A bit more definition

Let's use this json string

{
    "things": [
        {
            "name": "Alice",
            "age": 37
        },
        {
            "city": "Ipoh",
            "country": "Malaysia"
        },
        {
            "name": "Bob",
            "age": 36
        },
        {
            "city": "Northampton",
            "country": "England"
        }
    ]
}

To help us write some code, let's give ourselves a function, which should be self explanatory:

func solution(jsonString []byte) ([]Person []Place) {}

And some structures

type Person struct {
 Name string
 Age  int
}

type Place struct {
 City    string
 Country string
}

I've got two solutions to this problem. I would love to know of better ways.

SolutionA: map and type assert

If we tell json to unmarshal into a map we can get it to deal with the parts we know about, and the rest of it will go into an interface{}. As we loop over the json structures we use what we do know about the structures to pass the interface{} to some helper functions what will create one of our structs and add it to our list. Because the map we take in is a map[string]interface{} we will need to type assert our values

func solutionA(jsonStr []byte) ([]Person, []Place) {
 persons := []Person{}
 places := []Place{}
 var data map[string][]map[string]interface{}
 err := json.Unmarshal(jsonStr, &data)
 if err != nil {
  fmt.Println(err)
  return persons, places
 }

 for i := range data["things"] {
  item := data["things"][i]
  if item["name"] != nil {
   persons = addPerson(persons, item)
  } else {
   places = addPlace(places, item)
  }

 }
 return persons, places
}

func addPerson(persons []Person, item map[string]interface{}) []Person {
 name, _ := item["name"].(string)
 age, _ := item["age"].(int)
 person := Person{name, age}
 persons = append(persons, person)
 return persons
}

func addPlace(places []Place, item map[string]interface{}) []Place {
 city, _ := item["city"].(string)
 country, _ := item["city"].(string)
 place := Place{city, country}
 places = append(places, place)
 return places
}

SolutionB: Mixed Type struct

This solution involves creating an interim struct which can be used to represent either a person or a place

type Mixed struct {
 Name    string `json:"name"`
 Age     int    `json:"age"`
 City    string `json:"city"`
 Country string `json:"country"`
}

With this struct we can then unmarshal our json string into a list of these mixed types. As we loop over our Mixed structs we just need to examine each one to work out which type it represents, and then build the right struct from it

func solutionB(jsonStr []byte) ([]Person, []Place) {
 persons := []Person{}
 places := []Place{}
 var data map[string][]Mixed
 err := json.Unmarshal(jsonStr, &data)
 if err != nil {
  fmt.Println(err)
  return persons, places
 }

 for i := range data["things"] {
  item := data["things"][i]
  if item.Name != "" {
   persons = append(persons, Person{item.Name, item.Age})
  } else {
   places = append(places, Place{item.City, item.Country})
  }

 }
 return persons, places
}

These are just two ways I've used to solve these problems, I'd love to know how others have done it.

SolutionC: json.RawMessage (Updated 18Jan13)

Thanks to Jordan's comment and zemo on reddit there is another solution. Using the json.RawMessage structure in the json package we can delay unmarshalling the json structures in the list. We can then go through our list and unmarshal each of them into the correct type

func solutionC(jsonStr []byte) ([]Person, []Place) {
 people := []Person{}
 places := []Place{}
 var data map[string][]json.RawMessage
 err := json.Unmarshal(jsonStr, &data)
 if err != nil {
  fmt.Println(err)
  return people, places
 }
 for _, thing := range data["things"] {
            people = addPersonC(thing, people)
            places = addPlaceC(thing, places)
        }
 return people, places
}

func addPersonC(thing json.RawMessage, people []Person) []Person {
    person := Person{}
    if err := json.Unmarshal(thing, &person); err != nil {
        fmt.Println(err)
    } else {
        if person != *new(Person) {
            people = append(people, person)
        }
    }

    return people
}

func addPlaceC(thing json.RawMessage, places []Place) []Place {
    place := Place{}
    if err := json.Unmarshal(thing, &place); err != nil {
        fmt.Println(err)
    } else {
        if place != *new(Place) {
            places = append(places, place)
        }
    }

    return places
}

Here's the full gist:

Using vim's path to speed up your Go project

If you're working on a project in Go, and you're using vim, then you really need to check out path:

:help path

Here's a very quick tour.

Add this to you .vimrc:

set path +=/your/projects/gopath/src/**

** tells path to include all subdirectories. By adding your projects GOPATH entry it opens up access to two rather cool features that are useful in navigating your project.

:find

You can use :find to open a file instead of using :e. But :find will look in all of your project's directories to find the file. So rather than having to do:

:e /package/v10/another-dir/god.go

You just need:

:find god.go

You also get tab-completion for free - just in case you can't remember the file's name.

gf

gf is the useful little command that goes to the file under the cursor. Well providing you've set your path correcly you can navigate your project's imports with ease:

package main

import (
 "code.google.com/p/go.net/websocket"
)

By "gf"ing over the import vim will open the directory, giving you the list of files in that package.

Just don't dare navigate the directory using the arrow keys!

Update: 8th Jan 2012

On the subject of vim and Go it's worth remembering this little gem:

au BufWritePost *.go !gofmt -w %

Whenever a Go file is saved gofmt will be run against it.

go(ing) under

Lots of people like go and this year I began to find it unavoidable - I'm even ending the year with go being the main language that helps pay my bills!

One of the things that go lets you do is pass functions around.
I started the year talking about under from the j language. Which lets you 'bookend' some function with another function (a verb) and a function which undoes that function (obverse).
A great example is Pythagoras' theorem:

1. You perform some 'pre-processing' on each value (squaring)
2. You add all the values up
3. You square root the result

I started the year talking about under in clojure. Here it is implemented in go. Complete with functions being passed around, closures, and functions returning functions. Happy new year!

Running juju locally on 12.04 (or getting over agent-status pending)

I've just tried following the instructions here to get juju running locally on my brand new Ubuntu 12.04 (desktop) system.
No matter what I tried my wordpress and mysql charms seemed stuck in agent-status pending and ip address null. Even waiting a couple of hours.

I needed to take to a couple of extra steps to get it working that didn't seem to be documented in one place. So I thought I'd put them both here.

All it took was to add a rule to ufw:

sudo ufw allow from 192.168.122.0/24 to any

Hope this helps someone who was stuck in the same way I was.