String equality, identity and interning in Python

In a list of things I should have already known comes this. The difference between using 'is' and == on strings in Python.

Let's look at two strings. One unicode (u"unicode string") and one not "not unicode string".

Python 2.7.2+ (default, Oct  4 2011, 20:03:08) 
>>> type("foo")
type 'str'
>>> type(u"foo")
type "unicode"
>>> u"foo" == "foo"
True
>>> u"foo" is "foo"
False

So using == shows the two strings as equal, and 'is' doesn't. What's going on here?

Python interns its strings. Which means only one copy of each distinct string is stored. You can see this by using the built-in function id() to see the identity of our strings.

>>> a = "foo"
>>> b = "foo"
>>> c = u"foo"
>>> print id(a)
3074129864
>>> print id(b)
3074129864
>>> print id(c)
3074128400

You can see our normal strings have the same id because they are the same object. Our unicode string has a different id to our two 'normal' strings. Using the == operator asks python to compare equality of our two strings. Using 'is' compares the identity. As our unicode and normal string are different objects, comparing with 'is' returns false.

I wonder how many of us are guilty of misusing 'is' on strings?

Heroku and Ubuntu

So, I've just spent an hour or two trying to make a new heroku app on my ubuntu machine, and I was getting nowhere.

$git push heroku master
Agent admitted failure to sign using the key.
Permission denied (publickey).
fatal: The remote end hung up unexpectedly

I'd followed all the usual help on stackoverflow and heroku's excellent help section but was making no progress. Until I discovered this. The link to the bug report didn't work for me, but it's clear that I needed to set this SSH_AUTH_SOCK=0 environment variable first:

$export SSH_AUTH_SOCK=0
$git push heroku master

And now we're off!

The Sieve of Eratosthenes in Python

Whilst working on the 10 Io one liners to impress your friends post I felt I needed to turn to python to complete number 10 - the Sieve of Eratosthenes. My intention was to understand it in python as best I could, then simplify the python code until I had one line I could try to translate into Io. This post is about that attempt.
We start off by trying to translate the description in the wikipedia article into python line by line. Which gives us the following.

def esieve(n):
    primes = range(2, n+1)
    p = 2
    while p < n:
        for i in range(p, n+1):
            if p*i in primes:
                primes.remove(p*i)
        p += 1
    return primes

9 Lines isn't bad for a start, but that if statement can be cleaned up. What if instead of looking for items in our list one at a time. We make a new list of items to be removed, and remove them all at the end? We could use a set to hold our lists. This lets us use the minus (-) operator to give us a new set of items not in our marked set.

def shorter_esieve(n):
    marked = set()
    p = 2
    while p < n:
        for i in range(p, n+1):
            marked.add(p*i)
        p += 1
    return sorted(set(range(2, n+1)) - marked)

We only removed one line in that last attempt. Not great. But it looks like we're using a while loop and incrementing each step. Why don't we just do a for?

def shorter_esieve(n):
    marked = set()
    for p in range(2, n+1):
        for i in range(p, n+1):
            marked.add(p*i)
    return sorted(set(range(2, n+1)) - marked)

6 lines, getting better. Now here is the magic. We're using two for loops to generate a set of values. So we can just use a list comprehension to build our list, which we then use to make our marked set.

def shorter_esieve(n):
    marked = set([p* i for p in range(2, n+1) for i in range(p, n+1)])
    return sorted(set(range(2, n+1)) - marked)

And moving the assignment inline

def much_shorter_esieve(n):
    return sorted(set(range(2, n+1)) - set([p*i for p in range(2, n+1) for i in range(p, n+1)]))

And there we have it. The Sieve of Eratosthenes in one line of python. If you'd rather watch the refactoring happening step by step. Here's a video, set to suitable music.

10 Io one liners to impress your friends

It's been ages since I've done anything with the Io language. But after seeing this spate of 10 [language] one liners to impress your friends in Davide Varvello's post I thought I would spend the evening trying it with Io. Here's the gist: According to Davide's post the others in this category so far are:
Ruby
Scala
CoffeeScript
Haskell
Clojure
Python
Groovy
I'd love to see more. It's a great little activity to get started with a language.

Chicken Scheme on OS X 10.7 (Lion)

I'm making another attempt at going through SICP doing all of the exercises. I've decided that chicken scheme is going to be the way I will do it. But to get it going on Lion there are some hoops to jump through thanks largely to a new version of gcc.

The method I have found that works is.

1) Download the source from the chicken scheme website.

2) Following the advice here you should pass new compiler options to make:

make C_COMPILER=gcc-4.2 PLATFORM=macosx

3) To install the additional eggs you will need to pass new options to csc like so:

CSC_OPTIONS='-cc gcc-4.2' chicken-install

Software Craftsmanship & The Second Chasm

I've just been reading William Pietri's post Agile's second chasm in which he discusses why ideas, such as agile get watered down as they get taken up by the mainstream. I think there's some stuff hear that applies to the craftsmanship movement. You could probably argue whether or not the craftsmanship movement has passed the first chasm yet, But for the moment lets suppose that it has. What are the chances that we might fall into the second one that William describes in his post?

A Common theme

First off, both ideas have a manifesto - this in itself isn't a bad thing, manifestos are great at getting like minded people talking. The problem is it allows people to 'sign up' to an idea that they have no intention of fulfilling. I've seen CVs that boast about being signatories of the agile manifesto, but have never been part of an agile team. I've been invited to "daily scrum" meetings that were 2-hour-sit-down-and-go-through-spreadsheets-and-gantt-charts meetings. It seems to me that Craftsmanship also has the same "difficult to test" quality to it. Craftsmanship isn't so much a title - as a way of doing things. So you won't really know if someone is a craftsman unless you watch them doing their job over time. I wonder how long it is before I see a CV with a "Software Craftsman" certification, but is willing to implement a mess in order to meet a schedule?

A Difference

The way I see it, there is one important difference with the craftsmanship movement. Its focus is on individuals and their interactions, not teams. Ok, I know words like 'partnership' and 'community' are used in the manifesto, But the point is that it's up to the individual to better themselves and the code they work on, productive customer partnerships are something the individual should seek because it helps them better understand what it is they are being asked to do. The community is all about individuals meeting other individuals to help themselves improve. So how do I think this helps? Because it operates at a completely different level.

Agile has continuous improvement built into it, and of course, if you're doing it properly you will always change and adapt. But it has a core set of practices that are easy to get started with that allow you to call yourself an agile team. But Craftsmanship doesn't have this (at the moment). Try to do a quick google search for what software craftsmanship is, and you'll struggle for a firm definition. The best I've found so far is Uncle Bob's but even that doesn't have any specifics. Really, the only idea behind software craftsmanship is: "We will write the best code we can by constantly reviewing how to write the best code we can". It's a continuous learning process (one that is built into the medical profession by law). Continuous improvement on a personal level is not something that's so easy to fake

Python Shelving

I discovered the cool shelve module earlier. Sometimes it's nice to be able to store python objects using the pickle module. The shelve module builds on this by providing a dictionary that you can put objects into and access later. Check out this gist

Dojos and Katas

I recently got back from a very successful Socrates 2011 (Software craftsmanship and testing camp) where I spent just over 2 days hanging around with a bunch of smart people and spent the whole time either writing code, or talking about code. I was involved in a lot of discussions about software craftsmanship as well as a few coding dojos. Dojos and Katas were always something I've done when I've felt like it, but never as part of a habit. Following the conference I started using Stefan Roock's great site codersdojo. It allows you to work on a kata, and upload it to the site when you're done. It provides statistics about how long each step took as well as when the tests were passing or failing:

• FizzBuzz Python
• FizzBuzz Clojure
• FizzBuzz Haskell

Thanks to Stefan for a great site and helping me add Haskell Support! Take a look at my haskell fizzbuzz kata below and please checkout Stefan's site.

Touch Typing Part 2

Well, week 2 of learning to touch type is over. I'm in the region of 50 wpm. Which is almost where I wanted to be (just 10 wpm short). I've spent most of this week trying to improve accuracy, I can recommend GNU Typist which requires you to be 97% accurate to move on with the exercises. As a summary I would say it's only been two weeks and I'm just about at the speed I was at before I started, except now I don't need to look at the keyboard. I can definitely recommend giving touch typing a try if you haven't already.

Touch Typing Part 1

So, this the the first blog post I can claim to have touch typed. I realised about a week ago that while I was typing I was spending a large amount of time looking at the keyboard. So I decided I would fix it by learning to touch type. I'm getting a lot of support from the guys on the Software Craftmanship linkedin group. Here is a diary of my progress so far.

The starting point
My current method of typing involves looking at the keyboard while I type and using my index fingers (and occasionally the other fingers for familiar patterns) I average about 55-60wpm doing this.

The aim
Get to 60 wpm without needing to look at the keyboard (Although it would be nice if I could go faster than 60wpm).

Sunday
Spent the day looking at some websites to see what would help. I read Nick Morgan's blog post and felt inspired. By the end of the day I was used to home row.

Monday
Signed up to Typingweb and Typeracer. I can get to about 16wpm but occasionally I need to force my fingers back onto home row. When I try to go fast they default back to old habits.

Tuesday
By the afternoon my fingers were defaulting to home row, took a test on typing web and got up to 24wpm. Played typeracer in the evening and got up to 26wpm.

Wednesday
Did 5 races before work. Pushed my average up to 28 but the competitive element pushed me into bad habits. By the evening my fingers were feeling more comfortable sitting on home row than it did on monday. Feels like all I need to do now is keep at it and hopefully the speed will come.

Thursday
Felt myself falling back into old habits during the day. I reached 40wpm on typeracer but trying to go above that forces my fingers off home row.

Friday
Spent the whole day at work coding while touch typing. I wasn't typing very quick. But I hardly looked at the keyboard at all.

So that's it so far. I'm ending the week typing about half the speed I was going at the start of it (34wpm) but I no longer have to look at the keyboard while I'm doing it. So it's a step in the right direction. Now all I need to do is stick at it and hopefully I will start getting faster.

Thanks to Pascal, Corey and the rest of the guys on the Software Craftsmanship group for their support!

Subtitles

There have been a few occasions recently where I have been watching non english films with my partner and the subtitles have been so bad, it's not worth watching. Each time I have been able to find a perfectly acceptable SRT file that has decent subtitles. But if I'm watching on a DVD player this doesn't help. So, today, bored, I decided I would write a program that would parse an srt file and display the subtitle for me, it's a simple console app (in python at the moment but I plan to write another in C++ (and maybe one for my phone!)) which means, as long as I find the srt beforehand and start my app at the same time I start the film (or start watching on tv) I have pretty decent subtitles. Here's a short video of it in action.

Arduino Pager with Google App Engine Part 2

I've now updated the web app and the arduino to only show unread messages. Pressing a button on the arduino marks a message as read and grabs the next unread one.

Arduino Pager with Google App Engine

Today I decided to turn my arduino into a pager (albeit a not very portable one). The project had two parts.

1) A Google App Engine application that would store messages submitted via a HTML textbox and provide a way of querying these messages.

2) An arduino sketch that would make requests of the web app and display the results on an LCD screen. Here's the result:

http://mattywpager.appspot.com/

The hardest part was working out how an arduino can query this app using the shared ip address of appspot.com. This was acheived by using wireshark to see what my browser was doing. The resulting sketch simply sends this to the client

    client.println("GET /last HTTP/1.1");
    client.println("HOST: myappname.appspot.com");
    client.println("Connection: keep-alive");
    client.println("Cache-Control: max-age=0");
    client.println("Accept-Language:en-us,en;q=0.5");
    client.println("Accept-Encoding:gzip,deflate");
    client.println();

The whole thing took about 4 hours. But was great fun.

Agile in a Flash

Got Agile in a Flash the other day. At the end of our standups we pick a card at random and read it out. Any ideas we find relevant to our current situation we try to act on that day.

Hello, Screencast!

Simple Arduino Thingspeak Tutorial from Matthew Williams on Vimeo.

2010: A retrospective

As someone who considers himself an agile developer It seems only fitting that I do some kind of look back at the last year to see what went well and what I need to do better next time. In 2010 my big achievements were.

1) Android App published
2) iPhone App Published
3) Monthly blogging
4) Tracking software on a git repository

1 & 2 were done to provide some software that potential employees could download and try out as an example of my ability (also, they were fun to do). 3 was done to try to show to the world my interest and passion for what I do.4 was done because up until 2010 I only had my code saved on a hard drive which made swapping between laptops tedious. The last year has certainly been fun, and I've learned a lot of stuff.

For the next 12 months I am going to carrying on investigating functional programming. I still haven't got my head around monads! I have been playing around with the arduino board a lot recently to satisfy my low level programming itch that I can't satisfy at work. I want to try to get involved in an open source project this year as well, it would be good if I could combine this with the arduino work. I've spent the last few months looking, but haven't been able to settle on a a project. It would be nice to do that this year. Then there's also the matter of a couple of articles I have been writing that I want to publish. There's lots of stuff here to fit between work and family. So for the next year I would want to commit to:

1) Publish a blog entry or video that explains monads, teach someone at work how to use them.
2) Contribute to an open source project related to the arduino.
3) Finish one of my articles and submit it to some publishers for consideration to publish.

That seems about the right sort of level. Let's see how it goes