Colin's Journal

Colin's Journal: A place for thoughts about politics, software, and daily life.

January 23rd, 2016


I’ve found an increasing number of uses for closures in Go, to the point where they are one of my favourite features of the language.  Closures are created when an anonymous function is declared and it references variables in the surrounding function, for example:

func main() {
    a := 0
    b := func() {
        // a is shared between the main and anonymous function
    // Value of a is 1
    fmt.Printf("Value of a is %v\n", a)

Functions are first class types in Go, so a function can return a function that uses the variables and arguments of it’s parent.  Such functions are a good alternative to producing a number of low value traditional struct{} types with a single method.

Closures also help in producing loosely coupled functions that serve as end points for a web service.  Dependency Injection is used to provide the services required by the end point in a clearer way than defining end points as methods on a struct that holds all of the potential dependencies.

Closures are also good for creating iterators / generators, which don’t come up that often, but are a good way to encapsulate the complexity of navigating a complex data structure.  By passing functions to functions that return functions, it’s possible to create simple micro-languages that make filtering and processing data records much easier to think about.

Finally closures enable in-line inversion of control in a very natural way.  I’ve found this particular useful for iterating over SQL queries, significantly reducing boilerplate code and improving consistency of error handling.

While I’m clearly a big fan of using Closures in Go, you do have to be careful about exactly which variables you are capturing in a closure.  For example:

func loop() {
    var funcs []func() int

    for i := 0; i < 5; i++ {
        funcs = append(funcs, func() int {
            return i

    for _, f := range funcs {
        fmt.Printf("Func value: %v\n", f())

The closures being created in this loop all capture the same variable i that is shared between them.  As a result the output of the function is a series of 5s – the value of i at the end of the loop.  If we create a new variable on each iteration of the loop and capture the value there, we get the desired behaviour of printing out 0,1,2,3,4:

func loop() {
    var funcs []func() int

    for i := 0; i < 5; i++ {
        j := i
        funcs = append(funcs, func() int {
            return j

    for _, f := range funcs {
        fmt.Printf("Func value: %v\n", f())


October 4th, 2015

HTTP Push Notfications


Arran Beach

Arran Beach

Owlauth allows users to register a device (e.g. a phone) and then use this device to confirm their identify when logging into a website or application.  For this to work the Owlauth app running on the phone needs to be able to receive push notifications of authentication requests.  There are many centralised ways to push notifications to a phone, but as each domain owner can run their own Owlauth server, a centralised solution isn’t a good fit.

By taking advantage of Go’s cheap handling of concurrent processing with Goroutines, a simple HTTP based push approach can be used.  Clients connect to the server:

  1. Passing a Bearer token that authorizes them as a registered device.
  2. Passing an If-None-Match ETag if it has one.
  3. The HTTP GET then blocks until either:
    1. The HTTP Keep Alive timeout is approaching – in which case the server returns nothing.
    2. A new authentication request is available or an existing request goes away.
  4. When the GET returns, an ETag header reflecting the current state.
  5. The client loops, issuing a new HTTP GET request with the latest ETag.

The server returns before the Keep Alive timeout, which means that the client will reuse the same TCP (and SSL) session as for the first request.  This makes this timeout and GET operation an effective ping that proves the connection is still established.  If the TCP connection becomes invalid, the client’s HTTP library will open a new connection to the server, giving the desired reconnect behaviour.

The server implementation in Go is straightforward:

        var requestedEtag string
        requestedEtag = r.Header.Get("If-None-Match")

        authChannel := reqStore.RegisterListener(dev.DeviceOwner)
        defer reqStore.UnregisterListener(dev.DeviceOwner, authChannel)
        clientGone := w.(http.CloseNotifier).CloseNotify()
        var request *reqdb.PendingRequests
        timeout := time.NewTimer(DeviceRequestGetTimeout)

        for {
            select {
            case req := <-authChannel:
                if req.GetEtag() != requestedEtag {
                    request = &req
                    // Return the ETag
                    w.Header().Add("ETag", req.GetEtag())
                    return request, nil
                // Just loop and try again.
            case <-clientGone:
                // Nothing to be done - just return
                log.Printf("Device gone - returning\n")
                w.Header().Add("ETag", requestedEtag)
                return nil, nil
            case <-timeout.C:
                log.Printf("Device request get timeout reached.\n")
                w.Header().Add("ETag", requestedEtag)
                return nil, nil

The reqStore object keeps track of outstanding authorisation requests and provides the details on a channel to all registered clients that are sat waiting in this GET.

The only other element of the equation on the client side is adopting a suitable retry strategy when http connections are not working.  For desktops that could be just a simple back-off to a few seconds sleep.  For the Android client it needs to taken into account the current network state to avoid excessive battery drain.

I’ve now got a basic version of this working on my phone.  I’ll run it for a while and see how much battery impact it has.

September 8th, 2015

Killing Passwords

Passwords on websites and in apps are the bane of internet usage.  Much has been written (recently in TechCrunch – Kill The Password) on how painful they are to generate and remember.  Password managers help with the challenge, but are  a cumbersome band aid.

When building a website application, adding username and passwords is also painful, requiring extensive work to get working well and securely.  So, what are the alternatives?  There are some good ideas out there, but they tend to be complicated (OpenID Connect), or put large identity providers in a special position (Fido, OpenID Connect) or are centralized and cost money (e.g. Clef).

I think it’s possible to build a simple, distributed, secure authentication mechanism that allows users to login to sites without generating passwords.  The vision is that a user can authenticate themselves easily:

  • Enter your email address
  • See a pass phrase in the application / website
  • Check your phone – if the same pass phrase is displayed, tap authenticate and you are in

I’ve been experimenting with how this could be done, documenting the specification on an Owlauth Github page, and writing an authentication server in Go that implements the specification.  It’s not done yet – the current code sends email to the user with a link rather than a notification to a phone – but it’s close enough that it proves it can work.

In addition to the specification, I’ve also got a test application running (A larder app for tracking food best before dates) that authenticates using this method.

The challenge now is how to move this forward.  Finishing the implementation to allow device based authentication is straightforward (I’ve got most of the code done), but it’s of no use if there isn’t a community of developers interested in deploying it.

May 5th, 2013

SwiftKey and trailing spaces

During testing of the latest version of ExpenseClam (released just last week), I stumbled across a small issue that I’ve not seen before.

Until recently I was using a back port of the Jelly Bean keyboard on my phone. I really like the swiping of characters implementation and find the auto correct to be rather good, although the prediction is rather limited.

Since SwiftKey released their version of character swiping (called SwiftKey Flow), I’ve been using it on my phone, and it’s generally good enough that I’ve not switched back to the Jelly Bean keyboard.

Now on to the issue: trailing spaces. Having swiped across letters to form a word, SwiftKey populates the word, and adds a space ready for entering the next word. Unfortunately this space isn’t just for show, it is populated in the widget value, so when the contents of the field are saved it includes the trailing space.

In general this is mostly harmless, however if you spot the trailing space and delete it, you can end up with two database records that look the same, but are actually different. This makes auto complete on fields less useful and potentially confusing.

The fix is easy enough, I just trim the string before saving to the database, but it would have been better if SwiftKey had used the same approach as Google and inserted the space at the start of a swipe rather than at the end.

April 8th, 2012

Blackberry Playbook

I’ve recently acquired a blackberry playbook by porting ExpenseClam to this platform. The physical format of a 7 inch tablet seems to work very well. It is still small enough that carrying the device around is very easy, and holding it in one hand is entirely comfortable. Meanwhile the screen is just large enough that websites can be viewed in their desktop format, without having to zoom in and scroll.

That’s the good news. The bad news is that the quality of available apps is very low. Most of the big name apps are missing (e.g. no Skype). There are many ported Android apps, but these work only partially. Native Android features are missing and they have a tenancy to hang.

Native apps that I have found suffer from a clunky keyboard implementation, and flaky UI widgets. The Web browser is reasonable, but occasionally suffers from unresponsive UI elements.

My overall impression is that the tablet would be far more useful and interesting if it was running Android ice cream sandwich. This leaves me wondering whether I want to get either a 10 inch tablet our something around the 7 inch mark.

To sum up: when writing this post, I started on the tablet, but the application hung, and I ended up retyping it on my phone. It is hard to see the playbook platform being a success given it’s current state.

January 5th, 2012

App rankings in the Android market

For an app that only recently moved into the 100-500 downloads range, the paid version of ExpenseClam is doing remarkably well. As of writing this, ExpenseClam is number 65 on the list of top grossing apps in the Business category on the Android market.

Looking at the next lower ranked app however I think this is down to how the market treats apps that change categories. Number 66 is an app called SMS super faker, which appears to have always cost the same as ExpenseClam, but is in the 1,000-5,000 download range. According to AppBrain it switched from the Communication to the Business category on the 23rd of December 2010, 6 days before reaching the 1,000  downloads mark. The only explanation I can think of for having a lower gross revenue than ExpenseClam is that it has had far fewer downloads in its new category than previously.

Similarly number 70 (calendar scroll widget agenda) also has far more downloads than ExpenseClam, but has recently changed category.

The lesson to learn from this appears to be that changing category within the market is penalised, at least as far as the top grossing lists are concerned.  Whether this impacts app exposure is another matter. How many apps are purchased as a result of browsing the lower echelons of the top grossing list, versus searching and picking from the featured and staff choices lists is unknown.

January 1st, 2012

Tablet versus NetBook

I’m struggling to decide on whether a NetBook or a tablet would be better suited for me. The tablet would be an Android one, probably the new transformer prime from Asus. It would allow me to test tablet specific designs for ExpenseClam and would provide a more convenient form factor for web browsing than my phone. The transformer also has a keyboard attachment, so writing blog posts and even some web site design should be easy.

A NetBook on the other hand would allow me to do Android development while travelling, and even at home on the sofa, rather than isolating myself away in the study.

In theory I could use VNC, or some other remote desktop software to control my desktop from a tablet, but in practise I expect this to be clunky at best. I could get both a tablet and a NetBook, however I know that when travelling I would only want to carry one, not two machines with me.

At the moment I’m leaning towards the tablet. While ExpenseClam works fine on tablets, it uses the same layout as for phones. I should be able to design a layout that really takes advantage of the extra screen space available on tablets. It also feels like a more modern investment than what is essentially a small laptop.

This post is a bit of an experiment as it had all been written on my phone using the latest WordPress app. Overall not a bad experience, mainly thanks to the Swype keyboard.

September 7th, 2011

Android Market Experience

The Android Market continues to struggle in providing a great end user experience.  To illustrate, here’s what I went through when purchasing Carcassonne tonight:


Dolomites looking desolate

  • I had bookmarked the Carcassonne page on the market a few weeks ago, so when I decided to buy it, I followed the bookmark.  The advertised price was £1.75.
  • I clicked on purchase, and the site prompted me to login, which I did.  The “review your purchase” pop-up appeared, but listed the price as €1.99, and was missing the Confirm button.
  • Deciding that the phone version may work better, I searched on my phone for Carcassone, and saw the price listed as £3.49.
  • Thinking that a lower price sounded better, I tried various re-loads of the page in my browser with no luck – the “review your purchase” pop-up grew a “Confirm” button, but it was greyed out and would not work.
  • I tried searching in the web based version of the market for Carcassone.  The result was a listed price of £3.49.
  • Giving it one last go, I followed the search result link, and it brought up the page showing the price as £1.75.
  • Selecting purchase I was shown the confirmation dialog with a price of €1.99, and a working “Confirm” button.
  • Upon clicking Confirm the app started downloading on my phone – and the website showed a confirmation that I had purchased the game for £1.75.
  • Finally my Google Checkout receipt arrived, showing the actual transaction cost €1.99

It’s fairly amazing that my own app (ExpenseClam) receives any purchases at all given the difficulty involved!

I should add that it was worth persevering – the game is rather good!

June 20th, 2011

ExpenseClam – finding the currency

Stair case

Stair case

Yesterday I released ExpeneseClam, a business expenses tracking app for Android.  Recording expenses on a smartphone can be a painful experience, typos are common, expense claims are not always made in chronological order, and there is usually little time to note down how much a taxi cost.  ExpenseClam has a number of features to help make this easier.  Simple things like auto-suggesting expense descriptions based on the amount entered can make it much quicker to record common recurring expenses.

One of the niftiest features in ExpenseClam is something that you will only stumble across if you travel a lot: it auto selects the default currency for a new expense based on the country you are in.  This is done without needing user permission to determine location, carry out reverse geocoding or even internet access.

The secret sauce is Android’s TelephonyManager and it’s method getNetworkCountryIso().  This method returns the ISO code given by the mobile operator for the network that you are connected to (the Mobile Country Code).  From this a Locale object can be created, which then allows the Currency instance to be created.  None of this requires network traffic and it takes very little processing power to complete.

Changes in the default currency only happens when the country changes.  This is to avoid frustrating users who have to enter expenses in a different currency to this default.  The country derived currency is however prioritised in the list of currencies to make selection easier for what is likely to be a common choice.

February 13th, 2011

Android a challenge to OEMs?

Does the “Android Monopoly” really provide a challenge to the traditional OEMs? Is the answer to use virtualisation on the handset and put Android in a virtual machine sandbox while the OEM’s own O/S runs underneath it?  That’s Andreas Constantinou’s proposal in his latest blog post at VisionMobile. I’m not so sure.  Firstly let’s take a brief look at the history of the OEMs he identifies as losing out due to the success of Android: Motorola and Sony Ericsson.

Toronto sunset at the beaches

Toronto sunset at the beaches

Motorola’s success story pre-Android was the Razr.  This was a huge phone in 2005 and dragged Motorola into second place behind Nokia.  The success of the Razr had little to do with the software that it ran, and a lot to do with the very distinctive and stylish design. Motorola failed to innovate effectively off the back of this however, resulting in a whole series of similar phones and declining market share.  It is only with their adoption of Android that Motorola has been able to rescue themselves from terminal decline in the mobile market.  The Droid series of phones built for Verizon have been very successful for them, bringing the company (now split off as Motorola Mobility) back from the brink.

Sony Ericsson’s history contains more highlights than Motorola’s, but is also a mixed record.  They have used a number of operating systems over the years, with Symbian (using the UIQ interface) and Windows Mobile both playing a part.  Sony Ericsson has been able to differentiate with their Walkman and Cyber-shot brands, both of which relied more on innovation in hardware than software.  Sony Ericsson have produced a range of Android smart phones, which has done much to restore their financial health (to the tune of €1.1bn).

Hardware has always been a major differentiator between mobile OEMs, marking a significant difference between the mobile and personal computing industry.  This traditional strength is not undermined by Google being the driving force behind Android.  OEMs adopting Android get for free the things they have traditionally struggled with: a best-in-class smart phone O/S, app store and cloud services.  This allows them to focus on the part of the equation where they have seen past successes and that aligns to their core competencies.

The announcements coming out of Mobile World Congress this year (and CES before it) shows that there is plenty of room for innovation in hardware and software:

  • Sony Ericsson offering the Xperia Play.  This shows that both hardware innovation (slide out game control pad) can be done alongside corresponding software innovation that sits on-top of Android (the PlayStation Store).
  • Motorola announced the Atrix.  This again shows innovation in both hardware and software, with a docking station that provides a desktop experience when the phone is plugged in.
  • The LG Optimus 3D provides both 3D capture through dual cameras, but also a 3D stereoscopic screen.
  • Samsung are generally sticking with a tried and tested form factor, but are innovating with the screen, the thickness of the phone and the inclusion of NFC in the Galaxy 2.

If Google is restricting what OEMs can do on the hardware and software front, I don’t see much evidence of it in the products that they are announcing, or an opportunity to do more by using virtualisation.

Copyright 2015 Colin Stewart

Email: colin at