Colin's Journal

Colin's Journal: A place for thoughts about politics, software, and daily life.

August 31st, 2010

Publishing to the Android Market



Two weeks ago I published my first application to the Android Market.  LibraryThingScanner is an extremely simple application that speeds up adding books to LibraryThing.  The app launches the Barcode Scanner application to scan the ISBN, followed by the web browser to bring up the relevant LibraryThing search page.  The app does so little I’ve been in two minds as to whether it was worth publishing at all, especially as the LibraryThing website does not work particularly well on an Android phone (for example continuously popping up the virtual keyboard).

Once I’d coughed up the required $25 for the privilege of publishing to the market I found the process very straight forward.  There are certainly some quirks, such as being able to upload two screen shots or no screen shots, but the level of information required was very low.  It’s peculiar that I find uploading my software to the internet at large, with a potential audience of nearly two billion people, a minor step, whereas publishing onto the market felt like a more significant thing, despite the comparatively small potential audience of a few million.

I’m glad that I did take the plunge.  There was an initial spike in downloads as soon as I published the application, reaching 126 downloads and 97 active installs in a matter of a few hours.  Since then things have levelled off and the application is averaging 17 new active installs per day out of 35 new downloads.  I don’t know how accurate the statistics are for Android market application installs, but they are much more useful than any number of downloads for other software I’ve published online.

Seeing the number of active installs slowly creeping up, seeing the ratings (very slowly) coming in and now receiving my first comment (thankfully positive) makes the feedback loop from users much more satisfying than the very occasional email I receive regarding my other software.  It’ll be interesting to see how things carry on longer term and what kind of reaction anything else I may publish receives.

June 15th, 2010

It’s not fragmentation – it’s progress

There have been a lot of articles recently accusing Google of dropping the ball with Android by creating “fragmentation” within the Android platform.  This references either the number of base O/S versions (currently three versions make up 99.5% of active Android phones) or the fact that HTC, Motorola, Sony and others often put some of their own software on top of Android.

Yorkshire Windmill

Yorkshire Windmill

This trend of complaining about fragmentation has now extended as far as complaining about the iPhone OS (recently re-branded as iOS).  This new complaint is that Apple has also somehow fragmented their platform by introducing new devices with different hardware capabilities, in particular screen resolutions and densities (think iPad versus iPhone 4).

While it makes developer’s lives easier to have a single hardware platform to target, it’s also something that we are not used to.  From the earliest days of home computers there has been a huge variety of hardware and software to contend with.  Today’s desktop landscape is no different – developers need to decide which basic platform (Windows, MacOS, Linux) and what versions (Windows XP, Vista, 7?) of those platforms they are willing to support.

The development of larger and higher resolution screens isn’t fragmentation – it’s progress.  The Android platform provides a set of easy to use mechanisms that mostly make the extra size and screen density transparent to the developer.  Similarly the SDK makes it easy to know when you are using a feature that does not exist on earlier versions of the platform.  You can then either make it optional, or if you truly need such a feature, drop support for older phones and be glad that Google’s rapid pace of development makes your application possible at all.

When considering the mobile application environment today I think there are far more pressing issues than additional phone screen sizes to be concerned about.  The 30% cut that Apple and Google take from every application sold, Apple’s active censorship of artists and arbitrary banning of applications are far bigger and more pressing issues.

June 6th, 2010

Android, the beginning

In some respects I’m rather late to the Android / smart phone game. Many of my colleagues carry an iPhone, with a small (but growing) number carrying an Android phone some sort. After the first few weeks of using the HTC desire I can now say that I would struggle to go back to a normal phone.

The killer app for me is not the broad selection of applications, many of which fail to live up to their initial promise, but rather those few applications that keep me in better touch with the world. From news applications for the New York Times, Evening Standard and Guardian through Facebook, Twitter and email it’s much easier to keep up with the world as well as friends. When sharing a photo with the world is two taps away the barrier to doing so is hugely reduced in comparison to the desktop experience of going home, plugging in the camera, finding the right photo and finally uploading it.

The software on the desire has a few glitches on occasion, but for such a young platform with such grand ambitions, it is very usable and achieves a great deal. For example the automatic synchronisation of phone numbers from Facebook into the phone provides a huge amount of value. The amount of alternative software available for core functionality is less surprising than how well it integrates with the overall platform. It’s only rarely (and usually from HTC’s customization) that new software fails to fully integrate with existing apps.

In summary I’m extremely pleased to have caught up with the world and joined the throngs of start phone toting individuals.

January 7th, 2010

Enterprise Systems versus the Web

Leaf in a frozen puddle.  Taken in the Petit Jean Park, Arkansas, USA.

Leaf in a frozen puddle. Taken in the Petit Jean Park, Arkansas, USA.

Tim Bray’s assertion that IT professionals are implementing Enterprise Systems wrongly is a fairly seductive argument at first.  The one sentence summary of his argument is:

The community of developers whose work you see on the Web, who probably don’t know what ADO or UML or JPA even stand for, deploy better systems at less cost in less time at lower risk than we see in the Enterprise.

He goes on to list companies such as Facebook, Google, Twitter, etc as examples of doing it the right way.

I don’t agree with his analysis.  As commenters to the post were quick to point out, comparing a list of failed enterprise projects (including ones such as the NHS National Programme for IT that stretches beyond software solutions) to those few internet firms that have thrived is not instructive.  For every Twitter (rumoured to be barely making a profit) there are hundreds of failed internet start-ups.  IT department do not have the luxury to fail projects as frequently as start-ups fold.  While governance of enterprise projects is often slow and cumbersome, it’s designed to improve the odds of success above this base level.

The assertion that UML and other enterprise IT tools and techniques are not used by internet firms is unsubstantiated.  A quick search reveals that Google, Amazon and eBay have the enterprise role of “architect” among their job descriptions.  I suspect all of the mature internet firms have similar roles, and use many of the same enterprise techniques as more traditional institutions.

As Erik Engbrecht points out, comparing the legacy-system ridden architecture of a typical enterprise to the “internet app is our company” environment of Twitter is also not useful.  In many cases the quick dirty approach used to develop internet applications is the technique that creates the spaghetti architecture found within many enterprises.

There are undoubtedly lessons from internet companies that could be learnt by enterprise system professionals.  Many of the technologies used in the enterprise are overly complex in comparison to internet based solutions.  For example the poor use of SOA while integrating systems brings additional cost with limited benefits.  However such learning is not a revolution, and will not reduce the complexity of implementing enterprise systems substantially.

November 25th, 2009

Initial Impression of Google Wave



I’ve had a Google Wave account for a short while now, but (like many users I’m sure) the slow invite system resulted in me having very few people to test it with.  Last night I had my first real-time three way conversation, and it really highlighted the short-comings of the current user experience.

The most irritating issue with Wave’s current UI is that it’s slow as molasses when lots of things are going on.  Scrolling fails to work properly, clicks don’t register when they should, and you get false clicks as the screen re-draws under your cursor.  This is in an up-to-date version of Firefox (3.5.5), on a fast laptop and with a fast broadband connection.  I think a native client rather than a web app would be a significant improvement in this respect, though I doubt we’ll see one from Google.

The next challenge is fundamental to the way that Wave works.  Because any user can start a new response to any part of the wave, or edit any existing part of the wave, it quickly becomes difficult to track what is happening.  I admit that we were messing around, but the volume of text we had was relatively low, and it still became tedious tracking down which edits you’d read and which were new.

So my first impressions were not overly positive.  Having said that there are a number of features that Wave has that really do set it apart from email & IM:

  • With the Wave hosted on a server rather than on clients, adding a new person to the wave takes effect immediately.  The number of “looping x” emails I receive show there’s a real issue with the way we manage who is included on an email chain.
  • Real time typing makes the conversation more “alive”.  It helps drive home the fact that you are communicating with a person as you can see their thoughts happening in real time.  As an enhancement to IM this is great.  As most good email is reviewed and edited multiple times before sending it’s not going to be useful a lot of the time.
  • Breaking conversations up so that you can reply inline to a specific part.  This is fairly powerful and should make complex exchanges much easier to deal with.  The GUIs approach for marking bits as unread is clunky, but is still an improvement on email.

While I really like the principal behind Google Wave I’m yet to be convinced that the current implementation will be good enough to replace email and instance messaging.

September 10th, 2009

Fast Python

Wigan Pier in reflection

Wigan Pier in reflection

Python is my language of choice, something that I’ve used for a number of websites, applications and tools.  It’s speed of execution has never been an issue for me, even when using it to wrangle large XML files into something more useful.

Despite this, I’m happy to see the existence of the Unladen Swallow project.  Whether this tiny team of two Google engineers will really be able to make Python 5 times faster remains to be seen, but the attempt will surely be worthwhile.  Already improvements have been made, and brought back into the mainline of Python, and the approach being taken seems solid.

The project’s plan is both practical and well thought out.  There have been several attempts to re-implement or speed up Python before (PyPy, Pysco, IronPython, Jython), none of which have come close to challenging CPython in terms of adoption.  By accepting that CPython is the common implementation of choice, and concentrating on making it better, the project’s benefits should be widely felt.  Similarly, planning to gain the performance improvements through application of the relevant academic research, rather than trying to discover something new, should ensure a measure of success in a short period of time.

The weather is still mostly good, but with a chill in the evenings we are starting to use our wood burning stove in anger.  So far it’s proving a great investment, both enjoyable to use and very effective.

August 15th, 2009

Porting SimpleTAL to Python 3

I have finished porting SimpleTAL to Python 3.  Release 5.0 of SimpleTAL is for Python 3.1 and provides similar functionality as SimpleTAL 4.2 does for Python 2.5.  The differences between using 4.2 and 5.0 are documented on the SimpleTAL notes page.

Ivy and grave through a church window

Ivy and grave through a church window

At first the porting process was fairly easy.  I started by getting all test cases to run cleanly under Python 2.6 with the -3 flag, and then ran 2to3 to convert the basic syntax.  The next step was to run the test cases under Python 3 to highlight issues that required manual changes.  Sgmllib has been removed from the standard library, so I had to remove HTMLStructureCleaner from simpleTALUtils (it was unused within the library itself).  The Iterator protocol change from “next” to “__next__” meant my iterator detecting code had to be updated.

The changes to character set handling in Python 3 introduced slightly more complex changes for the template handling.  In Python 2.x the SimpleTAL library would handle all encoding / decoding itself, but in Python 3 this is not always required as there is now a clean separation between bytes and strings.

One issue that I hit when porting to Python 3 was the use of regular expressions.  In order for SimpleTAL to pass through singleton XML elements from the template (i.e. <tag /> rather than <tag></tag>) it needs to carry out a regex check against the raw XML that the SAX library provides.  This is done by retrieving the xml.sax.handler.property_xml_string property, which is documented as returning a string.  In practise however the Python 3 SAX implementation returns bytes, which at first I assumed the regex library would not work with.  A little bit of research later, and I learned that the regex library can work on bytes as well.

One final surprise was the huge performance gain moving from Python 2.6 to 3.1.  The SimpleTAL performance tests show a minimum speed increase of 60% (on the METAL test), with some tests clocking in 90% increases.  Both HTML and XML basic template expansions are now hitting over 1600 pages/sec on a single 1.7GHz CPU.

Copyright 2015 Colin Stewart

Email: colin at