Colin's Journal

Colin's Journal: A place for thoughts about politics, software, and daily life.

April 16th, 2003

It’s been a while since my last posting, almost a week in fact. During this time we’ve experienced summer and the reversion to winter. Some beer has been drunk, conversation partaken and food eaten. As you can tell not much of note has occurred, and so I’ve spared you the pain of reading about it as it happens.

Regarding things that did happen:

  • I read Howl’s Moving Castle upon recommendation. It’s OK, but as children’s books go there are far better (Maurice and the inevitable Potter series).
  • Although there don’t appear to be many levels it looks like Pingus is now worth playing.

April 10th, 2003

A day in the life…

A day of work well done, a sore ear. An afternoon of talking for hours on end, hashing out opinions and theories.

Hello?  Ah John you are there, do you know if Tim is joining us?
Sorry I can't hear you!  Are you on speaker?
How do we avoid the question of xxx coming up?
[xyz has left the call]
What are the customer's expectations?
I don't agree, they don't want that kind of detail...
[xyz has joined the call]
I sent an email out yesterday which described what that will take... 
OK we'll take that offline.  
Dave are you still there?
Sorry I've got another call coming up, send me an email...

April 10th, 2003

LiveJournal now has images in there RSS!

LiveJournal now produces RSS that contains images! This is great news, and something I had been hoping would be added.

The size of the images did bring out a visual bug in my aggregator’s display template/css which I’ve now fixed. I’m going to hold off issuing a new version though until some more time has passed so that I can include other bug fixes.

Live journal changing RSS format from 0.91 to 2.0 also meant all articles currently in the feed were flagged as new, which is unfortunate. There’s no easy way around the problem though, handling it would require schema changes and more complex logic than seems worth while.

April 9th, 2003

Visiting the US

According to these figures there are approximately 15-20% fewer people being inspected crossing into the US since before the 11th of September 2001. The figures don’t go back much further (they currently show seasonally adjusted figures, earlier figures show absolute numbers and so aren’t comparable). I wonder how much of this drop can be attributed to economic woes versus a drop in the desire to either leave (residents returning count in the inspection stats) or enter the US?

Contrast (if not comparison) with the UK is difficult because the immigration situation is more complex. EU nationals entering and leaving the UK are not “inspected” and leave no paper trail (you generally just wave your passport at someone). Travel within the UK’s common area (Isle of Man, etc) also confuse things. Despite this there is an easy to read press release here that covers the number of overseas residents entering the UK. The situation seems to be broadly one of no change:

The number of overseas residents arriving in the UK during the period December 2002 to February 2003, seasonally adjusted, was 16 per cent higher than in the same period a year earlier. However, it should be noted that the end of 2001 and early 2002 were affected by the September 11th terrorist attacks in the USA. Comparing December 2002 to February 2003 with the same period two years earlier (December 2000 to February 2001), there was an increase of one per cent in the number of overseas residents arriving in the UK.

April 9th, 2003

More musing on publish-subscribe

This is a follow on from my thoughts yesterday regarding whether a publish-subscribe model for RSS would be useful.

The more I think about it the less use I see for distributing RSS via a publish-subscribe service, like the one I did an initial specification for yesterday. While it could be considerably more efficient to distribute weblog changes via such a service, I don’t see a compelling reason to make the investment required in software to support such services. The result for the end user is the same as it is today, it just takes less bandwidth.

While the rss feed for this weblog is by far the most requested file on, it’s very low down on the bandwidth usage (less than 2%). From a client perspective the extra overhead of requesting and parsing RSS files versus talking to a subscription server is minimal, and with web based RSS aggregators the overhead disappears entirely (the host of the aggregator takes the load instead).

There is one use case that I can think of however which may benefit from such a publish-subscribe method. Imagine if most web pages had a ‘Watch This Page’ button on them somewhere (similar to the orange XML button). Selecting this button would subscribe the user, through there chosen subscription service, to notifications of changes to that page. Instead of having to remember to check back on pages occasionally for changes you could just click on this button, and you will now be notified if it changes. Would this be useful?

The infrastructure required to deploy such a system isn’t that difficult to roll out. The button would be a link to a file (potentially generated dynamically) which would contain the resourceID for this page, and the URL for the publishing notification server. On selection the browser would download this file, send it to a program installed locally, which in turn sends the details to the user’s chosen subscription notification server.

April 8th, 2003

Thoughts on a publish-subscribe service for RSS

A posting by Dan on the way we use RSS to notify users about updates to weblogs inspired me to consider an alternative. It’s not immediately clear that the polling of a web-server once an hour using conditional GET of an RSS feed is really a problem. As I say in the starting paragraph of my new article on the subject:

I’ve seen it estimated that a conditional HTTP GET on an RSS file takes about 200 bytes of bandwidth. That’s not very much at all, even with a thousand clients polling once per hour the total bandwidth cost in a month will be about 137MB. It’s still worth looking at alternatives though to see whether there is a more efficient way of being notified when a weblog is updated.

Is it worth the development effort required to reduce this kind of load? If we did, would it really work? I’m not convinced yet, but I have taken a stab at describing a web service that could by implemented as an alternative. I did consider existing alternatives like headline distribution in Jabber, but they still rely on polling at the end of the day.

Here’s my proposal, if you are interested in this sort of thing I would appreciate the feedback.

April 7th, 2003

A case study: Sods law

So there I was proudly stating that I had not had any kernel panics with the BeFS module, and how I had recovered my data when guess what happened? Yes, my music stopped, the screen stopped redrawing, and my keyboard did the “flashing all the lights” thing.

I’m not sure exactly what caused this kernel panic, but thankfully I don’t seem to have lost any data. The floppy drive has been on the way out for a while, and when I rebooted it was making a very sickening screeching noise. I’ve unplugged it for now, and I think I’ll get a replacement. So was it the module, the floppy, or something else entirely? Not sure…

April 7th, 2003

Digital Archaeology, or how I rescued my email

Many moons ago (approximately sixty by my reckoning) I bought myself a new computer, and having very carefully selected hardware that was supported, I installed BeOS. It was a fun, fast, life enriching operating system that was blazing the trail to a bright future. It also had very few applications that ran on it, and tended to crash rather a lot, particularly when web browsing.

BeOS came with it’s own disk file system (BFS), it’s own way of handling email (single file per message), it’s own “almost a database” way of organising data, and many other fancy features. As the fortunes of the startup behind the o/s waned, and Be Inc started laying off staff and changing direction, I started looking for an alternative.

I ended up choosing Linux, and found myself on a frustrating, slow, life shortening operating system that had many unfinished applications, and a web browser that tended to crash a lot. Thankfully as time progressed Linux has improved in leaps and bounds, to the point where there are lots of finished applications and browsing the web almost never leads to crashes.

During my migration to Linux I kept my existing BeOS installation to one side, thinking that one day I must really go back and retrieve my data off it. Several hardware upgrades later however, and I found that I couldn’t boot into BeOS anymore. I retrieved the boot CD and floppy from the other side of the Atlantic, and found that I still couldn’t boot into BeOS. So much for my data…

A couple of weekends ago I found a Linux module that handles BFS (or BeFS so as not to be confused with the other BFS that’s out there…) I compiled, installed, and tried to mount my BeOS partition. It worked! No kernel panic, no errors loading the module, just a mounted file system with all my data sat there.

Since then I’ve been going through my old BeOS system and pulling out various parts of it that I would like to keep around. I also discovered somethings that I had forgotten doing, like writing a POP3 client to handle downloading email (written as a work around for a bug in the client that shipped with the system).

Among the old data were all my old emails (a little over five thousand of them), but they were stored in a format that Evolution (my email client) refuses to read. Thankfully the format is very simple (full email text as a single file), and the mbox format that Evolution does understand is equally simple. I’ve written a tiny Python script to convert BeOS Mail to mbox format, and after a few iterations to shake out the bugs it has worked well enough to restore all my old email.

I’ve now got just over 19,000 emails in my system, dating back to June 1998, and hopefully I’ll be able to keep these around for many years to come. Just remember when transitioning systems that you need to move your data over as soon as is possible, because it only gets harder as time goes on…

April 5th, 2003

Free RSS Aggregator Released!

I’ve finished packaging together my RSS Aggregator. It’s at a point where you can use it on an every day basis without hacking code or fiddling with the database.

I’m releasing it on the off chance that someone else might need software that does a similar sort of thing, it would be a shame for two people to have to write it!

If you are curious as to what it looks like, here’s a screen shot of my “recently updated articles” page.

April 4th, 2003

Deploying LAMP – and holding back the flooding

Deploying a LAMP (Linux, Apache, MySQL, Python/Perl) application is difficult. I’ve just put together the briefest description of how to install my web-based, multi-user, RSS aggregation application – and frankly it requires a Unix administrator to do it. I new it would be difficult (I wrote this for myself, I’m just planning on releasing it on the off chance that someone else might want/need a similar thing), but when you finally write a document which describes the steps it’s driven home.

For a start there are eight different software packages that it depends on, although it’s a fair guess that four of them are installed by the distribution of Linux you are using (in theory this is cross platform, but that’s just one complication too far). Then there is database creation, schema creation, basic configuration data setup, the apache configuration, and finally the application configuration. Then you can log-in to the system and start using it…

I see that there is going to be an attempt to stop the worst of the flooding that happens to Venice. I don’t know enough about the politics and plans surrounding this to comment on the significance of this particular announcement, but it does raise a thought. I wonder how many other dynamic flood defences like this exist in Europe? I know about the Thames Barrier, but there are probably others…

Copyright 2015 Colin Stewart

Email: colin at