In February 2010 I created a mobile application called ASBOrometer, which I described as a tool that measures levels of anti-social behaviour at your current location. The basis of the app was two datasets made available for reuse on data.gov.uk. The app was able to achieve a viral success and has gone on to be downloaded over 200,000 times. Its development was my first exposure to the government's open data initiative and was a very positive one for me. I would like to share some of my experiences and observations made during the process.
Challenge and opportunity
When browsing through the data.gov.uk site, the first thing that strikes you is the enormity and variety of data that is now available. The second thing that strikes you is just how 'chaotic' it can seem. Different datasets are available in many different formats, some easier to work with than others. Some datasets contain very recent data and are refreshed frequently, others are quite stale. Some contain a very granular level of detail, others just provide high level aggregations.
A lot of this is obviously not ideal from a developer perspective, but totally understandable given the legacy of fragmented government entities and data collection methods. The Semantic Web initiatives, as well as cultural changes within government with respect to data will hopefully address this, but it is not something that can happen overnight.
And here lies the opportunity. I believe data.gov.uk is an untapped goldmine. There is huge potential value in these datasets, but many developers are being put off because they aren't necessarily immediately programatically accessible. Fortunately there are great tools and technology enablers available, many of which are free, that help to solve this. Now is the time to be building unique and interesting applications with this data. My advice is to get in early before the gold-rush begins.
Public appetite
ASBOrometer's success proved to me just how big an appetite there is for apps that use public data. Sometimes it is easy for those of us working in software to get carried away with building things that only appeal to the tech crowd - Twitter mashups, Foursquare clones and endless riffs off the same idea. The truth is that most people do not care about these things, hence they do not achieve success outside of the geek crowd. However, the majority of the general public DO care about house prices, healthcare, crime and public services. Applications using such data have the potential for true mainstream success
Show off
When I was asked by journalists about the reasons for creating ASBOrometer, they were probably looking for an interesting story about how anti-social behaviour had directly affected me and inspired me to develop the app. The simple truth however is that I was looking for work at the time and decided to build something as a means to promote myself and my abilities. It worked.
Content matters
The engineering effort that goes into building software is usually less difficult than solving the problem of generating the actual content, or building up a community of users whose role it is to create it. The data.gov.uk datasets offer developers rich content, available for their applications from day one.
No permission required
Innovation prospers in environments that do not require participants to ask permission to create something. Surprisingly, I get asked a lot about what restrictions were imposed on the use of the anti-social behaviour datasets I used. The answer is "open data" means just that - open and freely available for reuse, commercial or otherwise.
Free and open source tools
The challenge in using the majority of the datasets available today seems to be taking them from whatever existing format they are in, be it CSV, Excel, HTML tables or PDF, and transforming them into something that is more programatically accessible. Fortunately we are living in a golden age for web development, meaning there are a plethora of free or open source development tools available to make this task easier.
In developing ASBOrometer, my programming language of choice was Python. Python's built in csv module made reading from the various spreadsheets and cell separated text files easy. The open source geopy project was used to very quickly script a batch process that transformed names of the various towns and boroughs in the UK contained within the data into latitude and longitude coordinates suitable for querying in geospatial searches.
The backend of the application was built on Google App Engine, a powerful cloud based web application platform that supports Python and Java language runtimes. The open source GeoModel add-in was used to enable the location based proximity searches that are initiated by the mobile clients. App Engine is free to get started with and is built on Google's own cloud architecture, so it will scale as necessary. Though many applications may not need to scale beyond the free quotas, after ASBOrometer was featured in a mainstream newspaper it skyrocketed into the iTunes App Store top 10, eventually peaking at number 1. During this period, around 15 geospatial queries were being performed each second. The flexible scalability of App Engine meant that the application never once missed a beat.
Make it mobile
The decision to make a mobile app was an obvious one for me. The anti-social behaviour data had already been made available on the web by the Home Office on one of their portal sites. Mobile development offers developers two unique and powerful characteristics - the mobile device is always with you and it always knows where you are. Exploit these characteristics in your application to make it truly mobile.
Going mobile does not necessarily mean developing native apps or being anti-web. The fragmented and competitive state of the mobile market unfortunately does mean that developers must choose their target platforms, but geolocation as well as many HTML5 features are already available in the latest iterations of browsers present on the iPhone and Android platforms. Additionally, cross platform development frameworks such as PhoneGap and Titanium have matured quickly and are already a viable and web-developer friendly approach to mobile development.
Make it local
People are interested in what is happening in their local area. A local app can provide a completely unique experience to a user depending on their location. With the recent opening up of Ordinance Survey data (unfortunately not available to me during the development of ASBOrometer) there is even more opportunity to build interesting mashups, maps and visualisations that provide relevance to a particular region or even street.
In the UK, there is very often a sense of competition and rivalry between neighbouring towns or boroughs. By including a simple 'league table' that ranked the worst perceived UK boroughs for anti-social behaviour, ASBOrometer was able to tap into these local rivalries and provoke additional interest/controversy in the process.
Conclusion
The current opportunity for developers in open data represents only the tip of the iceberg. The work that data.gov.uk has achieved so far has set a strong precedent - it puts pressure on all organisations who place commercial restrictions on their data that would otherwise provide public benefit in being open. We have already seen Transport for London remove a raft of restrictions in the use of their data in the last week. Who will be the next to follow?
All I know is that now is a great time to be a developer.
Jeff Gilfelt is a freelance developer who created the ASBOrometer application and is working on
- FestivalMob - A mobile software platform for music, arts and cultural festivals - delivering branded content, schedules, maps and social experiences to iPhone and Android smartphones.
- Location Scout - Mobile application for Android and iPhone
- ASBOrometer - Mobile application for Android and iPhone
- android-mapviewballoons - Simple information balloon annotation for Android MapView
3 comments
Comment by Anonymous posted on
Great stuff Jeff. How useful/easy is the semantic web technology..?
Comment by paulw82 posted on
local apps would be great, especially if it periodically gave me information about up and coming events in my area or possibly local travel issues. Business could also benefit by signing up to promote their products / offers.
with technology moving this fast in relation to smart phones it`s got to be the way to go
Comment by omurphy posted on
Thanks Jeff, love the app. I use it to see what places I should avoid. Hopefully we'll get to see FestivalMob for Glastonbury by next year?