Well, we made it this far! it has been a rollercoaster ride but we finally have the new site up and running. When I first took on the role of Head of Data.gov.uk I knew that we had a challenging task ahead; making the best government data portal around into something even better. I think the amazing team at Data.gov.uk has achieved just that.
Adopting the mantra of 'meaningful experience' we wanted to achieve several things:
- Simplify the process of finding data without taking away the more complex aspects expected by the seasoned users
- Expand the amount of information available around a dataset as to promote better contextualisation
- Make it easy to 'discover' data be it through search or through perusal of publishers or tags
- Make the engagement aspects of the site more comprehensive, dynamic and substantial
- Strengthen the location data presence across the site
- Bring back the emphasis on new developments and innovation around open data technologies
- Make it easier and better for data owners to publish more data
With this in mind we set out to re-rationalize the experience of the site, as a result you will notice a series of new functionalities both in the catalogue and the engagement section.
The catalogue
The catalogue, as many of you know, is running on an instance of Ckan, the open source data cataloguing software. We took the latest version and expanded its features to fit the needs of the large volume of datasets in our catalogue. The new site's catalogue (the data tab in the new menu bar) runs exclusively in Ckan and allows us to take advantage of its advanced catalogue management functions.
Search is now more accurate and the search results page has been redesigned to make it easier both visually and in practice to find data and filter it further. You will notice many small details that will give you far more power to work with your search results.
The dataset record page has been greatly simplified for ease of read and compacted so you can quickly find the information that is important to you. We have also added information on how to contact the data owner, relevant tags, new details such as an openness score based on Tim Bernes-Lee’s 5 start rating for open data, A unique URI and a Json format link to be used with our API.
Each file (resource) in a dataset now has a record page providing further information, if available and allowing, if the file is a CSV or spreadsheet, to preview the contents of the file within the record.
New ways of browsing data
We have added new ways to discover and find data. The publishers option allows you to search through all of our data publishers by alphabetical listing, by autocomplete in a search bar and by a hierarchy tree. The Publisher ‘home page’, a combined search result page, provides you with all the information you need to browse across the whole hierarchy of a given publisher.
The tags browser is simply an alphabetical listing of all of our tags in a way that makes it visually easy.
Mapping data
We have also strengthened the code behind our map search and map preview functionality, with a new preview cart feature embedded throughout the search results and the dataset records.
Other data sources have also been updated, the organogram tool, an RDF based preview feature to display posts and salaries across public services, has been updated and a ‘time’ slider has been added so you can travel back in time to previous organograms for a given institution as well as a function to download different aspects of an organogram in several open formats.
Participate
This new section, encompassing our traditional forum, blogs and ideas, aims to be a more cohesive social engagement area by allowing us to highlight stories, discussions and topics as well as creating new engagement points.
We have also brought in new Location content from the UK Location Project and have extended the linked data section to provide yet more information and to facilitate better interaction in the future.
We now have a library where we can provide you with guidance on using the site, related documents and other useful assets to compliment the site.
These are just a few of the improvements we have made to the site, have a look, do a data search and pay close attention to the options offered to you, you will notice the difference. As always, we are keen to hear from you, so if you can also jump over to forum or drop us a note right here, we’d love to hear from you.
PS - We are not done with the site, we are always making it better, so new functionality and tools to use and view our data will be forthcoming, in the next few weeks we will publish on the site our 'tracker' information so everyone can see what we are working on.
For those interested (thanks to the OKFN for the links):
The Code
All the code for the CKAN part of Data.Gov.UK is open-source and available on github:
Some of the other extensions used:
- CKAN Harvest extension
- CKAN archiver extension
- CKAN CSW extension
- CKAN INSPIRE extension
- CKAN QA extension
- CKAN Spatial extension
- CKAN Social extension
15/2/13 UPDATE: For the latest links to the code, see: http://data.gov.uk/about-technical-details
Antonio Acuna
Head of Data.gov.uk
16 comments
Comment by Anonymous posted on
Congrats on the relaunched site. Looks and works nicely. Just one observation, having downloaded a few of the departmental open data strategies. I clicked the download arrow a few times before realising it hasn't been activated. For some reason you have to click on the horizontal button along side it (eg the one that says PDF 650kb, or whatever). Shouldn't the download arrow be activated too? Anyway, main point is many congrats on the team's work.
(PS An additional observation having just got this message when I tried to post my comment: "The URL of your homepage is not valid. Remember that it must be fully qualified, i.e. of the form
http://example.com/directory." Why? Anyway, if that's the case, shouldn't you mention it up front?)(PPS This is now my third attempt to post a comment. I've been told to remember to complete a word verification test that wasn't offered after I fully qualified my URL.)
Comment by Anonymous posted on
Thanks for the feedback and we are amending this button accordingly Romina
Comment by Anonymous posted on
Hi Romina
Very many thanks. I see you've activated the down arrow now but when I try to use it at http://data.gov.uk/library/open-data-white-paper, I'm met by: 'The page you requested does not exist. For your convenience, a search was performed using the query library sites default files Open data White Paper pdf.' However, the horizontal button alongside it does take me to the right place.
Cheers
t
Comment by Anonymous posted on
Tom - this error has now been fixed - thanks your yoru feedback.
Comment by TSTEEL posted on
The new site is indeed easier to navigate, but it is not perfect. I have had my publication rights removed and have had no response from team@data.go.uk via the web form or via email.
What use is an easier to navigate site if data can not be published?
Comment by AAcuna posted on
TSTEEL, who are you trying to publish for?
Comment by exstat posted on
I don't think that your blog says anything about the forum and ideas sections of the site. Both got seriously polluted by spam. The solution seems to have been to wipe all the comments made. Unfortunately some of those wiped were actually substantive and useful answers or comments.
Mucking out the stables without losing contributions of value was always going to be a difficult task. But are you planning to reinstate some of the contributions at a later stage? I appreciate that it is a lot of work to sort it all out, but if you are not going to do so, please say so. And is it planned to premod comments in the future, to avoiud a repetition of the sea of spam?
Comment by AAcuna posted on
Exstat,
You are right, I did not go into the details of the new participate section, which includes the ideas, forums and blogs sections of the old site. At the moment we do not have plans to restore any comments that were lost during the clean up of the spam issue we experienced.
I understand this may disappoint some of you, but the time and effort required to go back into the archives, identify which of those comments got lost and reinstate them can be better used improving the functionality of the site or the quality of the data provided, specially given that the actual number of lost comments is minimal compared to the sum total of all messages present.
We will revisit the whole commenting structure and its future towards the autumn. We have the services of Molom running in the background of the new site and it appears to be working well for now, so pre-mod is not an option for us, given that we have 8596 points at which comments can be made, plus blogs, forums and ideas; moderation is not practical.
Not ideal for those keen on seeing those comments back, but the only workable position for the team at the moment.
AA
Comment by Peter Parslow posted on
oOne of the useful comments on the old site was a suggestion to help deal with spam: add a "this is spam" button to each post, and delete it if enough users ticked. Of course it isn't entirely proof against malicious use, but I would support it as an idea that could help.
Comment by exstat posted on
I thought my previous comment on this blog hadn't made it, as nothing seemed to happen to make me think it had been. Anyway, I've now seen it now, and your reply.
We now seem to have the strange situation where a question is posed and nobody appears to have answered it, though in reality one or more poeple may have done so before the great despamming.
To be honest, that puts me off bothering to make further comments, because I have no confidence that they won't get wiped in some future exercise. Unfortunate, as comments can be genuinely useful and I hope some of mine were. So well done, all you spammers out there.
Comment by Jacknunn posted on
Great to have this wonderful site online, may it be the start of a new age of openess. In the interests of openess, just wanted to flag a typo 🙂
'an openness score BASED on Tim Bernes-Lee’s 5 STAR rating for open data'
Comment by AAcuna posted on
Corrected, Cheers!!!
Comment by Anonymous posted on
I've only tried the Met data download so this may not apply to other areas.
You've gone to some trouble to ensure that data can only be downloaded through a web browser for one date at a time. The only reason I can think of to do that is to prevent scripted downloads of multiple dates' data. To me that seems the opposite of 'open'.
Comment by AAcuna posted on
Rich,
The Met Office download service is a unique situation for us. In order to be
able to provide the data openly and free, we had to accommodate the fact that
the normal 'dataset' record concept won’t work for a dataset that is actually
composed of 3 files hourly for over 500 points. The sheer amount of
variables; date, location, values, made it necessary to use a more conventional
database structure for the delivery.
What you see there is a very beta version in order to put the data out and ensure is
available to you. I am in the process of looking into a new version of
the functionality. As there is a lot of computational aspects to taking
the 3 hourly files and disaggregating location, time and weather data into a database,
the data is using Azure's managed services to do it all automatically for
us.
A new API for Azure's marketplace is available that will give us far more
flexibility and bring about the current rich discovery features seen in the
Azure marketplace interface. So, give us a bit of time, I plan to have
this brought to its next version this summer.
AA
Comment by Peter Parslow posted on
I have seen a presentation PDF entitled "User Guide: Administrators and Editors" (it has your name in the PDF properties). It's quite useful; could you publish it on the site? Or if it is published, sort out why searching for "User Guide" doesn't find it.
My vested interest is that I am updating the UK Location Programme's "DMS Operational Guide", and it would be much better to refer out to the data.gov.uk guide rather than have to replicate the information (as happens with the current version of our guide, related to the old version of your site).
Thanks
Peter
Comment by Anonymous posted on
Surely, this is a joke!
Transparency in statistics is certainly not what you have offered me. I have to assume that the cumbersome and utterly awkward MESS that this site has become is a deliberate measure.
Who is accountable and answerable for this utter shambles please?