The Idea
The inventory of currently unpublished datasets will start to go live on data.gov.uk from 2nd September at 1700 for everyone to see what data the government holds.
This is where it gets interesting. We want to see what you think. So, you will have a chance to assess any of the unpublished items that you are interested in. Whether you are an academic, journalist, interested individual, entrepreneur, small to medium sized enterprise, big business, social enterprise, charity, or anyone else with an interest in government data we want you to tell us what we should be releasing and why.
We have tried to make the process as simple and smooth as possible, whilst still giving you a chance to give us as much detail as you would like to about the data you are interested in. We are looking to find out in which key areas you think the datasets will have the greatest impact, whether it be social, economic, public service efficiency or any other area that you can think of. Your feedback will be published openly on Data.gov.uk and will be used by the Cabinet Office Transparency team as well as to departments to prioritise their future dataset releases.
This exercise is intended to achieve two main goals. The first is to give us an idea of what users are particularly interested in so that we can help to get that data out as speedily as possible. The second goal is to identify those datasets that could have the broadest and most significant impact within the UK. These datasets will be made a part of the National Information Infrastructure as set out in the Government Response to the Shakespeare Review.
The data request process will continue for the time-being. This is in recognition of the fact that this first draft of the inventory will not be absolutely comprehensive and that there is still value in allowing individuals to request datasets that they know exist but which have not yet made it onto the inventory. It is envisaged that, in the longer term, as the inventory grows to encompass all government datasets, the user engagement process will replace the existing data request process.
How do I engage?
From Monday 2nd September 2013 at 1700, you will be able to start to explore unpublished datasets in exactly the same way as you search for any other content on Data.gov.uk. You can search all datasets or filter by whether the items are published or unpublished. You can search through all content, filter by publisher or search using key terms. When you find an unpublished dataset of particular interest, you can then click on it to look at the details: the publisher, a description of its content and notes about its possible release.
From here, you can either view other users’ feedback or select to give your own feedback. In order to take part in this process, you will need to be a user on data.gov.uk. This is very simple and quick. All you need to do is click on the ‘sign up’ link in the top right of your screen on data.gov.uk, fill out a couple of details and validate your email address.
The form itself contains 4 short sections for you to fill. In the first section you will have to decide in which area or areas you think the dataset will have greatest impact. You can select the appropriate box or boxes in which to enter your feedback and let us know how it will impact on you or the UK more broadly if it was released openly. Written content about the areas that you select is optional but will significantly increase our ability to understand and use your feedback and make the case to Departments to release the data as open data.
In the linked data section, we want to see whether you think the data could contain important information or vocabulary that could be referred to from other datasets. In the third section we would like you to let us know whether you are responding as an individual or on behalf of an organisation. In the final section, we would simply like to know if you are happy for us to contact you via email to follow up on your feedback.
Your responses will be published on data.gov.uk so that other users can see all of your input. The Cabinet Office Transparency Team, Departments, Open Data User Group and the Public Sector Transparency Board will start the process of assessing and prioritising data releases and assembling the National Information Infrastructure from 16th September. As such, the earlier you can give the feedback, the more likely it is to be taken into account. You will still be able to feedback after this date and departments will continue to see your feedback. However, in order to ensure that the contents of the initial iteration of the National Information Infrastructure are confirmed before the Open Government Partnership conference in October, we will need to start assembling it from mid-September.
You are also invited to a launch event on the 10th September. It will be hosted by the Cabinet Office Transparency Team at 1 Horse Guards Road, SW1A 2HQ.
I look forward to seeing as many people as possible engaging in this exciting process!
18 comments
Comment by jwilliams posted on
Could you explain a little more about the definition of "data set"? ie, what have government departments been asked to declare as an unpublished dataset?
Comment by Mattlloyd31 posted on
Please refer to this blog which discusses the government response to the Shakespeare Review. This should give you a good background about the rationale behind this exercise.
Matt
Comment by Owen Boswarva posted on
Following is the definition from the Cabinet Office's guidance to departments on developing the NII:
The definition of a dataset in the inventory will follow that used in the Datasets Code of Practice i.e.: information comprising a collection of information held in electronic form where all or most of the information in the collection
a. has been obtained or recorded for the purpose of providing a public authority with information in connection with the provision of a service by the authority or the carrying out of any other function of the authority,
b. is factual information which
i. is not the product of analysis or interpretation other than calculation, and
ii. is not an official statistic (within the meaning given by section 6(1) of the Statistics and Registration Service Act 2007), and
iii. remains presented in a way that (except for the purpose of forming part of the collection) has not been organised, adapted or otherwise materially altered since it was obtained or recorded
-- Owen Boswarva, 02/09/2013
Comment by tom_smith posted on
Hi Matt
Looks very interesting. Is this now live? If so, is there a link to see all the unpublished datasets on the system?
Thanks, Tom.
Comment by davidread posted on
Yes it's live. Searches now include unpublished ones by default. To browse, click on 'Data', 'Search', 'Unpublished datasets'. The direct link is: http://data.gov.uk/data/search?q=&unpublished=true
Comment by Mattlloyd31 posted on
Hi Tom,
Thanks for your interest. If you click here, you can search through all of the new, unpublished items.
Matt
Comment by Owen Boswarva posted on
Matt, can you clarify exactly what Cabinet Office means by an "unpublished" dataset? Are those simply datasets not previously recorded on DataGovUK?
If so, isn't that rather random for purposes of seeking feedback under the NII initiative?
My concern is that the new functionality seeks feedback on some datasets that are currently available for re-use under licence but not as open data (e.g. this one), but does not seek feedback on other datasets with an identical status (e.g. this one).
If the only difference is when the dataset was listed, that doesn't seem reasonable. Or have I misunderstood something?
-- Owen Boswarva, 02/09/2013
Comment by AAcuna posted on
Owen,
We are pushing the feedback functionality to all datasets in DGUK in the next few days as a second iteration of the project.
Cheers,
AA
Comment by James C posted on
Ed
Seems this is a new definition of "unpublished" or at least an extension/change to the acknowkledged defintiion. As Owen requested I think a clear explanation of your definition and the reason for it is required.
James
Comment by Owen Boswarva posted on
I've been looking at more of the dataset records and at the guidance given to departments. I'm pretty sure "published" in this context means "published on Data.gov.uk". That's an odd notion in itself since Data.gov.uk normally only catalogues datasets. By extension "unpublished" must simply mean those datasets for which the data holder has not yet filled out the usual Data.gov.uk pro forma.
This is all rather solipsistic and not particularly helpful to those of us who give words their normal meaning. For purposes of determining whether datasets are available for re-use (or even accessible to the public), there is no clear difference between the "published" and "unpublished" collections. On Data.gov.uk "published" doesn't necessarily mean a dataset has been released by the data holder and "unpublished" doesn't necessarily mean it has not. Weird.
-- Owen Boswarva, 08/09/2013
Comment by Owen Boswarva posted on
Based on the explanations we had at the Cabinet Office Q&A yesterday I think my understanding is correct, i.e. that "unpublished" datasets are simply those for which Data.gov.uk does not (yet) have a full metadata record.
If that's the case, I think the Cabinet Office is making an error in directing users specifically towards the "unpublished" datasets for purposes of providing feedback. There are thousands of non-open datasets in the "published" category. The feedback functionality has now been added to those as well, but if users are being pointed towards the "unpublished" datasets then the feedback metrics will skew towards those datasets.
-- Owen Boswarva, 11/09/2013
Comment by stephen anderson posted on
i have just ,joined this site,after trying to get public information.from various gov.uk departments,for a long time,now as i am nearly 64,i am the first to admit,computing,is not my cup of tea,but i do have to say,i find it really sinister,the government spouts,transpraency,yet refuses to even answer ,mail from the public,i ask WHAT-KIND-OF-TRANSPARENCY-IS-THAT.? sorry--but--that--is--anything-but--transparent, ON-THE CONTRARY--IT-IS SUBVERSIVE,AND-FACIST,i have asked for details,of the cost to the british electorate,in setting up--ATOS.?HOW MUCH ARE ATOS,EXECUTIVES BEING PAYED,ARE THEY ON BONUSES,HOW MUCH IN TOTAL?HOW MUCH ARE ADVISORS COSTING?HOW MUCH TO CHANGE FROM DWP.TO P.IP.,NOT DIFFICULT QUESTIONS TO ANSWER,YET THE GOVERNMENT AS REFUSED-TO SPEAK.? SINISTER--VERY--VERY--SINISTER,THEN WE COME TO M.P.s, EXPENSES,MEAL ALLOWANCES,CAR ALLOWANCES,TRAVEL ALLOWANCES,DECORATING ALLOWANCES,FUEL ALLOWANCES,,AGAIN,ALL HIGHLY SECRETIVE,EXTREMLY SINISTER,,THEN WE COME TO MY REQUEST,FOR COSTINGS,TO THE PUBLIC PURSE,OF WARS,IN IRAQ-AFGANISTAN,ETC,AGAIN EXTREMLY SINISTER,,IM SORRY BUT I SPEAK MY MIND,I FIND NOT ONLY ,ARE THE GOVERNMENT NOT TRANSPARENT.THEY ARE BORDERING ON PATHALOGICAL PARANOIA,AND SO ENMESSED IN POWER AND SECRECY,I GENUINELY FEAR,,THERE IS QUITE A BIT OF INSANITY,IN THE GOVERNMENT,AND I BELIEVE,EVERY SINGLE M.P. SHOULD UNDERGO ,PYHSCIATRIC,EVALUAION,EVERY SIX MONTHS,AS YOU ARE AS TRANSPARENT AS ,A UNIVERSAL----BLACK--HOLE, stephen.a. anderson,snr.
Comment by jtownend posted on
This is a really interesting initiative and a useful way of sharing views on the unpublished datasets. However I suspect there may be a lot of people who have an opinion on the opening up on these types of data-sets but might not be familiar with this type of consultation format (ie. outside the open data world - researchers, journalists etc.). It might be worth thinking about canvassing opinion in other ways too (if you're not already).
Comment by Owen Boswarva posted on
At least one government department, Defra, is planning to release their new Open Data Strategy in draft form for public comment. (I'm a non-exec on the Defra Network Transparency Panel.) That will provide an additional opportunity for people to provide feedback on priorities for open data release. I hope some other departments will take a similar approach.
However I certainly agree that Cabinet Office needs to push this discussion out into the wider world, beyond Data.gov.uk, if the Government wants to genuinely engage the full range of potential users of public data assets.
-- Owen Boswarva, 14/09/2013
Comment by Owen Boswarva posted on
For those of us on Twitter there is a hashtag for discussion of the National Information Infrastructure: #UKNII
Comment by Owen Boswarva posted on
Just to make sure I'm not missing anything, am I correct in thinking there's currently no functionality that enables ordinary Data.gov.uk users to see which datasets (and which departments) are getting the most feedback?
I thought the slides shown at the Cabinet Office Q&A last week showed additional functionality, but perhaps that was just for data publishers. I appreciate everyone can view the feedback for individual datasets but it's hard to get an overall picture ...
-- Owen Boswarva, 16/09/2013
Comment by Mattlloyd31 posted on
Owen, thanks for the enquiry. This functionality is being developed and the report will be made available in the coming weeks.
Comment by Owen Boswarva posted on
How complete are the UK Government dataset inventories?
It's unclear at the moment to what extent the departmental inventories provided to Data.gov.uk are complete or have been independently verified. For example have they been checked against the statements of administrative data sources that all departments maintain for purposes of statistical reporting?
Obviously, to the extent that the inventories remain incomplete that will erode the credibility of any attempt to use them to identify the National Information Infrastructure.
I've started a list of notable datasets missing from Data.gov.uk. The list is ad hoc, i.e. I haven't attempted any systematic survey. However I would be interested in hearing from anyone who has noticed other particular omissions.
-- Owen Boswarva, 17/09/2013 (@owenboswarva on Twitter)