The document below (and attached to this blog post) contains draft guidance for Local Authorities on how to comply with the Prime Minister’s call to publish each financial transaction over £500 from January 2011. As the guidance is in draft format, we would welcome any comments, suggestions or questions you may have. You can either post your comments below or contact the Local Data Panel directly. We look forward to hearing your feedback to the draft guidance.
Local Spending Data guidance
This guidance has been produced by the Local Public Data Panel, and builds on the draft guidance published in June this year by the panel, on comments made on that guidance, the Treasury guidance to central government departments on releasing their spending data, and experience from publishing the data, and, crucially, using and analysing it.
It is designed to give councils clear standards for publishing their spending data, setting both minimums that should be achievable by all councils, and gold standards for those councils with the ability to publish to a richer standard.
Because this is a rapidly evolving area, the guidance is expected to be revised in 6 months’ time based on experience, feedback, and at that point the minimum standards will be increased (as will probably the full, gold-standard ones). All councils should publish to the minimum standards within 6 months of the publication of the guidance.
Although this guidance specifically relates to the publishing by local authorities of spending data over £500, there is much to be gained by other local public bodies (health, police, waste authorities) publishing to the same standards where possible.
1. Overriding principles
The purpose of requiring councils to publish their spending data is to enable greater transparency, greater engagement and greater efficiency by allowing inspection by residents, and reuse of the data in innovative applications, mashups, visualisations and analysis.
Given this it is essential that councils publish their spending data (and other data too) with a licence and a format that allows reuse, and on the basis that data should be published unless there is an overriding reason not to (principally to protect vulnerable people and privacy of ordinary citizens). This means that councils should be be asking, what’s the most they can publish, not what’s the least they have to do.
The focus of the guidance is on how, pragmatically, to make the data available quickly rather than seeking to achieve full alignment across every entity. This is consistent with the Sir Tim Berners-Lee’s paper, ‘Putting Government Data Online’. Publishing raw data quickly is an immediate priority, but there are significant benefits to be gained by councils publishing structured, regularly updated data using open standards, and this guidance helps enable this.
There is, however, no reason why, if they have the resources, authorities should not publish in more ways, or in greater detail than in this guidance. For example, some may start publishing invoice data, others publishing the information as linked data (in which the data is semantically described using URIs), and we expect to include the successful practices learnt from such experiments in future versions of this guidance.
2. Licence, Timing & Contacts
Mandatory:
Data should be published in a timely manner and with a licence that allows open reuse, including commercially. We recommend the new data.gov.uk licence.
There should be a nominated contact person, listed on the page linking to the data files, to oversee the publishing of the data, and this person should also be responsible for answering queries to do with the data, and requests for further information. This will not only increase the skill levels within the council and improve relationships with reusers, but also reduce the necessity of going down formal routes (e.g. Freedom of Information requests).
Data should be published monthly, as soon as possible after month end but no later than 30 days after month end. This data is to be published in individual monthly files, and there should be a single web page which contains links to the individual files. Files should be named in a consistent manner with the date included in the file name, and each file name should be unique. You should avoid spaces or characters other than a-z, 0-9, _ and -.
Authorities may wish to publish the data for longer reporting periods e.g. quarterly, year to date, etc. There is no reason why they should not do this, but this should be in addition to the individual monthly files.
The Secretary of State for Communities and Local Government has committed local authorities to publish items of spending over £500 by January 2011. We recommend that where possible data is provided from 1 April 2010. This will add value to the user and demonstrate commitment to complete transparency.
Recommended:
When errors are discovered, or files are changed for other reasons, rather than ‘silently’ changing the file it is recommended to publish a revised and differently named version (together with the original files), e.g. 02_2011_v2, 02_2011_v1.
The spending data should be published as part of a wider open-data initiative, and there should be a dedicated open-data page/section on the council’s website (the recommended URL for this section is www.yourcouncil.gov.uk/opendata), as well as listing the files at data.gov.uk.
3. File Format
Mandatory:
The files are to be published in CSV file format. Microsoft Excel files should be converted to CSV. The CSV file must have precisely one header line with field names exactly as in the example file supplied. Values must be separated by a comma character, with a new line between separating transactions. Text values that contain a comma must have a ‘"’ character at the start and end of the value. This format is the behaviour of Excel when using the “Save As” function selecting CSV as the file type. There should be no comment lines, no ‘total’ lines and no blank lines.
Due to limitations of Microsoft Excel, it is unable to deal with non-Western characters (like Chinese). In cases where you have such payments, an alternative route to creating the CSV files in Unicode format will need to be found. In the interim, councils needing to publish data including non-Western scripts should provide Excel files. Note that ‘£’ signs can also cause problems for the same reason and thus should not be included in the file.
Authorities may wish to publish the data in additional formats as well as the CSV files (e.g. linked data, XML, or PDFs for casual browsers). There is no reason why they should not do this, but this is not a substitute for the CSV files, and PDFs in particular do not allow their contents to be reused. Authorities must take full responsibility for the content of their files.
4. Publication requirements and process
Consistent with producing raw data quickly, the published data should reflect how each individual item was originally recorded in financial systems, including all related information (e.g. supplier, invoice, procurement details).
It’s worth acknowledging that some authorities provide shared service operations, and these should provide the data in the required format to the authority to which they provide services. The entity that owns the data is responsible for its publication. Authorities will need to put arrangements in place with their shared service providers to obtain the necessary data.
Mandatory:
All payments to suppliers for goods and services and grants over £500 (including VAT) should be published, including all individual invoices, grant payments, payments to other public bodies, expense payments or other such transactions. Credit notes should also be published.
Salary payments to staff should not be included, nor should compensation payments made to individuals. Publication of salaries to senior staff will be covered under a separate guidance note. Certain other transactions may be redacted or be exempt from publication. These are covered in more detail later.
There are limited exceptions surrounding personal information that falls under the Data Protection Act, mainly when dealing with payments to vulnerable people (e.g. payments to foster parents). Payments to any redacted data entry should be replaced with the word ‘REDACTED PERSONAL DATA’ or ‘REDACTED COMMERCIAL CONFIDENTIALITY’.
Note that payments to individual consultants, or sole-traders or other supplying individuals should not normally be redacted, and in all cases supplier ids should be published. In the case that this may identify a vulnerable individual, because, for example, the payments are made to someone who happens to be a sole-trader supplying the authority, we would expect there to be different supplier records for the payments relating to the vulnerable person, and if not a new one should be created. There are no circumstances in local authorities where payment amounts and the record of the payment itself should be redacted as a whole.
The Freedom of Information Act should be used as a frame of reference when making these judgements. The credibility of transparency reporting would be seriously undermined if expenditures were subsequently published under FOI. You should bear this in mind when forming your judgements. It should also be noted that the data to be published, will, in most instances not disclose price, and therefore is unlikely to be commercially sensitive. Together with recent rulings by the Information Commissioner (favouring the right to information over ‘Commercial Confidentiality’) this makes it highly unlikely that any local authority transactions should be redacted for this reason.
Where payments relate to several different categories or invoices these should be broken down into their constituent parts.
Recommended:
The minimum threshold of £500 is a requirement, but there is nothing to stop local authorities publishing all spending, and as systems for redacting information relating to the identities of vulnerable people improve it may well be easier not to have any threshold, and this will certainly aid in tying individual transactions to the aggregated figures in the authority’s accounts.
Councils have also been told they are expected to start publishing new contract and procurement information from Jan 2011, and so all information linking procurement and accounts systems should be published, allowing spending to be linked to contracts and tenders. Procurement systems will also typically have much more information about suppliers (e.g. Company Number) than accounts systems and this will make it much easier for users of the data to identify them.
Publishing the supplier list (matching to the supplier ID) and also the authority’s Chart of Accounts will also help users make sense of the data and this should be published.
Because the publication of the data becomes a public view of an authority’s accounts there will be benefit to publishing corrections/journals.
The ‘meta-data’ about the accounts system should be published (including software on which the accounts and related systems run and procedures/associated programs used). This should be published as an accompanying plain text file, or on the web page linking to the data. This meta-data should also include any departures from the standards listed in this guidance.
5. Data content
The content of the published data must match the format set out in Appendix i, below. The data must follow the sequence of columns prescribed, and any formats or presentation conventions as set out in the general commentaries below. An additional fields that an authority wishes to published should conform to the general specifications (e.g. quoting values containing commas), and should be added after the fields listed here.
- Body
- This identifies the local authority to which the data relates, and means that the file is self-describing (i.e. all the information needed is within the file). It should be the URI that represents the local authority at statistics.data.gov.uk. In the case of UK local authorities, this is of the form of http://statistics.data.gov.uk/id/local-authority/XXXX (where XXXX is the ONS SNAC id for the authority). In the case of the GLA (which doesn’t have a SNAC id) the following URI should be used: http://data.ordnancesurvey.co.uk/id/7000000000041441 . You should also add the name of the authority in the Body Name field to aid readability.
- Date
- The date when the transaction took place. The date should ideally be the payment date as recorded in your purchase or general ledger. This is not the date when the supplier receives the funds. The date must be in the UK format DD/MM/YYYY. However, if it is not possible to source the data using the payment date, then you should first look at the possibility of using the date the invoice was input to the system, otherwise you should use the invoice date.
Use of date must ensure that transactions are only disclosed once, and must be consistently applied. If you are not using the payment date, you must note on the meta data the date you are using and why it is not possible to use payment date. You must also advise what parameters will be used for your disclosures (i.e. how is data being selected for inclusion, so that transactions are only disclosed once). - Transaction number
- This should be system transaction number for the payment held by the individual department. This may be useful to the authority, should there be any follow up questions about an individual transaction.
- Amount in sterling
- The amount disclosed should be the amount recorded on the finance system for each individual transaction. To identify the transactions to be disclosed you should identify all transactions above the reporting threshold with reference to the total amount payable inclusive of VAT. Transactions should also be selected gross of income. All amounts published must be in sterling and in pounds and pence. If a single invoice has been coded to multiple expense types, and/or expense areas the value shown should be the amount paid against each individual expense types/expense area combination, even where each entry at this level may be less than £500.
Where possible, recoverable VAT should be excluded from the amount disclosed (i.e. shown on a net basis). Where the recoverable VAT can be excluded from the disclosure it does not need to be separately disclosed, and can be omitted completely. Where removing recoverable VAT reduces the value of the transaction below £500, the transaction should still be disclosed. However, if the source of the data being used cannot separate out recoverable VAT then the gross amount should be used instead. You should state on the web page on which the data is published (or in the accompanying meta-data text file) if you are unable to publish amounts net of VAT. Where VAT is irrecoverable the total transaction amount should be disclosed. - Supplier details
- In order to properly understand where and with whom public money is being spent, it is important to be able to identify the supplier. This is often not possible from the name alone (which may not be the formal name of the company or organisation), and ideally publication would include the Companies House number (or equivalent in the case of foreign companies), charity registration or other unique identifier.
In practice this information often isn’t held by the accounts system, and because of that the guidance is that all authorities should publish the supplier name and supplier id (to distinguish between different suppliers with similar names and to match the same supplier when the name changes or is redacted).
In addition, where the information is held, authorities should publish the supplier VAT number (although there is no exact match between VAT number and organisation, experience has shown this is a very successful way of identifying suppliers), as well as publishing the supplier list separately as open data.
The name of the supplier should be as per your vendor record. If the payment you make to the vendor (e.g. a solicitor) is to enable them to make a payment to a third party, you need only disclose the name of the original vendor not the end beneficiary. If the same supplier appears more than once under different vendor records, this is how they will appear in the published record. Payments to individual suppliers should not need to be aggregated, and each individual transaction must be shown separately. The supplier name should appear in full. - Expense area
- This is the part of the entity that has spent the money. It should represent a meaningful part of your organisation structure or activity. This might be at directorate level, or another appropriate part of your organisation structure, possibly drawn from your cost centre structure. To aid transparency, it would be helpful if the structure you select were capable of being mapped onto your published organisation structure. Where a consolidated invoice is received, you may reflect the original coding of the transaction, rather than the subsequent reallocation to different parts of the organisation. However, you must still designate an appropriate expense area.
- Service categorization
- In order for spending to be comparable between periods and between local authorities it is important that it is clearly and consistently categorized. Depending on the accounts system this may be easy or quite difficult. There are two candidates for categorization – CIPFA’s BVACOP classification and the Proclass procurement classification system (which is widely used in local authority procurement systems).
It is worth noting that these provide different classifications – BVACOP is an accounts classification, and Proclass procurement classification. It’s also important to say that while Proclass is published under an open licence, at time of writing BVACOP is not, and is therefore not suitable for use when publishing open data, although there has been talk that this will change.
The recommendation therefore is that local authorities where possible should publish whichever classification they can (subject to BVACOP being published under a licence compatible with the data.gov.uk licence), and ideally both.
All authorities should however publish a consistent Expense Type (i.e. the Service description), together with the ID identifying the Service (possibly the nominal code on the chart of accounts).
Appendix i: CSV file format
| Column | Field Name | What Is Required | Reason For Inclusion | Additional Information | Inclusion status |
|---|---|---|---|---|---|
| 1 | Body | The statistics.data,gov.uk URI that represents the authority, e.g. Lichfield District Council is http://statistics.data.gov.uk/id/local-authority/41UD | To allow the file to be self-describing | Using a URI to identify the body means that it is uniquely identifiable to data users. The GLA should use the URI http://data.ordnancesurvey.co.uk/id/7000000000041441 | Mandatory |
| 2 | Body name | Name of the authority | Aids readability for casual reading | Mandatory | |
| 3 | Date | The payment date as recorded in departments’ purchase or general ledger. | To identify the date that cost the transaction took place. | If it is not possible to source data using the payment date invoice input date or invoice date may be used The UK date format (DD/MM/YYYY) should be used. Leading zeros should be used where necessary so that the string is precisely 10 characters (e.g. 01/09/2010) | Mandatory |
| 4 | Transaction number | A unique reference number for each individual expenditure transaction | To act as a reference number when dealing with enquiries or FOI requests. | The transaction number used in departments’ own systems may be used. | Mandatory |
| 5 | Invoice Number | The reference number for the invoice if it is recorded on the accounts system | To allow matching to invoices (one payment may be for several invoices, and one invoice may be settled by several payments) | Some councils have been discussing ultimately publishing invoices, and inclusion of this fields allows for an easy transition to this, should they do this. | Desirable |
| 6 | Amount | The actual value of the transaction | To identify the full cost of the transaction. | Amounts should be in sterling and inclusive of irrecoverable VAT. Values should be in pounds and pence. Each entry should include a decimal point and exactly two digits for pence. Pound or other currency signs must not be included. Income or other negative spend (eg corrections) should be show with a leading minus sign. Leading zeros should not be used. Commas should not be used to separate thousands of pounds. So, for instance, a payment of £25,123 should be shown as 25123.00 and a credit of £26,123.45 should be shown as -26123.45. If possible recoverable VAT should not be included and does not need to be shown separately. Expenditure needs to be published gross of any income. Where an invoice crosses multiple expense codes, the separate payment lines should be published even where they are individually worth less than £25,000. The amount should represent expenditure for an individual transaction and should not be cumulative. |
Mandatory |
| 7 | Supplier Name | The full name of the supplier | To identify the recipient of the spend | The name of the supplier named on departments’ own vendor record can be used. Where the same supplier has been recorded using different naming conventions, there is no need to aggregate. However these multiple versions will appear in the published record. | Mandatory |
| 8 | Supplier ID | The internal ID of the supplier in the accounts system | To allow suppliers with same name to be distinguished form each other and allow matching of the same supplier even if the name is changed | Mandatory | |
| 9 | VAT Registration Number | The supplier’s VAT registration number | To identify and aggregate supplier information even if the supplier descriptions are not consistent | Publishing the VAT number makes it much easier to match the supplier to external data, and thus makes comparisons across authorities much easier | Desirable |
| 10 | Expense Area | The Department or Directorate responsible for spending the money | To identify the area within the authority that has spent the money. The description needs to be meaningful and may possibly relate to the department’s organisational structure. This may be at directorate level or relate to the cost centre structure. |
Mandatory | |
| 11 | Expense Type | The description of the type of expenditure | To identify the general nature of the spend | The description of expenditure used against account codes held on departments’ own finance systems should be suitable for this purpose e.g. consultancy, stationery. Descriptions should make sense and be suitable for the public domain. |
Mandatory |
| 12 | Expense Code | The internal account code which represents the Expense Type | To allow expense types to be matched even when the wording of the description is changed | Desirable | |
| 13 | BVACOP | The BVACOP code for the transaction | To allow comparison of spending by finance category between periods and authorities | Desirable | |
| 14 | Proclass | The Proclass code for the transaction | To allow comparison of spending by finance category between periods and authorities, and to aid in connection to procurement data | Desirable | |
| 15 | Extended Description | A description in words for the details of the transaction | This might be stored as a commentary in the accounts system, and inclusion will help explain the payment | Desirable |
Appendix ii: examples of data to be included
The table below gives specific examples of transactions that should be included in publication.
| No. | Examples of transactions that should be published | Reason |
|---|---|---|
| 1 | Payments to other government and public sector bodies | All transactions whether with other public or private sector bodies should be included |
| 2 | Payments to government or other third party service providers | All transactions should be included |
| 3 | Payments to sole traders | Business rather than personal expenditure |
| 4 | Payments for secondees | Payment for service rather than personal or pay bill expenditure. However, if a secondee’s pay would become transparent, this should be redacted. |
| 5 | Travel and subsistence claims | |
| 6 | Service charge element of pension contributions | |
| 7 | Ex-gratia payments above contract price | The full payment cost is required |
| 8 | Credit notes | Needed to ensure correct transaction values have been recorded |
| 9 | Policy lending (other than to individuals, or funds management) | Regarded as spend |
| 10 | Gifts | Publishable under FOI |
| 11 | Rent and business rates | Standard expenditure costs |
Appendix iii: examples of data to be excluded/redacted
The main principles are expected to follow the exemptions provided by the Freedom of Information Act. Key redactions will relate to data that is protected under the Data Protection Act, particularly relating to children and vulnerable persons. Please note this should be read in conjunction with the examples of data to be included, above.
The table below gives examples of the types of transactions that may be redacted or excluded from publication.
| No. | Examples of transactions that may be excluded from publication | Reason | Redacted or Excluded |
|---|---|---|---|
| 1 | Salary payments to staff (including bonuses) | Personal information protected by the Data Protection Act | Excluded |
| 2 | Pension contributions (excluding service charge) and National Insurance Contributions | Excluded | |
| 3 | Severance payments | Redacted | |
| 4 | Payments to individuals from legal process - compensation payments, legal settlements, fraud payments | Redacted | |
| 5 | Competition prizes – where a normal part of operations | Redacted | |
| 6 | Settlements made with companies as an arbitration which is conditional on confidentiality | Commercial-in-confidence – exempt under FOI | Redacted |
| 7 | Potential betrayal of a commercial confidence, or prejudice to a legitimate commercial interest | Very rare and will need to be justified | Redacted |
| 8 | Transactions relating to the financing or underwriting of debt e.g. purchase of credit default swaps | Outside the definition of expenditure for this purpose | Excluded |
| 9 | Provisions or promises to pay not yet realised | Excluded |
Appendix iv: Links to useful guidance
- Original draft guidance and comments http://data.gov.uk/blog/publishing-itemised-local-authority-expenditure-advice-comment
- Treasury Guidance for central government http://www.hm-treasury.gov.uk/d/transparency_spend_over25100910.pdf[PDF]
- Tim Berners-Lee’s paper, ‘Putting Government Data Online’ http://www.w3.org/DesignIssues/GovData.html
- Balancing Public Interest and Commercial Issues http://data.gov.uk/blog/payments-over-%C2%A3500-%E2%80%93-balancing-public-interest-and-commercial-issues
- CLG press release re local authority spending http://www.communities.gov.uk/news/corporate/1606882
37 comments
Comment by Anonymous posted on
You call for 'free' distribution of their information but send your guidelines out in a proprietory format? If this were not an official government document I would think you were trying to be ironic?
Why not convert the document to nice semantic html and publish it online? You could even use CSS to properly format it for printing or converting to PDFs.
Sincerely, Craig Loftus
Comment by Anonymous posted on
The Word doc was put up there for speed (it was actually written using Google Docs, as this blog post explains), but was turned into HTML less than 24 hours later.
We could have waited till the HTML was done, but wanted to release it as soon as it was appproved as councils have been asking for it. Incidentally, turning it into HTML was somewhat problematic, apparently, as neither Google Docs nor OpenOffice could export the table in the document as an HTML table, so that had to be constructed by hand.
Comment by Anonymous posted on
The calls for better information to see what government and the public sector are doing and giving us, the tax payer, proof that they are operating in a value for money way are sound as a general principle.
I agree that information should be available but this requirement for accessible formats and ‘turn it into a web page’ is ridiculous. Do meet these demands will take time and resources.
We are therefore asking public bodies to prove they are being efficient by wasting time generating this evidence in pretty layouts because we are too lazy to sort through it ourselves!
Comment by Anonymous posted on
This blog has to be the most appalling example of non elected officials telling democratically elected councillors how to publish information.
No one can argue with transparency on how the publics hard earned taxes are spent, but we don't need to be lectured by faceless officials on what format to use. For the record my Council has already published the required information since April 2010. It appears to me that CSV is the preferred format as it allows it to be manipulated and then used by other outside bodies either for commercial reasons or to allow Central Government to do some nice graphs. This has nothing to do with being accountable to local taxpayers which should be at the heart of it.
We've taken the view that publishing in Excel and Pdf fits the need of our local population best and thats what we will do, I can't believe that in the current government mantra of localism and letting democratically elected local government do what the electorate wants that someone let this pile of drivel see the light of day.
Comment by Anonymous posted on
"All councils to publish details of all spending over £500 in full and online...[so that]...the public should be able to see where their money goes and what it delivers"
Extract from joint LGA / CLG press release 4th June 2010, endorsed by Eric Pickles. This press release makes no mention of the "reuse of the data in innovative applications, mashups, visualisations and analysis" as an Over-riding Principle.
"...[the Coalition Government]...want to free you to deliver for the public. We're not going to be micromanaging, second guessing, and interfering in your affairs any more. We're going to get on and let you get on with it."
Eric Pickles, Speaking at LGA Conference, 6th July 2010
Sorry Eric, this guidance would suggest otherwise.
Comment by Anonymous posted on
A plea to those decrying the use of standards in open data - without such standards comparison of spend data between authorities is impossible. I am interested in comapring the LA I live in with the one next door. Both are financially managed by the same set of LA officers, both use the same financial management software. Indeed both LAs have simialr sized budgets, populations and staffing levels. However, one has spent £6.4m and the other just 1.2 in the same 3 month period. Both have published their data in CSV but neither have used either the BVACOP classification or the ProClass scheme. Therefore, the only info is supplier, amount and the internal accounting label - only two rows out of over 700 records have the same "Expense Type" across both authorities. I could go through each line and make a stab at classifying it myself but I shouldnt have to - the major Financial Management Systems should spit this data out in an open standard format at the touch of a button. Shouldnt need to be any discussion on it. If Pickles wants an army of armchair auditors then the data has to be capable of being scrutinised. Faceless council officers hiding behind excuses should get out into the real world and see how far they get without a "can do" approach!
Comment by Anonymous posted on
Excellent - the only sensible comment on this whole idea. By the way, I wonder if any research has been done to find out how many local residents/electors/taxpayers have made use of this data? If it is less than ten per council (which wouldn't surprise me), publication should be scrapped as the benefits would not justify the cost.
Comment by Anonymous posted on
While I applaud the banning of Excel and PDF formats for publishing data, surely the logical format to recommend is OpenDocument format? This is more space and bandwidth efficient and is readable for free by anyone with a computer, whether through downloading various free software packages such as OpenOffice or Lotus Symphony, uploading to Google Docs etc., or in later versions of Microsoft Office. It is also XML based and therefore easily processed by automated systems. What's not to like?
Comment by PaulGeraghty posted on
I am a bit bemused by this requirement regarding the date:
"The UK date format (DD/MM/YYYY) should be used."
I'll take a stab and guess that the majority of the proposed csv data is originally coming out of some kind of RDBMS and unless this comparison sheet is wrong
http://troels.arvin.dk/db/rdbms/#data_types-date_and_time
the native format for a timestamp or date is YYYY-MM-DD
Let me further go on to guess that some of these data sets will be sucked up by others and further processed in order to data match, make comparisons, pretty infographics etc - and some of those will entail putting data back into a database - then why inflict all these extra data-processing cycles on us?
Somone takes a date out of a database, either uses some formatting function or resorts to some kind of split and reformat to make dashes into slashes, reverse the order and make the valid csv file.
Then someone somewhere else has to do the inverse to be able to sort and slice the data.
If the argument is "oh its confusing for english readers" well why aren't you stipulating a "12 Dec 2010" format?
Date formatting in a friendly way should be left to the discretion of the final consumer of the data, who may not actually need to display dates.
He/she is responsible for formatting it for public consumption, in the english dd/mm/yyyy format if their intended audience is indeed english.
Surely the csv file proposed is merely a temporary data-holding format - not meant as a display, if it was a display format you'd leave it as a pdf!
http://en.wikipedia.org/wiki/ISO_8601
Comment by Anonymous posted on
Yes, I've wondered about this. On a more general level shouldn't data formats comply with the 'UK Data Standards Catalogue' ? This aside, the Guidance makes no reference to information security requirements (from an ISO 27001 perspective) particulalry in terms of data integrity. Is this a concern to anyone else ?
Comment by Anonymous posted on
On one hand amusing but on the other worrying.
The date formatting required for reference.data.gov.uk/id/day is yyyy-mm-dd i.e. internationally recognised.
And I agree date formatting is at the discretion of the end user.
All the data should be exported where possible to international standards.
Regards
Dick Murray, Technology Specialist, Unit4
Comment by Richard Taylor posted on
I have written an article in response to the publication of this guidance:
http://www.rtaylor.co.uk/publication-standards-local-government-spending-data.html
I have made a number of suggestions intended to reduce the impact of redactions, and propose incorporating within the guidence a principle of only applying the minimal redaction necessary.
I have also suggested allowing lines describing aggregated items where the individual items are not otherwise present due to either redaction or due to their values being less than the threshold.
The guidance omits making some types of data mandatory on the grounds that not all councils will be able to provide it (eg. if they use alternative classification schemes to those listed). I have suggested guidance would be stronger if it required the publication of these fields where the council holds the information.
I will also be writing to my MP asking him to seek to ensure that at least the mandatory elements of the guidance are incorporated into the law requiring councils to publish each item of spending over £500 from January 2011. I am concerned that if the publication standards remain merely guidance councils will be free to ignore them, and without the guidence potential benefits of the government's aspirations will not be realised.
Comment by Anonymous posted on
Is there actually going to be a " law requiring councils to publish each item of spending over £500 from January 2011"?
Comment by Richard Taylor posted on
I expected legislation to follow the Prime Minister's commitment.
I am surprised no legisation is yet in place, I have written to my MP, suggesting that without a legal requirment to publish the Prime Minster will not fulfil his commitment:
http://www.rtaylor.co.uk/publication-standards-local-government-spending-data.html#comment-59157
Comment by threedaymonk posted on
The CSV requirements aren't specific enough: they don't cover what happens when a field contains quotes or blank lines. A file that meets the requirements of these guidance documents could still be ambiguous and impossible to parse.
However, there is a specification for CSV – RFC 4180 http://tools.ietf.org/html/rfc4180 – which does cover these edge cases (which, in my experience, occur rather regularly). I recommend that the guidance by updated to require compliance with this specification.
Whilst Microsoft Excel is unable to generate UTF-8–encoded CSV files, I believe that OpenOffice is able to import Excel files and generate valid UTF-8 CSV files. This might make it a viable intermediate step in a publishing workflow.
Comment by Anonymous posted on
In section 2 it says "We recommend the new data.gov.uk licence". I looked but I couldn't see this anywhere.
Comment by phil posted on
Hi - you can read about the data.gov.uk licence on the following page: http://data.gov.uk/terms-and-conditions
Thanks,
Phil
Comment by Anonymous posted on
Under the "You are free to:" it includes "exploit the Data commercially ".
This doesn't seem fair. Taxpayers money has been spent compiling and publishing the data, if money is going to be made from it commercially there should be some return to taxpayers to reduce their burden.
I thought the original idea was to allow general members of the public to scrutinise the transactions they had funded with their taxes but the direction this has taken with csv as the standard format and commercial re-use of data indicates that original idea has been lost somewhere.
Comment by Anonymous posted on
I fully endorse the comments about the .csv format being more accessible and valuable to commercial organisations than to the taxpayer. The guidance seems to be biased towards organisations who are able to analyse and subsequently exploit large amounts of data, rather than individuals who quite rightly should be able to see what their taxes are being spent on.
Comment by Anonymous posted on
In paragraph 7 of section on ‘Publication requirements and process’ the sentence:
‘It should also be noted that the data to be published, will, in most instances not disclose price, and therefore is unlikely to be commercially sensitive.’
This does not make sense to me surely the above overall mandate of ‘each financial transaction over £500’ is all about the price and spending?
Comment by Anonymous posted on
Hi,
Can you produce a worked example of the CSV file?
It is not clear from the guidance what it should look like. For example should columns 1 and 2 be repeated for every row. This might be useful for automated processing but would probably make it harder for the average human. Similarly it is not clear what would be redacted and what should be left in.
Thanks
Comment by Anonymous posted on
Just wondering if this applies to Parish and Town Councils
Comment by Anonymous posted on
Is this a legal requirement?
Comment by Anonymous posted on
See also the Local Government Group "practitioners' guide" published October 2010
http://lgtransparency.readandcomment.com/comparison-between-spending-guides/
Comment by Anonymous posted on
Hi
can anyone tell me if spending by schools is included in this exercise please - in particular local chequebook schools as the data on their spend is not easily extractable
Thanks
Paul
Comment by Anonymous posted on
The guidance suggests publishing from April 2010 - is this going to be the minimum cut off date or is it just guidance? Some councils are publishing from April 2010 and some are starting from Sept 2010 with no intention of going back further - surely an agreed date should be necessary in order to offer a full comparison of all council spending over the same period. If there is no requirement to publish from a specific date then there will be a mishmash of start dates resulting in 'incomplete' comparison.
Comment by Anonymous posted on
When it comes to publishing the names of sole traders we need to ensure that we take into account all the necessary privacy laws such as Data Protection. We need to consider what the expectation of the sole trader is as far as privacy is concerned.
For example, within local government there is a long established principle that the more senior someone is the more of an expectation there is that their names etc. will be put into the public domain. Alternatively might be quite reasonable to publish the names of junior staff if they have for instance a very public facing role.
The same sorts of considerations need to be taken for sole traders. If a trader has their details published all over the internet for instance then there is probably less expectation of privacy than someone with a less conspicuous role.
Any guidance on the matter ought to be produced in consultation with the Information Commissioners Office to ensure that privacy laws are followed.
Comment by Anonymous posted on
The safe option is to ensure you only use the sole traders company name in the data files. If a sole trader has chosen to register their company using their own name, then publishing details of the company even where this is the same as personal identity of the sole trader can't be disclosive becuase the company name is already in the public domain by virtue of its business and company regsitration.
Comment by Anonymous posted on
Could someone please clarify if we should be publishing Supplier VAT references as the specification on this site differs from that on the http://lgtransparency.readandcomment.com/ pages. The Data.Gov specification asks for VAT reference as desirable but the LGTransparency pages asks us not to include it as the VAT reference could lead to fraud.
Comment by Anonymous posted on
Please, can we all use a similar file naming convention and tag the data the same. I have been having a look round at what has been published and it take time to find, in fact you could say we make it difficult. On the plus side the data I have found on use of agency workers has been great for benchmarking and building a case for a joined-up approach.
Comment by Anonymous posted on
Different authorities use different procurement coding systems and each has its merits. Moving from one to another is prohibitively expensive. So why assume we're all going to either publish a Proclass code or no procurement code at all? Give us the option to use other encoding systems such as UNSPSC if we so wish - and to put this in the metadata.
Comment by Anonymous posted on
Why does the guidance on this site about what is to be published differ from that at http://lgtransparency.readandcomment.com/appendix-a-spending-data-template/? Which are we supposed to be implementing?
Comment by Anonymous posted on
I'm confused by the recommendation to list the files at data.gov.uk. There is a site called direct.gov.uk, supposed to be a one-stop shop for the public to find out everything to do with government, local or central. Why have another site with a link on council spending that most people haven't heard of? Surely it's better to link the data to direct.gov, and leave data.gov as a site about the issues of openness and transparency. Are there plans to do this?
Comment by davidpidsley posted on
Who is responsible for responding to these comment? Most of them are clearly relevant and pressing.
Comment by phil posted on
Hi David,
Thank you for your post. I can confirm that feedback to the guidance, both posted here and communicated via other channels, is collated and is being reviewed with a view to posting a consolidated response on data.gov.uk in the coming weeks.
Many thanks,
Phil
Comment by Anonymous posted on
I'm unclear what we are supposed to do if our local authority refuses to comply with the 'mandatory' requirement to publish data in CSV format?
Comment by shamylla posted on
Hello - i just wanted to know if there is one source where all the raw data is consolidated?