There is a lot of talk about APIs in government right now, and how they affect the work we are doing at GDS. We thought it was worth sharing how we approach APIs when it comes to data.gov.uk.
Data.gov.uk is the UK governments’s open data catalogue, listing data from government departments, agencies and other public bodies and making it available for public use. It is built on the CKAN platform, which provides much of the functionality we need, including a complete metadata API.
This API allows developers to perform nearly any action that’s also available to the users of data.gov.uk, meaning the API can be used to:
- show metadata about a single dataset;
- find publishers and the datasets they publish;
- search for all CSV files published by a specific publisher, within a date range, using search keywords;
- publish metadata, if you have the appropriate permissions.
These APIs are also used within the data.gov.uk codebase in various ways, including to provide the functionality in the user interface. If you’re interested, you can read the documentation for the CKAN API to find out how to use it.
Starting with APIs
The reason that our metadata APIs are so complete is because of the “API First” approach taken by CKAN. The CKAN framework means that any functionality added to the user interface automatically makes it available as an API. This promotes designing, building and using APIs as the first task once you begin developing a system - not as an afterthought, which is quite common
Designing good APIs
Designing good APIs at a later stage is hard in the same way that writing tests after you’ve written a system is hard. Doing Test-Driven Development makes it easier to consider the use-cases early in the development cycle and so is a faster and more efficient way of working.
Adding an API to an existing system can often lead to APIs that are incomplete and hard to use - especially when they are not actively employed by the development team. By using our own API to build data.gov.uk we are forced into designing APIs from the very beginning, making sure that they are usable. It’s important that they make sense both with respect to data.gov.uk and to external users and systems.
Consistency
Using APIs during the development of a system also makes it easier to identify any inconsistencies in the design, such as with naming, format or expected behaviour, and keeps us in line with the ninth design principle - be consistent, not uniform.
The same data will be presented in both the user interface and the API, providing a consistency of data presentation regardless of the user. This consistency is important as the system should match users’ expectations, and attempt to follow the principle of least astonishment. If the result of an API call is visibly different from the data the user is expecting, they will be unpleasantly surprised.
Security
Within data.gov.uk the same authentication and authorisation code is used to secure the user interface and the API. And as additions to the logic layer in data.gov.uk result in additions to the API, securing the API will also provide the same security to the user interface. This single auth/auth layer allows developers to reduce the attack surface area for malicious users and so is more easily maintained.
Following the Service Design Manual, we’ll stick with this approach as data.gov.uk evolves. This will ensure a well-designed and tested API is a fundamental component of the system. However, this doesn’t mean we will stop looking at ways we can improve our APIs. So if you have any feedback about the way you use our APIs or features you’d like to see, please get in touch.
2 comments
Comment by Rob posted on
The CKAN link links to a 404. I appreciate that it's largely a CKAN issue as their root is a 404 but maybe there could be a better place to link to?
Comment by Ross posted on
Thanks for noticing, I've spoken to the CKAN sysadmins, and they fixed it immediately. Should be working again now.