We’ve talked a lot about registers forming part of a digital ecosystem, providing data good enough to build services on. We’ve also outlined the characteristics of a register and how each characteristic helps registers remain authoritative and up to date. These aspects are integral to the value of registers, so we need to be sure they’re underpinned by solid technical features that handle the complexity in the software rather than passing it on to users.
Users of registers sit on both sides of the government data ecosystem. Data owners use registers as a mechanism for publicising and managing their core reference data. On the other side, many different kinds of data consumers across government use registers to support their internal functions or their public digital services. Our technical choices need to work well for both broad groups of users and the jobs they need to do.
Our user research shows that almost all our users share the same basic needs. In particular, they need to be able to download a copy of the register in a format that suits them, and easily keep that copy up to date. Custodians of registers also want to be able to make updates to the registers they are responsible for easily, and with appropriate ways of double-checking their changes. We realised that making register data too dependent on our software implementation made registers unnecessarily complex to use, and introduced extra complexity for us when developing and deploying new features and tools.
So, we’ve spent some time working on a format that can contain a complete description of a particular register. This means that an exact copy of a register can exist in other software or products that service teams or data owners are using, and be flexible to the diverse user needs across these groups. It also allows data owners to make updates to their registers in an assured and consistent way, and enables data consumers to take updates from a register which automatically picks up everything added since they last took an update. In practice, this helps us meet a few key user needs.
Data consumer user needs
For data consumers, this approach simplifies the process of getting register data by allowing users to download the entire structure of a register in a single file, and a single API call, rather than needing to build the whole picture from several different requests to different parts of the API. This means that consumers don’t need to know or understand the structure of the register model before requesting the data through the API. It also makes it easier to take a copy of a register and keep that copy up to date, because it marks each update and download so that users can choose to download just the entries and changes they don’t already have from their previous downloads.
So we’ve developed a Register Serialisation Format to make register data more portable, meaning it can be used and moved separately from the software used for hosting registers on register.gov.uk. This separation means that consumers can rely on the stability and longevity of the data, independent of work that continues on the register platform software.
Data owner user needs
Speaking to our users, we’ve learned that we need to accommodate a wide range of existing data management processes for our custodians: the people who manage the data in registers. Although register users registers won't see anything different on the surface, having an intermediate serialisation format means that a custodian can stick to the data format they already use. Behind the scenes, the data can be converted into the intermediate format before it's loaded into the relevant register.
We can meet a wider range of data management use cases with a relatively small range of update tools, without causing too much unnecessary disruption to custodians’ existing data management workflows. This means that instead of us trying to develop an entirely bespoke register update tool for each register or custodian, or developing just one update tool and asking all custodians to change their existing processes to fit it, we can be more flexible.
And there’s more
The work we’ve done in developing this intermediate format also supports some of our upcoming work. In making registers a more easily moveable chunk of data within our software implementation, we’ve laid the foundations for developing some key user features as elements attached to, but independent from, the register platform software. This will make it easier for users to get started with registers, because they won’t always need to manage their own version of the software to meet simple use cases.
In the longer term, we’d like the features we develop for users to act as a sort of toolkit that users can select from and iterate with, without having to work with a single very large piece of software if they don’t need to. It will also allow us to be more agile in developing new tools and features because we won’t need to regularly change our register platform software to incorporate or manage these new features.
We’ve also used the Register Serialisation Format to lay foundations for extra security features for custodians updating their own registers. One of the key characteristics of a register is that we can ‘prove data integrity’. Register Serialisation Format enables the integrity proven within the register platform to be contained within Register Serialisation Format file too, so the integrity is as portable as the data itself.
We can assure data integrity by generating updates with a unique identifier at the beginning, and a different unique identifier at the end. This enables a custodian to demonstrate that they have not changed or deleted any existing entries as part of the new update. On the opposite side, it allows users to download a register in many parts over time but also have confidence that, when put together, the many parts are a whole copy that precisely matches the current state of the register. If the parts of the register do not match up to the original, the user will know their copy of the register is inaccurate.
Next, we’re going to be applying the things we’ve learned and developed so far to develop mechanisms for updating registers that meet the needs of custodians. We’re also exploring the types of tools and features that consumers of register data would find most useful to support their work.
Get in touch and tell us what registers or data tools you’d like to see.
2 comments
Comment by Robert McCarthy posted on
Hi Ellie,
This sounds really interesting...
Through I don't understand why you haven't told us where we can find the specification for the "Register Serialisation Format" or did I miss a link somewhere?
Comment by Peter Parslow posted on
Is this "Register Serialisation Format" something new & specific to GDS registers, or have you been able to use an existing open specification for register exchange e.g. from the European Commission's ISA project (ARE3NA, Re3gistry), or ISO 19135 (as used by the NATO related Defence Geospatial Working Group, and the International Hydrographic Bureau), or the OASIS registry (ebRIM).
And if none of them are suitable, is the UK in a position to say why - ask for change?
Actually, perhaps it could have been a challenge on standards.data.gov.uk, because I'm sure I'm not the only person who can think of existing specs.