Handle with care: how can government departments identify and manage critical data assets

By on 30/11/2023 | Updated on 30/11/2023
Speakers at a roundtable session at Public Service Data Live in London September 2023

Data governance and compliance are bedrocks for good government – but they can also be challenging territory. Senior public servants discussed why during a roundtable during the Public Service Data Live conference

The public sector’s efficiency in managing increasing volumes of data ultimately relates to citizen trust in governments.

‘Picking the right data: how to decide the data you need to keep, ensuring correct governance and compliance’ was a roundtable hosted by Global Government Forum alongside knowledge partner Veritas Technologies on 14 September 2023.

The session, supported by knowledge partner Veritas, was held under the Chatham House Rule – meaning that we may not identify those speaking, but topics discussed included: the challenge of identifying ‘critical data assets’; how to handle ‘ancient’ data that there is no (apparent) need to keep; and whether there is a playbook for the establishment of a new departmental data function.

Challenges are multifarious: examples provided ranged from news events triggering urgent questions about retained data, through to departmental re-organisations leaving successor bodies unclear what to do with ‘old’ inherited documents.

The overall subject is of growing importance for multiple reasons, including for the public finances. “The cost implications are massive,” one participant warned. “We spend millions and millions on data that’s stored and which is no longer relevant.”

‘Really complicated picture’

“When I started to look at what data [the department] had, it was a really complicated picture,” said one participant in an opening observation. “It was almost impossible to work out what we had, who was collecting it, how it was stored and who was responsible.”

One priority – which many seemed to be wrestling with – is the identification of ‘critical data assets’. “Who’s going to be involved in the conversations? What is essential by law? Where does that leave some of the things that are less critical for some parts, but not for others?”.

One referred to more practical obstacles including trying to understand “how data is saved and what the tagging system is”.

In a similar vein, one referred to finding a situation where the same data was being held in multiple places simultaneously on apparently mismatched data management timescales. 

“One of the biggest things that I find is, we have to take data from one secure repository, like a database, and then we put the data somewhere else, which is another secure repository,” explained the participant. “We’ve sometimes got measures whereby we take the data off one secure repository after three months and ‘cleanse’ it, but we’ve then got it kept on another secure repository for another six months or nine months – or forever.”

Policies making old data ‘relevant again’

Dealing with surging volumes of digital data is front of mind for most organisations. But the sheer volume of (typically older) printed documents retained across the public sector emerged as a prominent theme.

“Just because it’s on papyrus and 2,000 years old doesn’t make it not data,” one participant, whose organisation deals with plenty of very historic documents, observed.

The Windrush scandal – when Caribbean migrants who had come to the UK legally were unable to prove their legal status due to some key information having been destroyed – was raised as an example how unknown future policy decisions could render ‘old’ data (in whatever form) essential again. “Subsequent policy decisions made data that had been destroyed now relevant again, and there was no way then to get it back,” a participant observed.

Another attendee referenced the practical question of who stores historical data. “Normally, it’s an external contractor that holds the information,” he said.

He added that “slowly but surely… data is being purged” across his department but that it can “take years” to do so. “We’re talking millions and millions of files,” he said.

Legal and interpretation problems

Participants had different views on the robustness of their departments’ processes in deciding whether data can be deleted.

References were made to legal ‘ambiguities’ and, where policies do exist, problems of interpretation. This is typically exacerbated when data management responsibilities transfer from one team or department to another, or people change jobs.

Reference was made to confusion and potential misunderstandings within the same organisation. “We’ve got one understanding, which is to keep all of our records up to date, and we keep it forever. Then, in the second place, ‘I can only access data for six months and then they’re going to wipe it’.”

One participant brought up how deleting data can fatally undermine attempts to measure performance, saying that if data is deleted after six months, performance cannot be tracked over a 12-month period. “From an evaluation point of view, you wouldn’t be able to ‘go back’ because we get caught up in conversations around GDPR and data protection,” he said.

A different attendee referred to uncertainty over future data needs causing tension between IT developers and data policy teams. Another spoke of how cloud storage raised questions and uncertainties. Most challenges seemed to relate to governance and communication rather than technical solutions.

‘Asset registers’ and accountability

Naturally, departments’ approach to data retention differs. But one participant questioned whether spending time deleting data was a good use of resources.

“Because we can’t predict the future in terms of use cases, why are we deleting anything when cloud storage costs nothing?,” he asked.

One attendee suggested that she found herself battling against colleagues to ensure data was not destroyed, saying that (in her department) it was occasionally necessary to find documents from hundreds of years ago.

One participant referred to a departmental ‘asset register’, which catalogues which information is stored, and a “governance process about what can and cannot be deleted”. “So, there is decision making and it’s logged. There is accountability,” he said. But this is easier for newer, digital data than it is for older, non-digital and potentially unsorted documents (data).

Participants sought answers about what to prioritise and who to turn to. One advised focusing on identifying the department’s critical data assets; another said their own next step would be to consult the CDDO (Central Digital and Data Office) about guidance on data asset registers.  

‘Complexity’ in government

Data management requires investment. But there was little sense from participants that public sector budgets are increasing to reflect the digital data deluge.

“Managing data, and taking stock of your data, is not necessarily the most exciting thing to do – although it is essential – [so] it slips down your to-do list,” reflected one participant.

Challenges often relate to people rather than processes, albeit the two intertwine.

“When people who set up systems depart, and every time there is a handover, a little bit gets lost,” said one attendee. “So, 10 years down the line, that data is distorted, it’s incorrectly labelled, whatever. How do you find that place that it’s consistent and trace back? The original intent – the lens that was used to interpret it – is lost.”

“The complexity is that each government department has its own set of rules, policies, processes – and government departments don’t really talk,” one participant said. “Yes, that’s getting better. But there’s still a lot of silo working. So how do we then break that up to have a consistency of approach across the board? And, critically, look at what the critical assets are that are retained across civil service.”

Automation for the people

The roundtable came to an end with one participant reflecting that, in his experience, private-sector organisations are finding themselves similarly challenged by the same questions.

“No-one’s on top of it in the private sector or the public sector,” he said. “There’s a very, very small percentage that have realised that they need to know what [data] they’ve got, and then they can make those informed decisions.”

That said, he was of the view that the public sector seems to have a higher proportion of decisions around retention or classification being taken through a ‘manual’ process. “That is one of the things that categorically just doesn’t work. With the sheer amount of data you’ve got to manage it becomes insurmountable,” he said.

“I think the key difference between the public sector and private sector in the way we handle data is that the private sector recognises that it’s an asset and it needs management, resources and investment,” concluded another participant.

Pressure to get things right, and technical solutions, is only going to increase as the volume of data held across the public sector continues to grow.

‘Picking the right data: how to decide the data you need to keep, ensuring correct governance and compliance’ roundtable was held at the Public Service Data Live conference on 14 September and supported by knowledge partner Veritas Technologies. It was attended by:
Daniel de Wet, Department for Work and Pensions
John Butcher, Department for Work and Pensions
Elizabeth Jeary, Ministry of Justice
Andrew Cleary, Department for Environment, Food and Rural Affairs
Harriet Fearn, Department for Education
Aftab Hussain, Department for Education
Lee Greenhill, Land Registry
Matthew Banks Veritas Technologies
Louise Lewis, Veritas Technologies
Ian Hall, Editor, Global Government Fintech

About Partner Content

This content is brought to you by a Global Government Forum, Knowledge Partner.

Leave a Reply

Your email address will not be published. Required fields are marked *