Crystal clear: Sir Ian Diamond on using data to profile the pandemic
The UK civil service has transformed its data operations during the pandemic, says Office for National Statistics chief Professor Sir Ian Diamond – web-scraping, integrating datasets, and building partnerships with businesses and foreign governments. And with more analysts and statistics professionals, he tells Matt Ross, it could move even faster
“We have seen a rapid and incredibly impressive use of statistics over the last year, in radical, novel and innovative ways,” says Professor Sir Ian Diamond. “During the pandemic, we have done things at much greater pace than we might have otherwise.”
As the UK’s national statistician, Diamond is permanent secretary of both the Office for National Statistics (ONS) and its parent body, the UK Statistics Authority; he is also head of the civil service’s analysis function and its statistics profession. So nobody has had a better view of the UK’s use of data since COVID-19 arrived – and Diamond is clear that civil servants’ achievements here rest on the painstaking strategic and capacity-building work of previous years.
“When the current government came in, they made the better use of data one of their major platforms,” he recalls. By the time he left academia to join the civil service, “I was able to walk in – in late 2019 – to an environment where the conversations were about ‘how to’, not ‘if to’” realise the potential of data.
Since September 2020, when the UK published its long-delayed National Data Strategy, the progress on data management has accelerated further. In January 2021, the government set up a Central Digital and Data Office chaired by LEGO’s chief digital adviser Paul Willmott. And the ONS is currently, Diamond explains, building “on behalf of the whole of government, an integrated data platform” through which departments can “share data in a way that will enable us to address really, really big policy questions.”
This platform has, for example, been used to “link vaccine data with census data, in order to understand the characteristics of people who have or have not had the vaccine.” In future years, says Diamond, it will help in tackling issues such as climate change – bringing together data from key fields including food production, transport, business and housing.
A new spirit
Alongside the creation of these new capabilities, Diamond believes, there has been a culture shift among civil servants. “We’re at a moment in time where people right across government realise that we need multiple departments engaged in addressing the big questions,” he says. “And to do that, we need to work together.”
The public sector’s work to shield vulnerable people when the virus first arrived shows this new spirit in action, Diamond says. This required rapidly pulling in staff and data sets from the Government Digital Service, National Health Service, local authorities and other bodies. For him, it exemplified “cross-departmental working, innovative use of data at pace, and really strong analysis.”
Meanwhile, Diamond explains, ONS staff set up a national household survey, testing selected households for both COVID-19 and antibodies every week, and sequencing every case found. This has provided a detailed picture of infection rates around the country, helping to shape government’s lockdown policies. A parallel weekly opinion survey has tracked people’s views and experiences, providing data on issues such as “the impact of the restrictions, loneliness, mental health.” And putting these data sets together has provided further insights into crucial topics such as ‘long COVID’: the ONS has found, for example, that 14% of infected people experience symptoms for at least 12 weeks.
As well as these test- and survey-based information-gathering techniques, says Diamond, the ONS has been using some “pretty radical methods” to access new sources of data. “To maintain our inflation estimates, we’ve been web-scraping to get price data,” he says. “To understand movement, we’ve been using telephony data and web-scraping Google Maps data. We’ve been using financial transaction data to understand spending.” Such methods, he adds, are “really important because we can do it at pace, producing economic indicators faster.”
Overcoming geography
All this progress on data handling is particularly impressive given that, ever since March 2020, the vast majority of statisticians and analysts have been working from home. Here, ONS staff probably had a head-start – for, having moved its HQ from London to the industrial city of Newport in South Wales a decade ago, the organisation has long operated a widely-distributed workforce. “We’re flexible in the way people work, and that enables us to be somewhat location-blind in our recruitment,” comments Diamond. “One of my very valued directors works and lives in Manchester.”
The ONS’s experience may also be valuable to UK departments looking to move staff out of the South-East under the government’s relocations policy – a key plank of its ‘Levelling Up’ agenda. Asked how the agency has been able to recruit and retain data professionals in Newport – a former industrial and port city without a significant tech sector – Diamond replies that “there’s no magic bullet”. Partnerships with universities and colleges have been important, he says: “In local areas, we are working very hard to raise the idea of a career in statistics with people who may not otherwise have realised how exciting and stimulating a career with us could be.”
The ONS’s flexible working offer helps it to cast the net more widely, making jobs accessible to those living far from its offices. Equally crucial, Diamond says, is the promise of an exciting, respectful and inclusive working environment – and one offering interesting career opportunities for ambitious, talented staff.
Assisting overseas, persuading at home
ONS staff have, for example, been working close with the statistical offices of developing countries – including Kenya and Ghana – and the UN Economic Commission for Africa. “We’ve been talking about our experiences in the pandemic, and sharing how we can help each other,” says Diamond. The ONS assisted Kenya, for example, in delivering its first digital census and the agency has put a lot of work into developing a global data platform to track progress against the Sustainable Development Goals (SDGs). “I think the SDGs are wonderful,” Diamond comments. “But they’re only as good as the monitoring that you are going to do of them. We’ve worked with colleagues right across the world to see how best we can provide harmonised data.”
Most of the ONS’s work, though, focuses on England and Wales – where COVID-19 has led to skyrocketing public interest in data. “I am really clear that we have had an improvement over the last year in the public’s understanding and demand for data,” he says. Yet people remain suspicious of government’s use of their personal information, requiring public bodies to tread very carefully around initiatives such as vaccination passports. How can data leaders calm these fears?
“Too often, I think we have just said: ‘Oh, give me your data!’,” replies Diamond. “But if we say: ‘This is why we need your data. This is how we’re going to use it. This is how it’s being protected. This is why the use is very clearly in the public interest – what’s in it for you or for others,’ then that is a different conversation. When we have these conversations with people, then overwhelmingly they are in favour of the use of their data.”
Educate, explain, oversee
Over the longer term, Diamond says, the government must “be tireless in promoting data literacy right across the population.” Plumbing trainees, for example, need data and numerical skills to succeed in the jobs market – “and these things will help you also when you’re watching the news to find out what’s going on with the pandemic. I’m passionate that data and numbers should be part of every educational programme, whatever you’re studying.”
Meanwhile, government must ensure that every use of the public’s data is justifiable and explicable. “If you can’t explain things, then you shouldn’t do it in the first place,” Diamond notes. Asked about incidents when that principle has not obviously been followed – such as the A-level algorithm, and the government’s abandoned plans to centralise data in its contact-tracing app – Diamond responds that “it’s very easy to say: ‘Oh, there’s been failures.’ But actually there’s been an enormous number of successes. And I do think there’s been a total commitment at all times to improvement and to getting things absolutely right.”
The UK government’s daily COVID-19 press conferences provide a useful example here. Following criticism of some of the presentations used to display information, “we at the Office for National Statistics were asked – and agreed very, very quickly – to embed a small team inside Downing Street to make sure that the quality of the presentation of those data on those Powerpoints was of the highest standards. I think we all agree that it’s possible to draw a pretty bad graph!” Things improved quickly, Diamond notes: “We should be proud of the visualisation of the data we are seeing at press conferences.”
Spread too thin
And what’s required to improve the civil service’s use of data? Diamond oversees the civil service analysis function – which includes professions such as economics and research as well as statistics – and works closely with digital and data staff, he replies, “and my answer is that we’re in very, very good shape – but there’s not enough of us. And I think that’s acknowledged. There are really super data scientists and analysts right across government, but we would all agree that there needs to be more data science and there needs to be more analysis.”
Throughout the pandemic, says Diamond, the ONS has been “doing everything that we possibly can to reimagine how we do things at pace with accuracy. And [we’ve been] using ever more radical and innovative data sources, doing so in a way which is entirely inclusive so that no one in our society is invisible; and for those who perhaps have been, they are now visible.”
And as vaccination programmes get the pandemic under control, will the sense of urgency fade – leading to slower progress on improving the civil service’s use of data? “Having made such rapid, impressive change in the use of statistics and data, will we somehow go backwards? I can’t see why anyone would want to! And in my opinion, government should demand that the trajectory is increased,” Diamond replies.
“We have the opportunity to make such a transformation in the way we understand our citizens’ lives, our society and economy; and therefore to improve our citizens’ lives,” he concludes. “And while I’m anything to do with it, I will do everything I possibly can to continue that.”
About Matt Ross
Matt is Global Government Forum's Contributing Editor, providing direction and support on topics, products and audience interests across GGF’s editorial, events and research operations. He has been a journalist and editor since 1995, beginning in motoring and travel journalism – and combining the two in a 30-month, 30-country 4x4 expedition funded by magazine photo-journalism. Between 2002 and 2008 he was Features Editor of Haymarket news magazine Regeneration & Renewal, covering urban regeneration, economic growth and community development; and from 2008 to 2014 he was the Editor of UK magazine and website Civil Service World, then Editorial Director for Public Sector – both at political publishing house Dods. He has also worked as Director of Communications at think tank the Institute for Government.
Related Posts
