An ideas accelerator: how the UK’s Data Challenge built two new public services
Gathering ideas from the workforce, the 2021 Civil Service Data Challenge put the best on the path to delivery, bringing benefits for UK citizens overseas, peatlands conservation and the net zero agenda. The winning teams and judges tell Matt Ross what they took from the Challenge – and offer some pointers for 2022’s teams
“The first stage of any crisis response is understanding the scale, because all your planning goes from there,” says Tom Duffield. In any overseas emergency, he explains, officials at the Foreign, Commonwealth and Development Office (FCDO) must rapidly estimate the number of UK citizens in the affected area – informing crucial ministerial decisions on matters such as evacuation plans and travel advice.
As head of the Business Intelligence Unit that supplies data to UK consulates in 186 countries, Duffield has often found himself trying to pull together these numbers – and until this year, gathering the information was difficult and time-consuming. Yet in the era of ‘Big Data’, he knew it should be possible to pull multiple data sources into a dashboard providing accurate, regularly-updated estimates of the distribution of UK citizens overseas. “The idea was rattling around in my head for too long – probably four years plus,” he recalls. But his day job left little room to pursue it: “I’ve never been able to carve out the time.”
Then, in the spring of 2021, Duffield came across the Civil Service Data Challenge. Run by the Cabinet Office, the Office of National Statistics (ONS), Global Government Forum and IT services business NTT DATA UK, the Challenge is designed to tap into the expertise and enthusiasm of civil servants – uncovering great ideas for how to make better use of data, and putting the best on a path to implementation.
Welcoming entries from civil servants in every grade, role and organisation, the 2021 Challenge gathered in nearly 170 ideas. A judging panel of senior data and digital specialists picked out the eight most promising – including Duffield’s – and teams of civil service volunteers were built around them, each equipped with the required mix of skills, connections and policy expertise. The teams researched and developed their ideas throughout the autumn, presenting to the judges at semi-final and final events held in a ‘Dragon’s Den’ format.
Duffield’s idea was one of the four that made it through to December’s final; and after that, things moved rapidly. When civil service chief operating officer Alex Chisholm, a ‘champion’ of the Challenge programme, wrote to the foreign secretary and FCDO permanent secretary praising the team’s work, Duffield was given funding for a 16-week “proof of value project”. The resulting ‘Pharos’ system was “supposed to be a prototype, but we’ve used it in the real world,” he comments.
Pharos currently pulls in 30 data streams from organisations including His Majesty’s Revenue and Customs (HMRC) and Facebook, and has already much improved the information given to elected leaders. “Now we have an objective way of putting an argument together in front of the minister: something we can rely on and have real confidence in,” says Duffield.
Other departments quickly cottoned on to the system’s potential: “The level of interest it’s generated across the civil service is quite astonishing,” he says, adding that deputy national statistician Alison Pritchard – who sat on the judging panel – is “very eager for the end solution to sit on ONS infrastructure, so it can be shared across government”.
Photo courtesy NTT DATA UK
Now Duffield is overseeing the development of a full-scale version expected to include key datasets covering, for example, flight passenger details and the locations of UK mobile phones overseas. He is clear that the programme owes its existence to the Data Challenge: Chisholm’s email “sparked things into action: we were given the resource immediately, and the license to take it forward,” he says. “We had senior backing; we were able to get the commercial engine up and running a lot faster than if I’d tried to do it on my own; and we were able to bring in a supplier to support us.”
Dam good concept
Tom Duffield’s story of an idea brought to fruition is not the only one to emerge from the Data Challenge. The overall winner was put forward by a team from Natural England, which proposed using artificial intelligence technologies to scour aerial photography of peatlands habitats in search of drainage channels – guiding work to dam them up, and thus to restore these precious wetland habitats and carbon sinks.
Having received the support of Department of the Environment, Food and Rural Affairs (Defra) permanent secretary Tamara Finkelstein – who served as a Challenge judge – and a first prize of development work worth £50,000 (US$57,000) from NTT DATA UK, this project is also now well into the delivery phase. “We have achieved some amazing results in terms of precision in detecting peatland features crucial for the restoration work of our area teams, and generating synthetic data to enhance algorithmic accuracy,” explains Mary Vayou, principal adviser on data science within Natural England’s Data Science Services Team. “So we have done the proof of concept. We are now at the stage where we’re discussing with Defra the continuation of this work as a project and the additional resources that we would need to optimise algorithms, scale them up, and deploy them as an application.” The project aims, she adds, to “develop a map of peatland features, initially for the whole of England, next for the UK and then – why not? – for the world!”
Like Duffield, Vayou explains that the Data Challenge has been important to delivery: the programme provides a “pathway to an innovative space that you wouldn’t reach through the usual structures,” she says. “It’s been a game-changer.” And it has also had wider impacts – driving the development of Cloud infrastructure across Natural England and its parent department. “Without access to Cloud technologies, we wouldn’t have been able to demonstrate the results we have now,” she says. “Because of this project, the Cloud technologies deployment programme was pushed forward: everybody could see the practical benefits of being able to work on big data.”
At the FCDO too, the Challenge has “accelerated interest in data literacy within the organisation from the top, especially within my own directorate,” says Duffield. “They’re saying: ‘Wow, you’ve done this within 16 weeks? What else can we do? What do you need?”
Think slowly, move fast
Asked which elements of the Challenge proved most helpful in carrying these ideas forward to implementation, Vayou – like Duffield – highlights its ability to ring-fence dedicated “time and space for innovation”, enabling busy people to step away from the daily grind and pursue promising concepts. Here too, she says, the effects of the Challenge reach beyond the eight long-listed ideas: having learned from this “case study in how to deliver tangible outputs from an innovative idea”, Natural England is now developing ways to “incorporate time for innovation into our structures and our ways of working”. Alison Pritchard has spotted the same need: “We’re all heads-down quite often, and that doesn’t give staff the space to be able to raise ideas to the surface,” she says. “The Challenge has done very well there.”
Other important factors include NTT DATA UK’s provision of facilitators and technical experts to advise and support the teams, and the involvement of senior data leaders who – while serving as programme champions and judges – helped the participants to make valuable connections and overcome obstacles. In combination, comments Duffield, these advantages enabled projects to move with “a momentum that was very, very alien to civil servants. The pace at which we were able to do things was absolutely astonishing: you shouldn’t underestimate how important the Challenge was in creating the opportunity to get out and deliver at pace”. In the words of Vicki Chauhan, NTT DATA UK’s head of public sector and another Challenge judge, “we’ve all learned something from this process” – including the heartening lesson that “you can actually make stuff happen in government, if you’re laser-focused”.
A development opportunity
For Duffield, the Challenge has provided “the best learning opportunity I’ve taken in as long as I can remember”: he particularly valued the experience of working with senior stakeholders across government during the research phase. “Because of how dynamic and fast-paced the project was, it was like condensing three years of learning into a couple of months,” he adds. Vayou too highlights the Challenge’s value in strengthening stakeholder management skills, along with the “exposure to very innovative technologies, which was great for the teams in terms of skills and experience”.
The Dragon’s Den format also aided the development of strong presentation skills, comments Pritchard: “You’ve really got to sell the idea and capture the moment.” And by taking ideas through from a sketched-out concept to a thoroughly-researched delivery plan, participants gained “an accelerated experience of how these things develop, with the barriers and challenges and the solutions as well”.
NTT DATA UK’s facilitators and technical experts too benefited from the programme, notes Chauhan – learning about new fields of policy, and “developing an appreciation for what civil servants have to battle with every day”. From accessing datasets to finding a common communications platform, they saw that “the basics, that we take for granted [in the private sector], you have to do differently in the civil service. And they took a lot of satisfaction from being able to make those things happen”.
How to win the Challenge
With the 2022 Challenge well underway, what advice do 2021’s winners and judges have for this year’s team members? Make full use of the Challenge’s ability to foster interdepartmental projects, replies Duffield: “If it’s something that has wider benefit across HMG [His Majesty’s Government], or can only be achieved by unlocking doors across HMG that you or your department can’t unlock, then the Challenge is perfect.”
In 2021, comments Chauhan, the most effective teams were those “with clear roles and responsibilities inside the team, and those that shared the administrative burden”. Those teams that “conveyed their passion about the subject matter” gave the best presentations, she adds. And Pritchard urges teams to focus on the benefits for citizens and public services: “It’s not the sexy, exciting technology or use of data that really matters,” she says. “Can you see a fit with a clearly articulated need, and how the solution addresses it?”
Asked about 2022’s crop of ideas, Chauhan comments that the long-list is “really strong. Actually, this time round I think we’ve got a good chance of getting more off the ground [and into delivery]. There’s some really good, practical ideas; I’m quite excited about this bunch!” Challenge managers are currently assembling teams around the eight long-listed ideas, in preparation for a research phase running to the semi-final on 7 December.
Learning the lessons
For Alison Pritchard, the Challenge has revealed just how much there is to gain from tapping into the expertise, enthusiasm and inventiveness of the UK’s civil servants. “The whole process has demonstrated that there is a vast volume of really good ideas that aren’t being picked up at sufficient scale,” she says. What’s more, “if you put enough focus on a small number of them and drive them through, then they reach a point of no return” – moving forward into implementation.
So the next question, says Pritchard, is: “How can people do this at a much broader scale?” Chauhan has been thinking along the same lines: the Challenge works because “you have the power of the teams coming together, with a single focus,” she says. “How do you replicate that across departments so that you don’t need a Challenge to make this stuff happen organically?”
That is a question for civil service leaders – who certainly need all the ideas they can get in today’s turbulent political environment. Meanwhile, Tom Duffield is clear that the Challenge is a powerful accelerant for promising ideas. “I went into it a little bit sceptical, because it was a new thing and it wasn’t really clear how the Challenge would work,” he recalls. “But I’ve only got positive things to say: I’ve been contacted by quite a lot of people who were thinking about putting in bids, and I’ve been gushing over it.”
There are two key benefits, he concludes: “One is the personal development opportunity; and secondly, it is a genuine route to delivery. I can confidently say that I would not have got [Pharos] to this stage if it wasn’t for the Data Challenge.”
The Civil Service Data Challenge has achieved some remarkable things – propelling two great ideas into delivery, while boosting the skills and careers of around 100 participating civil servants. But there is no alchemy behind its success: it simply taps into the vast unrealised potential of the civil service workforce, providing a channel for good ideas to receive the attention and development they deserve. And while the Challenge – now well into its second year – will continue to offer this route to implementation, the greater task facing civil service leaders is that of embedding its ethos and approach into business as usual.
“If you can just create the right environment, magic can happen,” says Vicki Chauhan. “It isn’t rocket science: we’ve just got to create the right structures around people.”
For more on the UK’s Civil Service Data Challenge, visit the dedicated website. If you think your country’s civil service could benefit from a similar programme, please contact the author via [email protected]
About Matt Ross
Matt is Global Government Forum's Contributing Editor, providing direction and support on topics, products and audience interests across GGF’s editorial, events and research operations. He has been a journalist and editor since 1995, beginning in motoring and travel journalism – and combining the two in a 30-month, 30-country 4x4 expedition funded by magazine photo-journalism. Between 2002 and 2008 he was Features Editor of Haymarket news magazine Regeneration & Renewal, covering urban regeneration, economic growth and community development; and from 2008 to 2014 he was the Editor of UK magazine and website Civil Service World, then Editorial Director for Public Sector – both at political publishing house Dods. He has also worked as Director of Communications at think tank the Institute for Government.
Related Posts
