Tackling high-quality test data generation at PyData's NHS Data Community Takeover

We were thrilled to host this month’s Leeds PyData event, featuring a special takeover by the NHS Data Community! A group of NHS data specialists joined forces with our PyData regulars to tackle the complex challenge of generating high-quality test data. Leeds PyData has grown to over 650 members, and the NHS Data Community is now over 250 members and is celebrating one year since its launch.

Background

Most NHS Organisations handle inpatient data in one form or another. Trusts collect this information to inform local decision-making and submit this nationally via our Inpatient CDS  (Commissioning dataset)  for consumption nationally and onward decision making at scale.

This data is also used for costing purposes, using the NHS’s “grouper” application which determines how much Trusts are paid for patient activities.

I have personal experience working with these data sets from the work I’ve done when I was at NHS Digital, with the Secondary Use Services Platform (SUS+ and Hippo worked on a HES migration) so I was very aware of the issues around high-quality test data. 

This data is used for myriad use cases such as planning, workforce arrangements, flow, process improvement and patient care. It’s typically presented in PowerBI or alike but flows downstream to various systems for onward consumption.

Therefore, having a strong data model with tight business rules is paramount, given the reach and importance of this data.

Unfortunately, quality test data is non-existent as some (if not most) NHS organisations, especially data that reflects real-life use cases and the depth of complexity the real world represents. This is particularly challenging given the weight of this dataset, it’s real people’s healthcare data and it deserves the respect we give it. 

Having quality test data would allow us to build & develop high quality & sharable data models, transformations, analytics, machine learning algorithms etc. which would be great for the NHS’s employees and patients alike.

The event

The event kicked off with teams discussing the barriers to quality data creation, especially within the NHS where data privacy and accuracy are paramount. Participants explored both generative and testing tools to create batches of synthetic data, generate parameters, and examine data schemas.

The high level process looked like this, we tried not to be overly prescriptive to allow the teams to define and try to work on a slice of the problem space.

The teams were provided with a sample schema, some tool ideas and places to collaborate and explore ideas.

The tools we identified were:

• Faker – https://faker.readthedocs.io/en/master/
• Mimesis –  https://github.com/lk-geimfari/mimesis
• NHS AI Lab – https://github.com/nhsx/skunkworks-synthetic-data
• Polyfactory – https://polyfactory.litestar.dev/
• Synth – https://www.getsynth.com/

Along with a sample schema for hospital data supplied by the NHS Data Community.

Identified challenges

One of the main challenges was creating “real-looking” representative data without compromising accuracy. Randomisation often led to problems in relational fields, causing inconsistencies and illogical data points. Teams grappled with ensuring intra-record consistency, handling complex inpatient data (episodes, spells, stays), and maintaining realistic distributions of features like gender ratios and dates of birth.

Solutions explored

• Team Mimosa: Focused on live coding based on team requirements. They used the Mimesis library (hence the team name) to generate data classes, emphasizing a balance between human control and machine automation. They used the public ORD API to enhance the data quality with references to real GP practices and discussed the importance of segmenting problem spaces for efficient solutions.
• Team “Why Fake Data?”: Delved into why synthetic data is necessary and used the Faker library to model data distributions accurately. They worked on feeding statistical parameters into the system, such as UK population statistics, to create scalable and useful synthetic data while ensuring logical consistency.
• Team “The Synthesisers”: Experimented with Synth, Faker, and ChatGPT to generate relationships between tables. They tackled interconnecting data problems by converting schemas into JSON and tweaking parameters to achieve consistent data across dimensions.
• Team “Jon and the Grinches”: Concentrated on adding noise to data to simulate real-world inaccuracies. They explored the complexity of inpatient data and aimed for intra-record consistency. The team considered using neural networks and variational autoencoders to generate synthetic data but noted potential vulnerabilities in passing real data through encoded layers.

Closing thoughts

We really enjoyed the hack, particularly joining forces with PyData. We learnt a lot from each other, enjoyed the opportunity to collaborate and certainly left with some food for thought and some new skills and tools under our belt. Hopefully this is just the start!

— Jonathan Gee, Mid Yorkshire NHS Trust, Head of Data.

It was inspiring to see everyone dive deep into the problem, think laterally, and collaborate effectively to develop innovative solutions. 

The big themes that came out of it were:

• Understanding of what the purpose of the test data is for: Are you trying to check something works, or check it doesn’t break (so ensuring poor quality data/property-based testing) or check it scales. Build specific solutions to address the purpose.
• No shortcuts: Intra-record consistency and relational consistency for secondary care data are trivial. It’s not a surprise,  but there are no shortcuts even with the tools we were using. A theme of the event was “wouldn’t this be easier if we took live data and pseudonymised it”.
• Try blended approaches: We found some tools were better at generating randomised “junk” data that was consistent, and others were good at generating realistic looking data – these can be combined.

A big thank you to our team captains, participants across all four teams, and the organisers who made this event a success!

Special thanks to:

• Jonathan Gee (Mid Yorks head of data/ NHS data community organiser)
• Mark Akers (Mid yorks data / NHS data community organiser) 
• Ed Hiley
• Adam Ellis
• Marius Atkinson
• Matt Wilkes
• Emma Thornton
• Suze Hawkins
• Roger G Coram