UK Voices: New Methods for Understanding the Impact of Social Change on Individual Lives, 2024

Byrne, George and Elliott, Jane and Friese, Carrie (2025). UK Voices: New Methods for Understanding the Impact of Social Change on Individual Lives, 2024. [Data Collection]. Colchester, Essex: UK Data Service. 10.5255/UKDA-SN-857940

The UK Voices Pilot Project explored how large-scale qualitative data could be collected and analysed to deepen our understanding of how people in the UK are experiencing and responding to rapid social change. It sought to evaluate both the feasibility of developing a representative qualitative resource and the potential of using Artificial Intelligence (AI), specifically Natural Language Processing (NLP) and Large Language Models (LLMs), to support qualitative data analysis. The project was structured around two interconnected work packages.
Work Package 1 (WP1) focused on testing a biographical interview protocol and exploring how such an approach could be scaled nationally. During a two-stage process, the first led by LSE and the second by National Centre for Social Research (NatCen), the team conducted 51 interviews across a diverse sample. The flexible biographical approach, beginning with the question “Can you tell me your life story?”, proved successful in eliciting detailed, reflective narratives. However, the pilot also revealed challenges relating to recruitment and response rates, particularly during the panel-based phase led by NatCen. These findings are crucial for informing future sampling strategies, interviewer training, and fieldwork planning. Despite recruitment difficulties, the interviews produced rich data relevant to a wide range of different social science research questions. This demonstrates that this form of data collection is both meaningful and achievable at scale with appropriate design and resourcing.
Work Package 2 (WP2) focused on developing and evaluating an interface designed to support researchers working with large corpora of qualitative interview data. The tool, QualQuest, was iteratively developed and tested using two datasets: 12 UK Voices scoping interviews and 73 transcripts from the Welfare at a Social Distance (WASD) Project. Early attempts to integrate LLM-based summarisation were found to be unreliable. A Retrieval Augmented Generation (RAG) architecture was subsequently adopted, enabling the more accurate retrieval of thematically relevant direct quotations in response to natural language queries. Structured testing showed that the final version performed well in terms of recall but less so in terms of precision QualQuest proved especially useful in helping researchers identify relevant transcripts and thematic content quickly, though false positives remained an area for future refinement.
Outreach and Knowledge Exchange Activities included convening regular meetings with an international advisory board (IAB), which has now evolved into a global network of researchers working on large-scale qualitative data projects. We also presented our work at academic events and held workshops with researchers from a range of fields and with non-academic stakeholders, including hosting an expert symposium on AI and Qualitative Research in Collaboration with the ESRC Centre for Sociodigital Futures. This showed that there was enthusiasm for the large-scale collection of biographical qualitative interviews while providing opportunities for critical reflection on data collection and analysis methods.
The UK Voices Pilot met its core objectives by demonstrating the feasibility of collecting and analysing large-scale qualitative data and producing valuable findings on both methodological design and using automation to retrieve relevant data for large qualitative corpora. While it should be noted that a pilot project of this size does not represent a perfect simulation of a much larger and more complex project, the work done lays a strong foundation for future investment in national qualitative infrastructure, including the potential for a large, repeated cross-sectional panel of biographical interviews supported by automated data retrieval tools.

Data description (abstract)

This data consists of 12 semi-strutured, participant led, qualitative biographical interviews, conducted during the scoping phase of the ESRC UK Voices Pilot Project. The aim of the interviews was to test a Topic Guide for producing a large-scale qualitative general purpose data set that provides insights into people's experiences and life in the UK. It is linked to an additional dataset of 38 interviews, conducted during the larger pilot.

UK Voices developed a methodology for building a general-use qualitative interview dataset to provide insights into how the UK population experiences and navigates accelerated social changes, including climate change, political polarisation, and inequality.

The project is piloted methods for large-scale qualitative data collection and analysis to enhance the UK’s social science research infrastructure. Funded by ESRC and running from October 2024 to June 2025, the project was organised into two main work packages. The data in this deposit was collected during work package 1, which first tested qualitative interview techniques for gathering in-depth data from a broad sample of the population, refining methods for large-scale qualitative research. This built on existing projects, such as the American Voices Project, to develop a methodology tailored to the UK context.

The second work package explored the use of generative AI and Natural Language Processing (NLP) tools to streamline the analysis of the extensive qualitative data collected. By leveraging these tools to assist in identifying and analysing key sections of text, this phase addressed some of the challenges regarding scaling qualitative research, which is often limited to small numbers of participants. The project ultimately aimed to create a flexible research platform that merges qualitative methods with innovative software tools, enabling more efficient analysis and broader exploration of critical social issues. The findings from this pilot have been shared with the wider social science community through reports, workshops, and conferences, laying the groundwork for future large-scale and cross-national qualitative research.

Downloads

Altmetric