# XQA-Chest **XQA-Chest** is an expert-annotated chest X-ray (CXR) dataset created to support research on clinical AI evaluation, radiology report labelling, pathology classification, and image quality assessment. The dataset contains frontal chest X-ray images from Cambridge University Hospitals NHS Foundation Trust, associated radiology reports where available, and expert annotations from clinical readers. XQA-Chest is designed to facilitate the study of inter-annotator variability, discrepancies between image-based and report-based findings, and the relationship between expert diagnostic image quality ratings and automated image quality metrics. The dataset includes image-level labels, bounding boxes, report-level annotations, and image quality ratings based on a standardised set of clinical findings and diagnostic quality criteria. The dataset is split into two components: 1. **XQA-RR** — paired image-based and report-derived annotations for CXR pathology classification. 2. **XQA-IQA** — expert diagnostic image quality ratings for reference and degraded CXR images. ## Dataset components ### XQA-RR: Radiology report and image annotations XQA-RR contains annotations for **650 frontal chest X-rays** and their corresponding radiology reports. The studies were acquired at Cambridge University Hospitals NHS Foundation Trust between 2019 and 2021. Each study consists of a frontal chest radiograph, either AP or PA, and a paired radiology report. The dataset was sampled to include both normal and abnormal examinations across a standard set of thoracic findings aligned with CheXpert and MIMIC-CXR conventions. Cases with missing reports, non-diagnostic image quality, or duplicate examinations from the same encounter were excluded. Images and reports were annotated independently. For image annotation, each radiograph was reviewed in an image-only setting without access to the report or clinical history. Each finding was labelled as negative, uncertain, or positive, with optional bounding box localisation. Disagreements between the two image annotators were resolved by a senior radiologist, producing conflict-resolved image-derived labels. Reports were annotated separately using the same finding categories. Report annotators labelled each finding as negative, uncertain, positive, or not mentioned, based on the report text. ### XQA-IQA: Image quality assessment XQA-IQA provides expert diagnostic image quality ratings for frontal chest X-rays. This component contains **323 reference frontal CXR images** from Cambridge University Hospitals clinical scanners and **1,571 degraded image variants**. Degraded variants were generated using controlled changes to scanner visualisation settings, such as contrast and sharpness, to create images with different types and levels of diagnostic quality degradation. Each degraded image was rated independently by expert readers, including radiologists and reporting radiographers, using a four-point Likert scale reflecting diagnostic usability. These ratings support comparison between expert clinical judgement and automated image quality assessment metrics, including full-reference and no-reference measures such as PSNR, SSIM variants, HaarPSI, SNR, NIQE, PIQE, LPIPS, and related metrics. The released IQA annotations include raw expert ratings and aggregate quality scores. To account for differences in individual reader scoring behaviour, ratings can be normalised per annotator using z-score standardisation and aggregated into a z-score Mean Opinion Score (z-MOS). This enables rank-based comparison between human expert assessment and automated IQA metric scores. ## Dataset layout ```text XQA_IQA_ratings_annotations.csv XQA_IQA_images_part_1.zip ├── degraded/ │ ├── 001_iqa_1.png │ ├── 001_iqa_2.png │ └── ... ├── reference/ │ ├── 001_iqa.png └── └── ... ... XQA_IQA_images_part_4.zip ├── degraded/ │ └── ... ├── reference/ └── └── ... XQA_RR_radiology_reports.csv XQA_RR_annotations.zip ├── reports_annotator_1.csv ├── reports_annotator_2.csv ├── images_annotator_1.csv ├── images_annotator_2.csv ├── images_conflict_resolved.csv ├── bounding_boxes_annotator_1.json └── bounding_boxes_annotator_2.json XQA_RR_images_part_1.zip ├── 001.png └── ... XQA_RR_images_part_2.zip └── ... ``` ## Label set XQA-RR uses a 14-label chest X-ray finding scheme aligned with MIMIC-CXR and CheXpert conventions: * No Finding * Enlarged cardiomediastinum * Cardiomegaly * Lung lesion * Lung opacity * Edema / Oedema * Consolidation * Pneumonia * Atelectasis * Pneumothorax * Pleural effusion * Pleural other * Fracture * Support devices ## Label encoding For image-level annotations, each label is encoded as: * `0` → Negative * `1` → Uncertain * `2` → Positive For report-level annotations, the same values are used, with an additional empty value: * `0` or `0.0` → Negative * `1` or `1.0` → Uncertain * `2` or `2.0` → Positive * empty cell → Not mentioned in the report An empty cell in report annotations does **not** mean negative. It means that the finding was not mentioned in the report. ## Image-level annotations Image-level annotations are provided by two expert image annotators and a conflict-resolved reference produced by an additional senior radiologist. **Files:** * `images_annotator_1.csv` * `images_annotator_2.csv` * `images_conflict_resolved.csv` **Example:** ```csv study_id,No Finding,Enlarged cardiomediastinum,Cardiomegaly,Lung lesion,Lung opacity,Oedema,Consolidation,Pneumonia,Atelectasis,Pneumothorax,Pleural effusion,Pleural other,Fracture,Support devices 001,2,0,0,0,0,0,0,0,0,0,0,0,0,0 002,0,0,2,0,2,0,0,0,0,0,0,0,0,0 ``` ## Report-level annotations Report-level annotations are provided for the findings section of each radiology report. Reports were annotated using the same 14-label finding scheme as the image annotations. **Files:** * `reports_annotator_1.csv` * `reports_annotator_2.csv` **Example:** ```csv study_id,No Finding,Enlarged cardiomediastinum,Cardiomegaly,Lung lesion,Lung opacity,Oedema,Consolidation,Pneumonia,Atelectasis,Pneumothorax,Pleural effusion,Pleural other,Fracture,Support devices 001,,,,,,,0.0,,,0.0,,,, 002,,1.0,1.0,,2.0,,2.0,,,0.0,0.0,,, 003,,,,,,,2.0,,,0.0,,,,2.0 ``` Although values may be stored as floats in the CSV files, they follow the same encoding scheme described above. ## Bounding box annotations Bounding boxes are provided for positively identified image findings where localisation was annotated. **Files:** * `bounding_boxes_annotator_1.json` * `bounding_boxes_annotator_2.json` **Example:** ```json [ { "study_id": "002", "bounding_boxes": { "Cardiomegaly": [ [1406.2, 1651.3, 1129.0, 361.6] ], "Lung opacity": [ [883.9, 1747.7, 518.3, 353.6], [2675.8, 1812.0, 397.8, 285.3] ] } }, { "study_id": "003", "bounding_boxes": { "Consolidation": [ [400.3, 1176.6, 786.7, 665.9] ], "Support devices": [ [1214.6, 376.1, 289.8, 462.4] ] } } ] ``` Each bounding box is stored in pixel coordinates in the format: ```text [x, y, width, height] ``` Missing labels indicate that no bounding boxes were provided for that finding in that study. ## Image quality assessment annotations The XQA-IQA component provides expert diagnostic image quality ratings for reference CXR images and degraded variants. **Files:** * `XQA_IQA_ratings_annotations.csv` * `XQA_IQA_images_part_1.zip` * `XQA_IQA_images_part_2.zip` * `XQA_IQA_images_part_3.zip` * `XQA_IQA_images_part_4.zip` **Image folders:** * `reference/` — original reference chest X-ray images * `degraded/` — degraded image variants generated from the reference images Each image quality rating is based on a four-point Likert scale reflecting diagnostic usability. Higher scores indicate better perceived diagnostic image quality. Where provided, aggregate scores summarise expert opinion across readers. The IQA component can be used to evaluate how well automated image quality metrics align with expert judgement of diagnostic usability. ## Recommended use cases XQA-Chest is suitable for: * Studying inter-annotator disagreement in CXR interpretation * Comparing image-based and report-derived pathology labels * Evaluating the effect of reference standard choice on model performance * Evaluating the effect of reference standard choice on model rankings * Training and evaluating multimodal medical AI models * Studying uncertainty and robustness in clinical machine learning * Evaluating CXR image quality assessment methods * Comparing automated IQA metrics against expert diagnostic usability ratings * Developing and testing radiology report labelling methods ## Important interpretation notes Report-derived labels and image-derived labels should not be treated as interchangeable. Report annotations reflect the contents of the radiology report, while image annotations reflect expert assessment of the image without access to report text or clinical history. A negative report-derived label means that the report explicitly describes the finding as absent. An empty report-derived label means that the finding was not mentioned. Image-level uncertainty reflects uncertainty in image interpretation. Report-level uncertainty reflects uncertainty expressed or implied in the report text. For many analyses, users should report results separately for image-derived and report-derived references rather than collapsing them into a single ground truth. ## Data protection and anonymisation The clinical data were anonymised or de-identified before release in accordance with relevant governance, ethical, and data-sharing requirements. Direct patient identifiers were removed. Dates and locations were removed from released metadata, and pseudonymisation codes were changed before sharing. Radiology reports were reviewed to ensure that patient-identifiable or sensitive data were not included. The dataset is intended for research use only. Users must not attempt to identify individuals or link the data to other sources for re-identification. ## Access Access to XQA-Chest is restricted and managed through the UK Data Service ReShare safeguarded data access route. Files are made available only to approved users registered with the UK Data Service and licensed under the UK Data Archive End User Licence conditions. Requests for access require depositor permission and review by Cambridge University Hospitals NHS Foundation Trust’s data access committee. Approved users must comply with all applicable ethical, governance, confidentiality, and data-use conditions. ## Citation If you use XQA-Chest, please cite the dataset repository. While the accompanying paper is not yet available, citing the repository alone is sufficient. Once the paper citation has been added to this README and the ReShare record, users should cite both the dataset repository and the accompanying paper. ### Dataset repository ```bibtex @misc{reshare858610, publisher = {UK Data Service}, author = {Cambridge University Hospitals NHS Foundation Trust and Panagiotis Fytas and Anna Breger}, title = {XQA-Chest: Expert-Annotated Chest X-ray Dataset for Pathology and Image Quality Assessment}, abstract = {Chest X-ray (CXR) machine learning relies heavily on automated evaluation using reference standards that aim to approximate clinical judgment. However, commonly used report-derived labels for pathology classification and generic image quality metrics for image reconstruction may not reliably reflect expert clinical assessment. XQA was collected to support systematic investigation of how evaluation choices affect model performance and ranking in both CXR pathology classification and image quality assessment (IQA). XQA is split into two complementary components. XQA-RR supports research on pathology classification and contains paired expert image-based and report-derived labels for thoracic findings from a clinical cohort at Cambridge University Hospitals NHS Foundation Trust. XQA-IQA supports research on diagnostic image quality assessment and contains expert ratings of CXR image quality for reference images and degraded image variants. Using XQA, we evaluate supervised image classifiers, self-supervised models, and fine-tuned vision-language models, including ResNet, DenseNet, MedKLIP, GLoRIA, and ConVIRT. The results show that changing the evaluation reference can substantially affect both estimated model performance and model rankings. In parallel, the alignment of IQA measures with expert judgment depends strongly on the choice of measure, with commonly used metrics such as SSIM and PSNR often failing to reflect expert assessments of diagnostic usability. XQA was developed to enable controlled comparison across image-based labels, report-derived labels, and image quality measures. The collection is intended to support research on clinical AI evaluation, CXR pathology classification, radiology report labelling, image quality assessment, and the clinical suitability of machine learning systems for CXR interpretation.}, url = {https://reshare.ukdataservice.ac.uk/858610/}, keywords = {Chest X-ray, CXR, medical imaging, radiology, clinical AI, machine learning, pathology classification, image quality assessment, expert annotation, report-derived labels, image-based labels, reference standards, model evaluation} } ``` ### Accompanying paper ```bibtex TODO: Add paper citation once available. ``` ## Contact For questions about the dataset or access conditions, please contact the dataset depositors through the ReShare platform. Primary contact: ```text Panagiotis Fytas University of Cambridge pf376@cam.ac.uk ``` ```text Anna Breger University of Vienna anna.breger@univie.ac.at ``` Cambridge University Hospitals NHS Foundation Trust point of contact: ```text Professor James H. F. Rudd University of Cambridge / Cambridge University Hospitals NHS Foundation Trust jhfr2@cam.ac.uk ``` ## Licence Access to XQA-Chest is governed by the UK Data Service ReShare safeguarded access conditions and the UK Data Service End User Licence. Licence: [UK Data Service End User Licence](https://ukdataservice.ac.uk//app/uploads/cd137-enduserlicence.pdf)