Study Reveals Poor Reporting Standards in COVID-19 AI Models

A recent systematic review published in Diagnostic and Prognostic Research has highlighted a significant concern regarding the quality of reporting in machine learning-based prognostic models for COVID-19. Conducted by Ioannis Partheniadis and his team, this comprehensive study assessed 53 research articles that described a total of 71 distinct prognostic models, comparing adherence to reporting guidelines between machine learning (ML) and traditional statistical methods.

Key Findings from the Research

Using the TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) checklist for conventional models and the TRIPOD + AI extension for machine learning models, the researchers found that the adherence rates to reporting standards were notably low in both methodologies. However, the discrepancies were significant: the mean adherence rate for machine learning models was recorded at just 28.4% compared to 38.1% for conventional models.

The Importance of Transparent Reporting

The central aim of this research was to evaluate how completely prognostic prediction models for COVID-19 report their findings. Prognostic models play a crucial role in predicting specific health outcomes, such as the likelihood of severe illness or death, based on patient data. Throughout the pandemic, a surge in the development of these models occurred, driven by extensive datasets and improved access to analytical software. Nevertheless, the effectiveness of these models in clinical settings hinges on their ability to provide clear and precise reporting.

Incomplete reporting can significantly hinder the efforts of other researchers who wish to replicate findings, interpret data accurately, or further validate the models for clinical applicability.

Research Methodology

The investigation involved thorough searches of databases such as MEDLINE, Epistemonikos.org, and Scopus, extending to July 31, 2024. The research team identified 1,529 records across 17 systematic reviews, systematically narrowing this down to 53 studies that met their criteria. They focused on studies that included model development along with internal and external validation. Importantly, they registered their review protocol in advance on the Open Science Framework, ensuring a clear methodological framework before any data extraction began.

Model Characteristics and Reporting Gaps

The included models were almost evenly divided, with 58.5% being traditional statistical methods and 41.5% being machine learning approaches. These models primarily targeted patients hospitalized with confirmed COVID-19 cases, often drawing data from single-country sources like China and the United States. Common factors analyzed included age, levels of C-reactive protein, and oxygen saturation. The critical outcomes most frequently projected were in-hospital mortality rates and severe illness incidents.

Upon evaluating compliance with the TRIPOD and TRIPOD + AI guidelines, the results showed a concerning low adherence rate across the board. Notably, not a single study met the complete abstract reporting requirements. Only 29% of conventional model studies and 13.6% of machine learning studies included adequate titles. Moreover, no studies provided full sample size calculations, a lack that raises concerns over potential overfitting.

Method reporting displayed numerous weaknesses in both types of models, particularly in outlining data preprocessing methods, defining predictors and outcomes, and addressing missing data challenges. Such inadequacies threaten the reproducibility of these models, leaving clinicians without the necessary tools to evaluate their applicability to diverse patient populations.

Adherence Differences and Challenges

Distinct differences surfaced in adherence to reporting guidelines between the two methodologies. For example, the data source reporting in traditional models had a compliance rate of 90.3%, contrasting sharply with a mere 22.7% for machine learning models. Conversely, predictor selection and preprocessing were reported at a higher rate in machine learning models (50%) compared to traditional ones (6.5%). Overall, adherence to crucial components, such as predictor definitions and study blinding, was alarmingly low across all models.

The researchers also noted that no study justified or calculated its sample size, an essential aspect of robust study design. Additionally, there was a complete lack of adherence when it came to reporting the stability and performance of ML models, which significantly impairs their reliability.

Implications for Future Research

The overall mean adherence was recorded at only 34.07%, with traditional models outperforming machine learning methods. The authors highlight that the introduction of the TRIPOD + AI guidelines—launched in April 2024—could explain lower compliance rates for AI-based studies published prior to that date.

Concerns surrounding transparency and accessibility of final model data were pronounced, as only 12.9% of studies fully made their final model available, indicating barriers to replication and validation efforts. The authors emphasized the urgent need for improved adherence to reporting guidelines, advocating for stronger editorial oversight and awareness among researchers regarding the importance of thorough reporting practices.

In conclusion, the findings underscore the necessity for the research community to rigorously apply TRIPOD or TRIPOD + AI guidelines to enhance the quality and usability of predictive models, ultimately contributing to better clinical decision-making.