(119) AI Meets Evidence Synthesis: A Comparative Review of AI Performance in Systematic Literature Screening

Background: : Artificial intelligence (AI) based tools are used to assist with the time-consuming screening process and support systematic literature reviews (SLRs). The AI models vary on their designs--ranging from general-purpose foundational models to custom or disease-state-specific pipelines to tools purpose-built for systematic reviews.

Objective: : This AI-assisted review attempts to methodically analyze and compare the performance of different AI approaches used for SLR screening: foundational models, custom-built systems, and SLR-specific platforms. In addition to offering recommendations for choosing AI solutions for systematic review tasks, the objective is to evaluate which tools are most appropriate for scalable, high-quality evidence synthesis.

Methods: : We included all studies reporting accuracy statistics for AI-based screening for SLRs. This review was completed in AutoLit (Nested Knowledge). An AI-driven Smart Search identified 3,251 candidate studies, of which 20 were included based on Robot Screener-assisted screening. Accuracy, sensitivity, specificity, and F1 scores were extracted from underlying studies and compared across model types qualitatively and with summary/descriptive statistics.

Results: : The performance of various AI techniques for screening systematic reviews differs widely. Foundational models (GPT-4, Claude-3.5, Gemini-1.5, BONSAI + KRS) averaged 81.7% accuracy and 0.77 F1 score; custom models (NLP-assisted classifier, Deep learning classifier, FastText + SVM with AL, pBERT) show great metrics show high sensitivity in some cases (e.g., up to 98.7%), but had average 81.2% accuracy and 0.50 F1 score. Tools designed specifically for systematic reviews, while most included a human-curation step, provided the best performance, exhibiting 99% accuracy on average and 96.1% sensitivity, though F1 was not available.

Conclusions: : Purpose-built SLR technologies are more accurate and more appropriate for large-scale screening in SLRs than foundational models and custom models, which exhibit varied performance and may lack specialization in the SLR screening task. Curation may be needed in any AI-assisted methodology, but AI tools displayed consistently high accuracy and sensitivity.

Sponsorship:: None.