Background: The RIGHT-Ad@pt checklist is an international standard to assess and guide the reporting of adapted guidelines. However, the assessment process is repetitive and lengthy. Large language models (LLMs) demonstrate their capabilities to process and generate human language, offering a promising solution to improve efficiency.

Objective: To develop Ad@pt-AI, a prompt-driven automated tool leveraging an existing LLM to assess the reporting quality of adapted guidelines based on the RIGHT-Ad@pt checklist.

Methods: We will develop and validate Ad@pt-AI through multiple processes: 1) establishing a prototype using an existing LLM and creating initial prompts based on the RIGHT-Ad@pt checklist and user guide, enabling the AI to assess 34 checklist items and retrieve relevant quotes; 2) iteratively optimizing Ad@pt-AI using prompt engineering until an F1 score of 0.75 is achieved; 3) conducting user testing of the optimized Ad@pt-AI in internal and external settings. We built a ground-truth set by annotating 120 adapted guidelines identified from previous scoping reviews. In user testing, each sample will be assessed by three groups: 1) Ad@pt-AI model, 2) a combined scholar evaluation team (internal and external scholars), and 3) a hybrid scholar-AI approach.

Results: We have established the Ad@pt-AI prototype and are currently finishing the annotation. We will report the optimization process and results at the GIN conference 2025.

Discussion for scientific abstracts: Ad@pt-AI can save time and resources while ensuring substantial accuracy and consistency in assessing the reporting of adapted guidelines. Its establishment process will also inform future automation in adapting clinical practice guidelines.

Background: Artificial intelligence (AI)-based platforms specialized on the identification of scientific publications may potentially support searches in evidence synthesis studies.
Objective: To evaluate how AI-based platforms can support the search for primary studies in the context of evidence synthesis and to propose a structured, reproducible and platform-agnostic methodological framework.
Methods: As our main case study, we queried three AI-based platforms to identify randomised controlled trials (RCTs) investigating mometasone in patients with rhinitis. We evaluated the performance of several different strategies (i.e., prompting strategies and search approaches), comparing how many eligible RCTs we identified using AI-based platforms versus in a systematic search in multiple electronic bibliographic databases. We also compared meta-analytical results obtained with the RCTs identified by querying AI-based platforms versus those obtained in the context of a systematic review. We developed a methodological framework and a reporting checklist for the use of these AI-based platforms.
Results: A strategy involving searching multiple AI-based platforms identified 56.3% of eligible RCTs investigating mometasone in patients with rhinitis (85.7% of all full papers with DOI). Meta-analytical estimates were similar when considering only the RCTs identified using AI-based strategies versus those identified in the context of a systematic review. Consistent results were obtained when querying AI-based platforms one month later.
Discussion for scientific abstracts: Querying AI-based tools can support searches in evidence synthesis studies, complementing classical methods or helping to test the comprehensiveness of search queries for electronic bibliographic databases. However, AI-based tools can have shortcomings for identification of gray literature.

Background. Data extraction is a critical but error-prone and labor-intensive task in evidence synthesis. Unlike other artificial intelligence (AI) technologies, large language models (LLMs) do not require labeled training data for data extraction.

Objective. To compare an AI-assisted to a traditional human-led data extraction process.

Methods. Study within reviews (SWAR) based on six ongoing systematic reviews of the US Agency for Healthcare Research and Quality. Within each review, we utilized a prospective, parallel group comparison of a traditional human-led data extraction approach with an AI-assisted approach. In the AI-assisted approach, the LLM Claude conducted initial data extraction, which was verified by a human reviewer. Blinded data adjudicators compared the results of the two data extraction approaches and resolved discrepancies be checking study reports.

Results. The six systematic reviews of the SWAR contributed 9,341 data elements, extracted from 63 randomized and non-randomized studies. Concordance between the two methods was 77.2%. The accuracy of the AI-assisted approach compared with the adjudicated reference standard was 91.0%, with a recall of 89.6% and a precision of 98.9%. The AI-assisted approach had fewer incorrect extractions (9.0% vs. 11.0%) and similar risks of major errors (2.5% vs. 2.7%) compared to the traditional human-led method, with an average time saving of 44 minutes per study. Missed data items were the most frequent errors in both approaches.

Conclusion. The use of an LLM can improve the efficiency and accuracy of data extraction in evidence synthesis. Results reinforce previous findings that human-led data extraction is prone to errors.

Background: Systematic reviews (SRs) are the highest standard of evidence, guiding clinical guidelines, policy, and research. However, labor-intensive screening delays timely information synthesis.

Objective: To evaluate the performance of large language model (LLM)-driven screening methods compared to dual human screening, the current gold standard.

Methods: We compared the performance of optimized LLM prompts against human reviewers across five SRs covering prevalence, intervention benefits, diagnostic accuracy, and prognosis questions. Single human (screening all available titles/abstracts) and dual human (sequential dual screening of titles/abstracts and full texts) workflows were evaluated. The original review authors' inclusion/exclusion labels were the reference standard.

Results: LLM-based abstract screening had performance (mean sensitivity: 98.5%, specificity: 85.1%) superior to single human reviewers (mean sensitivity: 89.3%, specificity: 92.9%), achieving similar accuracy in four SRs (p>0.05) and exceeded human sensitivity in all five, with significantly higher sensitivity in three reviews (p<0.05). LLM-based full-text screening (mean sensitivity 97.4%, specificity 91.1%) also outperformed dual human reviewer workflows (mean sensitivity 75.1%, specificity 97.8%), with significantly higher sensitivity in four SRs (p<0.05). Traditional human screening for 7,000 articles took 530 hours and cost $10,000, while our method completed screening in one day for $430.

Discussion: LLMs can match or exceed human performance in SR screening, while significantly reducing time and cost. LLM-driven automation can streamline screening, allowing human reviewers to focus on deeper scientific analysis and accelerating the synthesis of study conclusions.

Background: Plain language summaries enhance understanding of recommendations by providing clear, accessible content. At GIN 2022, we demonstrated the feasibility of automating creation of these summaries, substantially improving readability. However, challenges in factual accuracy remained.

Objective: Evaluate if advanced AI models further improve readability and factuality in generating plain language summaries compared to GPT-3.

Methods: We revisited a dataset of 444 recommendations accompanied by "additional information." We generated plain language summaries using Claude 3.7 Sonnet. Readability was assessed via the Flesch Reading Ease score. Three reviewers qualitatively evaluated readability and factual accuracy on a subset of 40 randomly chosen summaries, comparing the results to the previous GPT-3-based system.

Results: Claude 3.7 Sonnet markedly improved readability compared to GPT-3, increasing the mean Flesch Reading Ease score for recommendations by 42.53 points from baseline (9.33 to 51.86), and 13.87 from GPT-3 (CI: (11.96, 15.78)). Qualitative evaluation showed substantial readability improvements, with 38 out of 40 recommendations rated as improved or much improved, compared to 25 in 2022. Regarding factual accuracy, Claude 3.7 Sonnet significantly reduced outright hallucinations compared to GPT-3 (1 vs. 4 cases). However, it frequently introduced oversimplifications or misinterpretations (7 cases), effectively doubling the overall rate of summaries requiring fact-checking (8 vs. 4 previously).

Discussion: While the last three years brought enormous improvements in terms of model quality, assuring factual accuracy still remains a nontrivial task. Therefore, this area requires further careful research and we advise against using these tools without human supervision.