# Improving results and conclusions about harms in systematic reviews of drugs

> **NIH AHRQ R01** · UNIV OF NORTH CAROLINA CHAPEL HILL · 2024 · $400,000

## Abstract

PROJECT SUMMARY/ABSTRACT
Systematic reviews (SRs) of interventions might report inconsistent results and conclusions about patient
safety, and they might differ from other sources of patient safety information such as real-world evidence
(RWE) and drug labels. SRs underpin clinical guidelines by the US Preventive Services Taskforce,
professional societies, and national insurers. Guidance says that SRs should address both harms and benefits
of interventions, yet most SRs are designed to assess benefits and consider harms as secondary outcomes.
Moreover, guidance for conducting and reporting SRs includes scant advice about the different types of data,
and different study designs, for harms compared with benefits. In a preliminary study, we found that 70 SRs
reached different conclusions about the harms of a single drug, and conclusions in SRs were not consistent
with the drug label. Inconsistency suggests that some conclusions about harms in SRs might be wrong. In this
study, we propose to use innovative methods, including natural language processing (NLP), to assess the
consistency of methods, results, and conclusions about patient safety in >19,000 SRs published from 2004—
2024. Advances in NLP have created new opportunities for evidence synthesis using models that can identify
and interpret text for extraction (e.g., from journal articles). Specifically, we will leverage state-of-the-art NLP
tools and methods, including those based on large language models, to extract and synthesize evidence on
harms from SRs, RWE studies, and drug labels at a granularity that has not been attempted previously. Then,
we will compare SRs that include the same drug (we estimate there are >4,000 overlapping SRs), and we will
compare SRs with RWE and drug labels. We will investigate whether SRs use methods for assessing harms
that might lead to inconsistent conclusions about patient safety. We will also investigate whether SRs use
methods that lead to differences compared with RWE and drug labels. For example, SRs often exclude RWE.
Moreover, SRs are usually limited to one health problem; by comparison, drug labels include harms that could
occur when people take the drug to treat different health problems. Highlighting specific drugs for which SRs
might misrepresent risk of harm, and identifying why certain conclusions might be wrong, could immediately
help patients and healthcare providers make better-informed decisions. Using our findings, we will develop
improved guidance for synthesizing and reporting harms in SRs, which should ultimately improve conclusions
about patient safety in SRs and clinical guidelines. Specifically, we aim to answer the follow questions:
Aim 1. Are results and conclusions about patient safety in SRs of the same drug consistent?
Aim 2. Do SRs use RWE, and do the results and conclusions of SRs address harms found in RWE?
Aim 3. Is information about harms in SRs consistent with information in drug labels?
Aim 4. What recommendations could i...

## Key facts

- **NIH application ID:** 11048663
- **Project number:** 1R01HS029877-01A1
- **Recipient organization:** UNIV OF NORTH CAROLINA CHAPEL HILL
- **Principal Investigator:** Halil Kilicoglu
- **Activity code:** R01 (R01, R21, SBIR, etc.)
- **Funding institute:** AHRQ
- **Fiscal year:** 2024
- **Award amount:** $400,000
- **Award type:** 1
- **Project period:** 2024-09-30 → 2029-08-31

## Primary source

NIH RePORTER: https://reporter.nih.gov/project-details/11048663

## Citation

> US National Institutes of Health, RePORTER application 11048663, Improving results and conclusions about harms in systematic reviews of drugs (1R01HS029877-01A1). Retrieved via MORVS 2026-10-26 from https://api.morvs.ai/grant/nih/11048663. Licensed CC0.

---

*[NIH grants dataset](/datasets/nih-grants) · CC0 1.0*