According to a recent study published in JAMA Oncology, the findings of published meta-analyses on cancer treatments can be recalculated only in 2 cases out of 3. This does not imply that the findings are not true, the authors say, but confirms that issues of reproducibility even in systematic reviews and meta-analyses are quite common.
The authors of the study led by Cole Wayant of the Oklahoma State University Center for Health Sciences in Tulsa, examined 154 meta-analyses from systematic reviews in oncology: only 2.375 of the 3.696 meta-analytic estimates (64.3%) were reproducible in theory. In the other cases, the authors of the meta-analyses did not provide enough details on methods and data to allow readers to independently recalculate the findings. Noteworthy, all the meta-analyses included in the evaluation were cited by the National Comprehensive Cancer Network (NCCN) guidelines for the treatment of cancer by site.
Reproducible research practices are essential to guarantee that the findings are trustworthy. In an accompanying editorial, Joseph M. Unger, biostatistician at the Fred Hutchinson Cancer Research Center (Seattle, WA, USA), points out that “trust in reported evidence can be viewed as mediating the relationship between proof of treatment efficacy and adoption of new treatments into community care”. In fact, systematic reviews and meta-analyses of phase 3 clinical trials constitute level 1 evidence for clinical practice guidelines. If oncologists distrust study findings, they may be reluctant to follow such guidelines and to adopt the newly proposed treatments.
The reproducibility of meta-analyses could be ensured by the full adherence to specific guidelines, such as the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines (an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses) and Cochrane guidelines.
Interestingly, the authors of the research discovered that forest plot meta-analyses had a higher reproducibility than other meta-analyses. “We recommend journals encourage authors to present all meta-analyses in figures because standard graphical output for meta-analyses in most statistical packages includes a list of included studies and numerical point estimates” they write. The ability to reproduce meta-analytic effects would lead to more reliable results, and, ultimately, to improve clinical care.
Leave a Reply