The scandal of unreported trials has been known about for decades, prompting a variety of initiatives, legislative changes and campaigns. Sophie Fessl asks what impact these have had on reporting practices, and how far patients and clinicians can be confident that the evidence they can now access tells the full story.
In 2010, Alessandro Liberati, former director of the Italian Cochrane Center, explained his struggle to decide on his treatment options for multiple myeloma. “When I had to decide [in 2003] whether to have a second bone-marrow transplant, I found there were four trials that might have answered my questions, but I was forced to make my decision without knowing the results because, although the trials had been completed some time before, they had not been properly published… I believe that research results must be seen as a public good that belongs to the community – especially patients.”
Sixteen years after Liberati was frustrated by non-published trials, it remains the case that far from all studies make their results public after completion. Slightly less than half of all clinical trials conducted in Europe posted summary results on the European Clinical Trials Register, according to an analysis published in the BMJ in 2018 (BMJ 362:k3218). And while commercial trials have a publication rate of 68.1%, with just 11% the rate is much worse for non-commercial trials (ibid).
The problem with secrets in medicine
Non-publication affects everyone, says Till Bruckner, founder of UK-based transparency advocacy organisation TranspariMed. “On the one hand it makes it really hard to assess whether treatments are safe and effective. On the other hand, a huge amount of research is going to waste. The same trials might be duplicated several times, with patients volunteering hundreds of hours of their time to participate in clinical trials.
“And patients also do this because they want to help scientists find new treatments. When results are not published, it’s just a betrayal of patient rights.”
An unpublished trial doesn’t just let the invested funds go to waste, but may lead to a duplication of essentially the same trial and the same cost.
How much evidence remains hidden?
This ‘filing drawer problem’ cuts to the heart of evidence-based medicine. Or, to paraphrase Bruckner’s first point, how evidence-based is evidence-based medicine, if we don’t have all the evidence? Withholding of results, or a selective publication of positive results, has affected both clinical decision-making and health technology assessment.
IQWiG, the German Institute for Quality and Efficiency in Health Care, wasn’t able to draw a conclusion about the harm or benefit of stem cell transplantations for the treatment of multiple myeloma, as 10 years after their completion, the results of three large trials had not been published, says IQWiG’s director Jürgen Windeler.
“We get a distorted view of reality. For one trial, which was conducted in Germany with public money and is essentially finished, we cannot get access to the results.” But the situation can be handled differently – an Australian research group provided individual patient data to IQWiG, and with additional analyses, IQWiG arrived at a positive conclusion in the assessment of an added benefit.
Jörg Meerpohl, a paediatric oncologist and director of the Institute for Evidence in Medicine, at the University of Freiburg, Germany, points to several known sources of evidence distortions.
“The results of unpublished studies are, tendentially, not as positive as the results of published studies”
“We know that a substantial proportion of trial results is not reported. We also know that the results of unpublished studies are, tendentially, not as positive as the results of published studies. Therefore, there can be situations – and they are known to have occurred in the past years – in which systematic reviews of published studies came to a different result, and therefore indicated a different conclusion, than when, at a later time point, the results of all studies that had actually been carried out were looked at.”
These examples included the positive assessment of oseltamivir, which led to several billion dollars being spent on Tamiflu, with questionable effectiveness.
A question of ethics
Apart from the scientific dimension, the problem of non-publication has an ethical dimension, as Roger Wilson, honorary president of Sarcoma Patients Euronet, points out. “You have to ask, why do patients enter trials? One reason, of course, is because they are looking for a treatment which is better than the standard of care that exists at the moment. And the second reason is altruistic: they want to help other patients, they want to improve the standard of care, for the benefit of everyone.”
Shining a light on dark data
The FDAAA tracker site publishes details of trials registered on the US ClinicalTrials.gov website that fail to report their results on time. This screenshot, taken on 16 May 2019, shows that on 14 May 2019, the time of the latest update, 65.6% of trials had reported by their due date. It also shows the total amount the US regulators, the FDA, could have fined sponsors who are late in reporting (more than $3 billion dollars) and how much the FDA has actually imposed in fines ($0). The site was built by the Evidence-Based Medicine DataLab at the University of Oxford, and is run as part of the AllTrials campaign.
Source: https://fdaaa.trialstracker.net/
Wilson argues that a moral contract is established when a patient enters a trial. “When you give a patient the option of entering a trial, you, the clinician making that offer, are taking on the moral responsibility for addressing that altruistic ambition of the patient. And that altruism is only going to be met if the trial is published.”
Legislative steps have been taken
The revised Declaration of Helsinki from 2013 notes the ethical obligation to report clinical trial data, whether positive or negative. Legislators have recognised that non-publication of trial results is a problem, and have taken – some – steps to address it. In the US, the FDA Amendments Act 2007 (FDAAA), requires sponsors to post results on the ClinicalTrials.gov database within 12 months of completion. However, this requirement only extends to certain types of trials. The situation in Europe appears even murkier. Several interviewees disagreed over whether there are any legal requirement for trial sponsors to post summary results to the EU Clinical Trials Register (EUCTR), and if so, whether such a requirement would extend to all types of clinical trials.
Academic trials have a much worse track record than industry-led trials
The analysis published in the BMJ in 2018 examined the publication of trial results on the EU Clinical Trials Register on the basis that “following the 2012 EC guideline 2012/c302/03, sponsors must ensure that all trials registered on EUCTR since 2004 disclose their results to the EMA within 12 months of trial completion; phase I trials are exempt unless they are denoted as being part of a paediatric investigation plan.”
At the time the study was published, only 49.5% of trials where results were due had posted results to the EUCTR (BMJ 2018, 362:k3218). The real reporting rate might, however, be even worse: 29.4% of trials listed as completed did not include a completion date, although required, so the authors could not assess whether results were due to be reported. One caveat is that publication in a journal article, conference presentation, or as part of a meta-analysis was not included as, according to the study authors, this does not meet the requirements of the EC guideline.
This study pointed out that academic trials have a much worse track record than industry-led trials: sponsors doing fewer trials and non-commercial sponsors both have low rates of reporting. “Often, people have this impression that evil pharma withholds data,” says Windeler, “But in this case, academic groups are no better than industry. It might have been different about ten years ago, but now, industry is clearly under stricter observation and cannot afford to hide things.”
Non-reporting is prevalent for oncology trials
How bad is non-reporting in oncology? Jaime Perez-Alija, medical physicist at Hospital de la Santa Creu i Sant Pau in Barcelona, decided to investigate this question for radiation oncology, together with his colleague Pedro Gallego – initially, just for fun, he says. “We were dealing with cancer, so we thought publication of trial results would be more or less okay. We were very surprised to find what would look like a massive failure to publish results of completed trials.”
The observational study, published in 2017 (BMJ Open 7(9):e016040), showed that 84% of trials in radiation oncology registered with the ClinicalTrials.gov database did not post summary results within at least 16 months of trial completion, and only 45% reported results in a peer-reviewed journal. Again, industry-funded trials had higher reporting rates in the registry. A separate analysis found that four to six years after clinical trial abstracts are submitted and reported at ASCO, 39% of oncology trials remain unpublished in a peer-reviewed journal (The Oncologist 2016, 21:261–8).
Reasons for non-publication
Why are so many trial results relegated to the filing drawer? The OPEN Consortium (to Overcome failure to Publish nEgative fiNdings) was set up in 2011 to explore possible reasons and develop “evidence-informed recommendations focused on reducing dissemination bias”.
Meerpohl, who led the Consortium, sees it as a collective problem. “It is a complex problem with many different players, involving everyone from researchers, journals and funding agencies, to ethics committees. Each of these groups contributes to the problem a little bit.”
Ana Marušić, Chair of the Department of Research in Biomedicine and Health at the University of Split School of Medicine, in Croatia, and editor of the Journal of Global Health, questioned authors about reasons and solutions for non-publication as part of the OPEN Consortium. “In our focus group, authors said that: yes, they are guilty of not publishing. But they also said that the problem is that the system is not supporting them, that they have to rush from grant to grant without any time or funding to finish everything up.”
Journals, on the other hand, are often not interested in publishing negative results, IQWiG’s Windeler points out. “Negative results are often seen as boring and not worthy of publication. Journals and journal editors are often not interested in negative results, as positive results are seen as easier to sell and more important for progress, but that is of course nonsense.” Marušić, too, sees the pressure to publish as detrimental to full transparency. “Everything in the academic system now is geared towards publication. Maybe there should be incentives for posting results in registries, and journals should take on another role – picking up interesting results, providing a space for post-publication review after summary data or full data is published.”
The higher rate of industry reporting may be partly due to better awareness of requirements, says Windeler. “If you ask around in German universities whether researchers are aware that they should register and publish their trials, I think that in many cases you would encounter a lack of understanding and awareness. There is no support in this system for researchers to fulfil European requirements – and the institutions who should support this, including universities, funding agencies and ethics commissions, are not asking researchers to adhere to regulations.”
Finding a way forward
UK universities are now questioned about their adherence to European requirements for publishing results. In October 2018, the Science and Technology Committee of the House of Commons released a report, calling for increased transparency in clinical trial reporting. Norman Lamb, the chair of the committee, wrote to more than forty UK universities asking them to verify that the institutions are putting systems in place to comply with all reporting requirements. Universities whose track records are not improving may be questioned about their non-compliance in the autumn of 2019.
“Ethics committees should introduce this idea of ‘no further study from you will get through ethics until you have published”
In his letter, Lamb acknowledges the efforts of the AllTrials campaign, an international initiative co-founded in the UK in 2013 by Ben Goldacre, journalist and researcher at the University of Oxford, with the mission to get “All trials registered, all results reported”. Many patient groups support AllTrials. “The whole theme that AllTrials has driven has had a very strong patient input from day one, and I’ve supported the campaign right from the very start,” says Wilson.
While patient pressure appears to have some effect in the UK, Windeler does not see anything similar happening in Germany, which has quite a poor track record on reporting results. “At the moment, I cannot see statements or other forms of public activities from patient groups in Germany that exert pressure on politics or funding agencies, unfortunately.”
Other ideas for solutions are being proposed. “One possibility, and the one that usually comes first to mind, is using money to solve the problem,” says Windeler. “Funding agencies could tie part of the funding to the publication of results – so a part of the money is only released once the results are made publicly available. Or funding agencies could ask for proof that results of previous trials have been published before processing an application for new funding.”
TranspariMed’s Bruckner agrees: “Why should tax money be given to universities and other research institutions that have behaved unethically by not reporting results? That’s definitely a question funders should ask.”
A stronger involvement of ethics committees in the oversight of trial reporting is also proposed. “Ethics committees should introduce this idea of ‘no further study from you will get through ethics until you have published’,” Wilson suggests, “and the researcher’s employer, e.g. their university, should be advised of this. Ethics committees should take a look at studies that appear to have passed their proper publishing date.”
Measures aimed at solving the problem of non-publication will never be effective unless they are actually implemented. In the US, the FDA can levy fines of up to $10,000 for each day that has passed since a trial is due to have published its results. However, as far as is known, this sanction has never yet been applied.
Meerpohl sees both the problem and the solution as collective in nature. “When we investigated the reasons, our impression was that all the players involved blame each other. But we think that the solution has to be a concerted one. Everyone can contribute to solving this problem.”
If we replace “researchers” with “research system” the question posed by Alessandro Liberati in 2004 still holds true (BMJ 2004, 328:531): “How far can we tolerate the butterfly behaviour of [the research system], moving on to the next flower well before the previous one has been fully exploited?”
Leave a Reply