Evidence-based medicine is the cornerstone of medical practice, and yet clinicians are rarely offered training in the tricky business of finding, evaluating and making sense of the evidence they need. Anna Rouillard reports on an ESO-Cochrane Masterclass which seeks to fill that gap.
With the abundance of scientific literature on cancer treatments available, choosing the right one should be a simple matter of consulting the evidence and identifying which best answers the needs and priorities of your patients.
That is easier said than done, however, as around one million papers from clinical trials have been published to date – much of it presenting conflicting results, sometimes derived from poor-quality research methodology, and often addressing questions of marginal interest to patients and practitioners.
The ability to find and evaluate all the relevant studies, and draw robust conclusions from the totality of the evidence, takes skill and practice, but it is not generally taught in medical school. So when the European School of Oncology started teaming up with the Cochrane research collaboration to offer a week-long Masterclass on how to use, evaluate and conduct systematic reviews, there was no shortage of applicants.
Carlos Cargaleiro, a critical care cancer nurse from Portugal, currently working in the Royal Marsden Hospital in London, explains why he was so pleased to get a place on last year’s course, which was held as usual at Queen’s University, Belfast.
For me, giving patients the best possible available care is very important. But this means being constantly up to date on the latest evidence, which is a huge challenge when thousands of new articles are being published every single month,” he says. “As a nurse, I work as a member of a team which uses a specialised protocol. If I am suggesting changing practice based on evidence from a systematic review, I need to be able to present strong evidence to senior nursing staff. For this reason, I need to be confident of the quality of the articles included in the review, and that the review itself was done based on reliable methodology.”
The methodology for appraising and synthesising evidence taught at the Masterclass has its roots in an innovation introduced 40 years ago, when the statistician and researcher Gene Glass presented research findings in psychotherapy in the form of ‘an analysis of analyses’ (Educ Res 1976, 10:3–8). The advantage of this type of ‘meta-analysis’ was that it made use of all the available evidence by combining and averaging the results of several studies.
It wasn’t until the Cochrane Collaboration was set up in 1993, however, that the methodology of systematic reviews began to be developed across all areas of healthcare, covering how to define research questions, identify relevant studies, assess the quality of the studies, summarise the evidence (which may include meta-analysis), and interpret the results. Now known simply as Cochrane, today it involves a network of 40,000 researchers across 130 countries, working in 52 review groups, and the systematic review has become accepted as a cornerstone of evidence-based medicine.
Systematic reviews in cancer
Mike Clarke, Director of the Northern Ireland Hub for Trials Methodology Research, who co-chairs the ESO–Cochrane Masterclass, has been heavily involved in systematic reviews for the past 25 years, and says they have played a crucial role in developing our knowledge about the comparative risks and benefits of different cancer treatments.
In the seventies and eighties, the early years of systematic reviews, this methodology enabled us to identify some cancer therapies that weren’t working, such as old-style immunotherapy. Some of the big successes of systematic reviews have been in breast cancer, where we’ve shown that drugs like tamoxifen are beneficial, that chemotherapy is beneficial, and that ovarian ablation or suppression in the absence of chemotherapy is beneficial.”
Some, such as the breast cancer reviews, are substantial research projects, with a large amount of funding, while others may be done in researchers’ or practitioners’ spare time over several years.
Twenty years ago, there were hundreds of reviews available,” says Clarke. “Now there are tens of thousands done, with eight or nine thousand appearing every year.” Their proliferation, he adds, shows that they are increasingly considered good pieces of scientific research that practitioners and researchers want to contribute to.
Fergus Macbeth, joint coordinating editor of Cochrane’s Lung Cancer group, says that in some countries, systematic reviews have had a significant impact on clinical practice. “In the UK, every time NICE (National Institute for Health and Care Excellence) develops a guideline, a systematic review is routinely conducted.” This is not the case in every country, he adds. “In some countries hospitals rely on national or local guidelines of varying quality, or simply follow the practice set by the most senior person in the department.”
Collaborating to improve decision making
The Cochrane Collaboration was founded in 1993 under the leadership of Iain Chalmers, then director of the UK’s Cochrane Centre, with a mission to “prepare and maintain systematic reviews of relevant research to help improve decision making in healthcare”. Its roots lay in earlier work done at the UK’s National Perinatal Epidemiology Unit on the effects of care in pregnancy and childbirth, which was undertaken by Chalmers and others after he discovered that some of the obstetrics practices he had been taught were unsupported by reliable evidence. Chalmers’ mentor, Archie Cochrane, said that obstetrics was “the least scientifically based specialism in medicine”, and challenged him to carry out a systematic review of the available evidence.
The results were published in a two-volume book, with a shorter paperback version for women. They were also published in a digital format, for ease of updating. The Cochrane Pregnancy and Childcare database formed the start of the Cochrane library. The Cochrane Collaboration was established a few years later.
Today it is a huge network comprising more than 40,000 contributors across 130 countries, working in 52 review groups, most of whom do their work on a voluntary basis, using an agreed systematic and transparent methodology.
If you are interested in conducting a systematic review to find and evaluate the evidence on a particular topic, you can contact Cochrane to find out how to get involved. http://community.cochrane.org/
Macbeth is an ardent believer in the power of Cochrane reviews to deliver strong evidence for decision making. Over the years, his own lung cancer group has completed 45 reviews, 39 of them on treatments, including multimodality therapy and even holistic therapy, in different types and stages of lung cancer, as well as a few on prevention, diagnosis and early detection. He points out, however, that because Cochrane reviews generally only consider evidence from randomised controlled trials, there are whole areas of oncology that are not well covered. “Clearly some very important questions in oncology are related to the best ways of managing the patient in non-pharmacological ways, and there is too little high-quality research done in these areas. This may be because of the research infrastructure and the way research is funded, which currently prioritises studies on new drugs and genomic medicine. So evidence is sorely lacking in these other areas.”
As Clarke explains, systematic reviews are useful when a body of research evidence has built up on a topic, but people are struggling to interpret it, and where bringing all the evidence together and analysing it will bring clarity to the problem. A large amount of evidence is not needed for this exercise to be of value, he stresses. “Reviews that resolve uncertainty and give guidance to decision makers are perhaps the most important, but reviews also need to be undertaken in areas where research is sparse, since they can serve to highlight the fact that there is insufficient evidence available on a particular topic and that the existing research cannot answer the question reliably.”
A critical approach
The quality of available research may be as much of a problem as the quantity, as John Ioannidis pointed out in his recent paper ‘Why most clinical research is not useful’ (PLoS Medicine 2016, doi:10.1371/journal.pmed.1002049). He highlights frequent problems with transparency, with reports lacking key information on data, methods and analysis that could give readers the opportunity to evaluate for themselves the credibility of the reported results. Highly selective study populations may limit the applicability of findings to real life patient populations. The questions asked, or the endpoints measured, may relate only peripherally to the issues doctors and patients need answers to. And he points out that doing a meta-analysis of flawed studies doesn’t address the flaws, and may in fact compound them.
This, says Clarke, is why systematic reviews are so important, because they don’t just aggregate data, they take a critical look at the quality of the available research evidence. “The review can draw attention to how flawed the existing research is, which enables decision makers to realise that what they thought was proven, may not actually be proven.”
Non-publication of trial results as well as outright fraud may also jeopardise the reliability of the available evidence, he adds, which is why systematic reviews commonly use funnel plots to identify inconsistent data, which may point to publication bias.
Poor quality trials cannot be improved by systematic reviews, he argues, and inadequate and inappropriate review methodologies can lead to unreliable findings, even if the trials are good quality. “It might be called a systematic review”, he explains, “but this doesn’t mean it actually is a systematic review.” That is why it is so important that practitioners learn how to evaluate the quality of a systematic review. “They need to be able to assess how well studies have been sought, and whether the answers actually make sense.”
Skills for clinicians
Cargaleiro feels his five days at the Masterclass have left him far better equipped for this task. “My ability to analyse and evaluate systematic reviews has improved considerably,” he says, and he lists some key lessons he has learnt about how to conduct these sorts of reviews himself. “You need real teamwork to do systematic reviews,” he says. “Having a broad spectrum of experts at the table is essential to enable you to come up with a good research question. And once you’ve defined your research question, you need to know how to search for studies.”
When selecting search terms, he adds, it’s important to be imaginative, as the studies thrown up by a search will depend on the chosen spelling or terminology. ‘Caesarean’, for example, will yield 17,000 results on PubMed, whereas ‘cesarean’ brings up 53,000 results, and ‘c-section’ 48,000 results.
Language barriers are another problem. “Not all research is translated into English, and such findings will not be published on sites such as PubMed. If you are doing a review of evidence in English, you need to make sure the results will be applicable to the population it is intended for.”
But the Masterclass is not just about the theory. Participants get the chance to put some of the theory into practice, in sessions where they present their own ideas for systematic reviews to one another. “When you have different people listening to your ideas and providing feedback, your project idea can only improve,” says Cargaleiro. “They will help you see that maybe your idea is too broad, and you need to focus on a narrower theme, or that in fact you are trying to answer two questions when you should only be focusing on one.”
Clark believes these sorts of skills should be taught far more widely to reduce waste in medical research and ensure that clinically important gaps in knowledge are identified and researched. He argues that no new research should be done until a review is made of what already exists, to avoid duplication and identify gaps – which may seem like common sense, but very often doesn’t happen.
He also points out that systematic reviews are, usually, fairly economical to carry out. “They are scientific projects that require resources, but much of the resource use has already been spent by doing the studies. The research studies may have cost hundreds of millions of dollars, and the review is bringing all that evidence together, and, statistically, has potentially much greater power than any individual study.”
Cargaleiro agrees, and says that cancer practitioners should look for opportunities to feed into research prioritisation so the right questions are answered, and they should be aware of the available evidence, and how to assess it, or it will go to waste. “It’s important that research is not done just by researchers. They have a key role, but it is also imperative that clinical staff are involved. The worst possible scenario is that research is done that people in the field do not use.”
What we learnt
Twenty-four participants from 13 countries and from a range of cancer disciplines – surgical oncology, radiation oncology, medical oncology, urology, nursing and pharmacy – attended the 2016 ESO–Cochrane Masterclass. This is a flavour of what they took away:
– “It helped me a lot to have the collaboration of the other participants.”
– “I refined my question and that helped me to define with more quality the inclusion and exclusion criteria.”
– “Formulation of the question: it emphasises the importance of getting it right from the beginning.”
– “I had never received such information before, because it is not taught in my country. Now I can tackle all parts of systematic reviews and know how to do it in the most straightforward and correct way.”
– “The practical sessions gave invaluable insights about systematic reviews and allowed me to construct a critical reflection about my own work.”
– “The interactive group sessions were probably the best part.”
– “It is very helpful to know that there are different kinds of biases and that there are tools to predict the degree of heterogeneity among included studies.”
– “I frankly and honestly believe that Mike Clarke is the teacher that all of us should have at least once in a lifetime.”
– “My project is still a work in progress but I hope that with the support of my mentors and colleagues I will conclude it soon.”ESO hopes to run a third ESO–Cochrane Masterclass on Systematic Reviews in Summer 2018.
Leave a Reply