Books, websites and databases to promote critical thinking about treatment claims, 2003 to 2018
The James Lind Initiative (JLI) was a work programme inaugurated by Iain Chalmers and Patricia Atkinson to press for better research for better health care. It ran between 2003 and 2018, when Iain Chalmers retired. During the 15 years of its existence, the JLI developed three strands of work in collaboration with the authors of this paper, and with others.
The first work strand involved developing a process for use by patients, carers and clinicians to identify shared priorities for research – the James Lind Alliance.
The second strand was a series of articles, meetings, prizes and other developments to raise awareness of the massive amounts of avoidable waste in research, and of ways of reducing it.
The third strand involved using a variety of approaches to promote better public and professional understanding of the importance of research in clinical practice and public health.
JLI work on the first two themes has been addressed in previously published reports. This paper summarises JLI involvement during the 15 years of its existence in giving talks, convening workshops, writing books, and creating websites and databases to promote critical thinking about treatment claims.
During its 15-year life, the James Lind Initiative worked collaboratively with others to create free teaching and learning resources to help children and adults learn how to recognise untrustworthy claims about the effects of treatments. These resources have been translated in more than twenty languages, but much more could be done to support their uptake and wider use.
The James Lind Initiative is closing at the end of next month. Here's an account of our efforts over the past 16 years to promote critical thinking among patients and professionals about treatment effects claims https://t.co/GOJWbI4Fq5
Why comparisons must address genuine uncertainties
The design of treatment research often reflects commercial and academic interests; ignores relevant existing evidence; uses comparison treatments known in advance to be inferior; and ignores needs of users of research results (patients, health professionals and others).
A good deal of research is done even when there are no genuine uncertainties. Researchers who fail to conduct systematic reviews of past tests of treatments before embarking on further studies sometimes don’t recognise (or choose to ignore the fact) that uncertainties about treatment effects have already been convincingly addressed. This means that people participating in research are sometimes denied treatment that could help them, or given treatment likely to harm them.
The diagram that accompanies this and the following paragraph shows the accumulation of evidence from fair tests done to assess whether antibiotics (compared with inactive placebos) reduce the risk of post-operative death in people having bowel surgery (Lau et al. 1995). The first fair test was reported in 1969. The results of this small study left uncertainty about whether antibiotics were useful – the horizontal line representing the results spans the vertical line that separates favourable from unfavourable effects of antibiotics. Quite properly, this uncertainty was addressed in further tests in the early 1970s.
As the evidence accumulated, however, it became clear by the mid-1970s that antibiotics reduce the risk of death after surgery (the horizontal line falls clearly on the side of the vertical line favouring treatment). Yet researchers continued to do studies through to the late 1980s. Half the patients who received placebos in these later studies were thus denied a form of care which had been shown to reduce their risk of dying after their operations. How could this have happened? It was probably because researchers continued to embark on research without reviewing existing evidence systematically. This behaviour remains all too common in the research community, partly because some of the incentives in the world of research – commercial and academic – do not put the interests of patients first (Chalmers 2000).
Patients and participants in research can also suffer because researchers have not systematically reviewed relevant evidence from animal research before beginning to test treatments in humans. A Dutch team reviewed the experience of over 7000 patients who had participated in tests of a new calcium-blocking drug given to people experiencing a stroke. They found no evidence to support its increasing use in practice (Horn and Limburg 2001). This made them wonder about the quality and findings of the animal research that had led to the research on patients. Their review of the animal studies revealed that these had never suggested that the drug would be useful in humans (Horn et al. 2001).
The most common reason that research does not address genuine uncertainties is that researchers simply have not been sufficiently disciplined to review relevant existing evidence systematically before embarking on new studies. Sometimes there are more sinister reasons, however. Researchers may be aware of existing evidence, but they want to design studies to ensure that their own research will yield favourable results for particular treatments. Usually, but not always, this is for commercial reasons (Djulbegovic et al. 2000; Sackett and Oxman 2003; Chalmers and Glasziou 2009; Macleod et al. 2014). These studies are deliberately designed to be unfair tests of treatments. This can be done by withholding a comparison treatment known to help patients (as in the example given above), or giving comparison treatments in inappropriately low doses (so that they don’t work so well), or in inappropriately high doses (so that they have more unwanted side effects) (Mann and Djulbegovic 2012). It can also result from following up patients for too short a time (and missing delayed effects of treatments), and by using outcome measures (‘surrogates’) that have little or no correlation with the outcomes that matter to patients.
It may be surprising to readers of this essay that the research ethics committees established during recent decades to ensure that research is ethical have done so little to influence this research malpractice. Most such committees have let down the people they should have been protecting because they have not required researchers and sponsors seeking approval for new tests to have reviewed existing evidence systematically (Savulescu et al. 1996; Chalmers 2002). The failure of research ethics committees to protect patients and the public efficiently in this way emphasizes the importance of improving general knowledge about the characteristics of fair tests of medical treatments.
By Alan Cassels, Foreward by Sir Iain Chalmers, 2015
” In the complex, ever-evolving realm of modern medicine, how can you even begin to understand what’s hocus-pocus and what really works? Best-selling author and researcher Alan Cassels answers with a single word: Cochrane.
Though largely unknown to the public, the Cochrane Collaboration is made up of more than 30,000 medical researchers and consumer representatives from more than 100 countries – unbiased experts and investigators who parse the science of modern health care and delve deep into the evidence (or lack thereof) to determine what works and what doesn’t.
In this frank, factual and entertaining volume, Cassels draws from more than 160 interviews to shed light on this international cadre of medical truth-seekers whose rigorous work helps prevent medical misjudgement, reduce unnecessary suffering, preserve lives and circumvent the squandering of billions of dollars. “
” I don’t think industry are concerned about patient confidentiality except in so far as they are concerned to avoid being sued for injuries in clinical trials. ”
There is a very informative and fascinating debate going on between Dr David Healy and Ben Goldacre regarding options, thoughts, strategies for better clinical trials access and transparency.
Dr. David Healy posted “fucked” here and I posted a summary here.
Dr Ben Goldacre responded here and here. I republished his reply here.
Dr. David Healy then clarified here – see below – I only added few related links (in the original text response) with the purpose to bring more clarity and/or references to the readers.
Again let me invite you to read “fucked” post 22 comments – about clinical trial data access and pharmaceutical industry transparency.
” The first point to make is this post isn’t about AllTrials.
AllTrials is a footnote. It’s about the dismay that many felt at EMA backsliding. It’s about how it was obvious that something like this was on the cards. Against this background uncritical endorsement of industry looked like a bad idea. There was a desperate need to stay awake. It looks like too many of us have been asleep. Ben offers an outline of the AllTrials strategy here. It’s helpful to have this. His accusation that these posts misrepresent campaigns, smear people, shout abuse, and hector from the sidelines looks like a description of posts by others elsewhere. With very few exceptions any comments to the various posts on this blog that in any way fail to support Ben or AllTrials have been deleted. The post repeated an alternate analysis – that the main thing industry wants to hide are adverse event data. In a post 18 months ago I outlined how to achieve this industry would in public deploy the issue of patient confidentiality as a main justification for hiding data. In this it seems to me they have been assisted by Iain Chalmers editorial with Patrick Vaillance and now by Ben. The historical evolution of the confidentiality issue is that the first informed consent forms said nothing about not showing your data to anyone else. Unnoticed industry have slipped in a “we will of course show your data to no-one clause”. At the EMA conference on data access in November 2012, I made two points. The second was that industry would assert the notion of their privacy rights – which they have done. The other was that no one signs to have their data sequestered. Afterwards, Iain Chalmers congratulated me on the point – I thought we were on the same page. Whether adverse event data is key or not, Peter Gotzsche through the European Ombudsman and Tom Jefferson and Peter Doshi through Tamiflu and RIAT seem to me to have done more in practical terms to move the issues forward than anyone else. It leaves me wondering why there is an endless call to celebrate Ben and not Peter or Tom. Some of us have been working the GSK system and can see what the pitfalls are. Even if not redacted, this is a system that will make it close to impossible to analyse CSRs properly. But if it’s not proclaimed by AllTrials first it seems like such insights are unwelcome. In several posts before the latest debacle I outlined how in my opinion there was a real chance that magnificent though he has been and clearly morally right, Peter Gotzsche’s efforts may do more harm than good. Even without taking GSK’s preposterous data access system into account, pushing for data adds to the undue premium being put on RCTs Twenty years ago the moral case for access was as strong and the risks consequent on failing were much less in that we were less hypnotized by RCTs than we are now. Far from responding shrilly, Peter Gotzsche recognized the risk and we have been collaborating ever more closely since. The issues are so complex we might all be making mistakes – the only people unwilling to concede this seem to be AllTrials. The push for data access remains morally compelling but there are other things that can be done that might be more effective. As the BBC program a week ago on Thalidomide, and previous posts here, make clear, industry fear a boycott more than anything else. It is the only thing they have ever responded to. At the moment the focus is on a bunch of bureaucrats in EMA, who aren’t there with a brief to protect us other than by regulating the wording of advertisements. The focus should be on doctors who treat patients. We could refuse to use drugs where there is no access to the data. It shouldn’t even take courage to do this. In my opinion, this is the call that’s needed now rather than a call to support more of what AllTrials have been doing. But who will lead such a call? Along with colleagues I put forward a softer version of a boycott – an AbbVie – which encouraged doctors and patients to use drugs but to report on the adverse events which would in fact make these chemicals better medicines. It would be difficult for government or anyone else to gainsay this win-win option in the way they might come out against the lose-lose of a boycott. There is a conflict of interest here. RxISK.org has a stake in this idea. It was set up before AllTrials to move ideas like this forward. I suspect those of us working on RxISK in the evenings and at weekends have been putting far more hours into the effort than the AllTrials team have. At the end of the day, I may well be wrong on this, but I personally think AllTrials have been naïve. I don’t think industry are concerned about patient confidentiality except in so far as they are concerned to avoid being sued for injuries in clinical trials. Recent decades have seen industry put Litigation Support Defences in place. As outlined a decade ago in Let Them Eat Prozac, putting a premium on clinical trials has been a key element in their litigation support strategy. Seen from this vantage point AllTrials offers Pharma a lot – all without the effort of having to conspire or fund a conspiracy. Playing straight into industry’s hands is a hazard for all of us. Good intentions aren’t enough to save us. I’d rest more comfortably if the key players in AllTrials had a track record in bringing adverse events to light or even a record of supporting those trying to do so – if they’d really antagonized industry good and proper. It’s not that partnership isn’t nice but perhaps after playing hard to get first. ”
This book should be in every school, and every medical waiting room says Ben Goldacre
The 2nd edition of Testing Treatments was published in 2011. It urges everyone to get involved in improving current research and future treatment, and outlines practical steps that patients and health professionals can take together to do this.
How do we know whether a particular treatment really works? How reliable is the evidence? And how do we ensure that research into medical treatments best meets the needs of patients? These are just a few of the questions addressed in a lively and informative way in the free book written by Imogen Evans, Hazel Thornton, Sir Iain Chalmers and Paul Glasziou.
The full text of the 2nd edition is available for free download, and the paperback and ebook editions can be purchased from the publisher or from online booksellers. The text is already available in some other languages, and translations into additional languages are being prepared.
More than 10 years after Open Access publishing became widely available, lack of awareness persists concerning its potential to reduce waste, as does confusion over the difference between Open and Free Access
Among the important topics that Paul Glasziou and colleagues address in the Waste in Research Lancet Series:
the key issue of access to published knowledge receives little mention.
Reducing waste from incomplete or unusable reports of biomedical research
Research publication can both communicate and miscommunicate. Unless research is adequately reported, the time and resources invested in the conduct of research is wasted. Reporting guidelines such as CONSORT, STARD, PRISMA, and ARRIVE aim to improve the quality of research reports, but all are much less adopted and adhered to than they should be. Adequate reports of research should clearly describe which questions were addressed and why, what was done, what was shown, and what the findings mean. However, substantial failures occur in each of these elements. For example, studies of published trial reports showed that the poor description of interventions meant that 40-89% were non-replicable; comparisons of protocols with publications showed that most studies had at least one primary outcome changed, introduced, or omitted; and investigators of new trials rarely set their findings in the context of a systematic review, and cited a very small and biased selection of previous relevant trials. Although best documented in reports of controlled trials, inadequate reporting occurs in all types of studies-animal and other preclinical studies, diagnostic studies, epidemiological studies, clinical prediction research, surveys, and qualitative studies. In this report, and in the Series more generally, we point to a waste at all stages in medical research. Although a more nuanced understanding of the complex systems involved in the conduct, writing, and publication of research is desirable, some immediate action can be taken to improve the reporting of research. Evidence for some recommendations is clear: change the current system of research rewards and regulations to encourage better and more complete reporting, and fund the development and maintenance of infrastructure to support better reporting, linkage, and archiving of all elements of research. However, the high amount of waste also warrants future investment in the monitoring of and research into reporting of research, and active implementation of the findings to ensure that research reports better address the needs of the range of research users.
The Important Differences between Free Access Publications and Open Access Publications
” More than 10 years after Open Access publishing became widely available, lack of awareness persists concerning its potential to reduce waste, as does confusion over the difference between open and free access. Sir Iain Chalmers inadvertently highlighted this in his tweet about the series. We thought it would be useful to clarify the differences illustrated by this example.
Free access means: the article is free to read; it may not be reused (including translated) without permission; authors and readers may be charged for copying the article, and authors may be prohibited from posting their article on an institutional server. “Free” rights may be withdrawn at any time by the publisher, as occurred with the QUOROM paper on reporting of meta-analyses Moher D, 1999. This was published in the Lancet in 1999, was originally made free but was subsequently placed behind a paywall (but which since the time of the writing and submission – and rejection – of this comment to the Lancet has become freely available again, with no indication if it is a permanent state).
Open Access is defined as: free, immediate access online; unrestricted distribution and re-use rights in perpetuity for humans and technological applications; author(s) retains rights to attribution; papers are immediately deposited in a public online archive, such as PubMed Central. These principles, backed up by internationally accepted licenses from Creative Commons, means in practice that anything published Open Access can be read and reused in perpetuity by both humans and machines.
House of Commons, Science and Technology Committee, 2013
Clinical trials are the experimental foundation on which modern medicine is built. Trials also make a significant contribution to the UK economy and can provide patients with an important means of accessing the most exciting and innovative new treatments, before they reach the market.
Here you can browse the House of Commons Science and Technology Committee report together with the Proceedings of the Committee.
” 3.9 – I hope that the Science and Technology Committee will agree with Jeremy Paxman that the current situation is indeed “nuts”—unethical, unscientiﬁc and uneconomic nuts. “ ” 3.10 – My efforts to prompt improvement in clinical trial transparency over most of the past 30 years have manifestly failed. However, it is becoming clear that Sense about Science’s recently launched public campaign (www.alltrials.net) and Ben Goldacre‘s bestselling book Bad Pharma may be “game changers”. For the ﬁrst time in over 30 years I feel that there is reason to hope for substantive progress. I think that those who continue not to take under-reporting of research seriously will ﬁnd themselves on the wrong side of history. I hope that the Committee will see to it that, after decades of inadequate action, something substantial will be done to deal with the current, indefensible situation. ”
Why researchers MUST publish ALL results of clinical trials
” South Africans participating in clinical trials should demand that results are published ” says Cochrane Collaboration founder Sir Iain Chalmers.
” Much of the world’s clinical research was wasteful and failed to tackle issues that were most important to patients ” Sir Iain told delegates at a conference hosted by the Medical Research Council (MRC).
” Worse still, only 50% of the research was published, which meant doctors and patients were left in the dark about possible side effects, concealing evidence of potential possible harm from doctors and patients alike ” he said.