Tag: Sir Iain Chalmers

The James Lind Library’s Introduction to Fair Tests of Treatments

JLL Book of Essays, About Fair Tests, May 2019

Introduction

At various times in our lives and to varying levels of intensity, we all use, provide or pay for health and social care. As we decide what to do, take, offer or buy, we need evidence that is reliable, robust and trustworthy about different options. Even before James Lind’s experiment comparing possible treatments for scurvy on HMS Salisbury people had recognised that getting this evidence requires strenuous efforts to reduce bias –but that achieving this is often not straightforward. This book of essays from the James Lind Library is our attempt to illustrate some of the challenges encountered and how to overcome them.

We will take you on a journey through the sometimes stormy waters of why treatments need to be tested, rather than being based on assumptions that “it must work” before the treatment has even been tried, or based on impressions after it has been used a few times, through to the need for fair tests comparing alternative treatment options. We will show why genuine uncertainties must be identified and addressed, and how research to find the most effective and appropriate treatments need to build on research to identify the most effective and appropriate methods for doing that research. We will navigate through the reasons why comparisons need to be fair at the outset, and then kept fair as the treatments being tested are given;outcomes are measured;and results are analysed, reported, and combined in systematic reviews of all the relevant, trustworthy evidence.

We have not cluttered the chapters with references to all the source material on which we have drawn. For that level of detail, please follow the links to the fuller essays on the James Lind Library website. Instead, where we know of reviews of methodology research which are relevant to a topic, we have listed these at the end of each chapter.

By the end of the book, we hope that you will recognise how, to bring benefits of research to patients and the public, systematic reviews of fair tests are needed to provide key elements of the knowledge needed to inform decisions about health and social care, while taking into account other important factors, such as values, preferences, needs, resources and priorities. We also hope that, as you finish the book, you will share the sense of enlightenment, education and enjoyment that we have gained from preparing it.

Finally, we dedicate this book to England’s National Institute for Health Research. Without the Institute’s 16-year-long support for the James Lind Initiative, the home of the James Lind Library during that time, neither the Library nor these essays would have been possible. And we also wish to acknowledge the role the Institute plays in recognising the vital contribution of research to the delivery of health and social care that is effective and efficient, and the Institute’s leadership in ensuring that the research itself is effective, efficient and reliable, with minimal waste.

Abstract

1.3 Why treatment comparisons must be fair

Untrustworthy treatment comparisons are those in which biases, or the play of chance, or both result in misleading estimates of the effects of treatments. Fair treatment comparisons avoid biases and reduce the effects of the play of chance.

It is not only failure to test theories about treatments in practice that has caused preventable tragedies. They have also occurred because the tests used to assess the effects of treatments have been unreliable and misleading. In the 1950s, theory and poorly controlled tests yielded unreliable evidence suggesting that diethylstilboestrol (DES) helped pregnant women who had previously had miscarriages and stillbirths. Although fair tests suggested that DES was useless, theory and unreliable evidence, together with aggressive marketing, led to DES being prescribed to millions of pregnant women over the next few decades. The consequences were disastrous for the women and their children, who experienced infertility and cancers as a result. The lesson is that a treatment that has not been reliably shown to be useful should not be promoted.

Problems resulting from inadequate tests of treatments continue to occur. Again, because of unreliable evidence and aggressive marketing, millions of women were persuaded to use hormone replacement therapy (HRT). It was claimed that, not only could it reduce unpleasant menopausal symptoms, but also the chances of having heart attacks and strokes. When these claims were assessed in fair tests, the results showed that in women over 60, far from reducing the risks of heart attacks and strokes, HRT increases the risks of these life-threatening conditions, as well as having other undesirable effects.These examples of the need for fair tests of treatments are a few of many that illustrate how treatments can do more harm than good. Improved general knowledge about fair tests of treatments is needed so that –laced with a healthy dose of scepticism –we can all assess claims about the effects of treatments more critically. That way, we will all become more able to judge which treatments are likely to do more good than harm.

DES DiEthylStilbestrol Resources

The James Lind Initiative

Books, websites and databases to promote critical thinking about treatment claims, 2003 to 2018

Abstract

Background:
The James Lind Initiative (JLI) was a work programme inaugurated by Iain Chalmers and Patricia Atkinson to press for better research for better health care. It ran between 2003 and 2018, when Iain Chalmers retired. During the 15 years of its existence, the JLI developed three strands of work in collaboration with the authors of this paper, and with others.

Work themes:

  1. The first work strand involved developing a process for use by patients, carers and clinicians to identify shared priorities for research – the James Lind Alliance.
  2. The second strand was a series of articles, meetings, prizes and other developments to raise awareness of the massive amounts of avoidable waste in research, and of ways of reducing it.
  3. The third strand involved using a variety of approaches to promote better public and professional understanding of the importance of research in clinical practice and public health.

JLI work on the first two themes has been addressed in previously published reports. This paper summarises JLI involvement during the 15 years of its existence in giving talks, convening workshops, writing books, and creating websites and databases to promote critical thinking about treatment claims.

Conclusion:
During its 15-year life, the James Lind Initiative worked collaboratively with others to create free teaching and learning resources to help children and adults learn how to recognise untrustworthy claims about the effects of treatments. These resources have been translated in more than twenty languages, but much more could be done to support their uptake and wider use.

Reference. See also PMC539653. Image credit stemlynsblog.

Medical research often ignores relevant existing evidence and patient needs

Why comparisons must address genuine uncertainties

The design of treatment research often reflects commercial and academic interests; ignores relevant existing evidence; uses comparison treatments known in advance to be inferior; and ignores needs of users of research results (patients, health professionals and others).

A good deal of research is done even when there are no genuine uncertainties. Researchers who fail to conduct systematic reviews of past tests of treatments before embarking on further studies sometimes don’t recognise (or choose to ignore the fact) that uncertainties about treatment effects have already been convincingly addressed. This means that people participating in research are sometimes denied treatment that could help them, or given treatment likely to harm them.

When researchers continue to embark on (same) research for decades without reviewing existing evidence systematically.

The diagram that accompanies this and the following paragraph shows the accumulation of evidence from fair tests done to assess whether antibiotics (compared with inactive placebos) reduce the risk of post-operative death in people having bowel surgery (Lau et al. 1995). The first fair test was reported in 1969. The results of this small study left uncertainty about whether antibiotics were useful – the horizontal line representing the results spans the vertical line that separates favourable from unfavourable effects of antibiotics. Quite properly, this uncertainty was addressed in further tests in the early 1970s.

As the evidence accumulated, however, it became clear by the mid-1970s that antibiotics reduce the risk of death after surgery (the horizontal line falls clearly on the side of the vertical line favouring treatment). Yet researchers continued to do studies through to the late 1980s. Half the patients who received placebos in these later studies were thus denied a form of care which had been shown to reduce their risk of dying after their operations. How could this have happened? It was probably because researchers continued to embark on research without reviewing existing evidence systematically. This behaviour remains all too common in the research community, partly because some of the incentives in the world of research – commercial and academic – do not put the interests of patients first (Chalmers 2000).

Patients and participants in research can also suffer because researchers have not systematically reviewed relevant evidence from animal research before beginning to test treatments in humans. A Dutch team reviewed the experience of over 7000 patients who had participated in tests of a new calcium-blocking drug given to people experiencing a stroke. They found no evidence to support its increasing use in practice (Horn and Limburg 2001). This made them wonder about the quality and findings of the animal research that had led to the research on patients. Their review of the animal studies revealed that these had never suggested that the drug would be useful in humans (Horn et al. 2001).

The most common reason that research does not address genuine uncertainties is that researchers simply have not been sufficiently disciplined to review relevant existing evidence systematically before embarking on new studies. Sometimes there are more sinister reasons, however. Researchers may be aware of existing evidence, but they want to design studies to ensure that their own research will yield favourable results for particular treatments. Usually, but not always, this is for commercial reasons (Djulbegovic et al. 2000; Sackett and Oxman 2003; Chalmers and Glasziou 2009; Macleod et al. 2014). These studies are deliberately designed to be unfair tests of treatments. This can be done by withholding a comparison treatment known to help patients (as in the example given above), or giving comparison treatments in inappropriately low doses (so that they don’t work so well), or in inappropriately high doses (so that they have more unwanted side effects) (Mann and Djulbegovic 2012). It can also result from following up patients for too short a time (and missing delayed effects of treatments), and by using outcome measures (‘surrogates’) that have little or no correlation with the outcomes that matter to patients.

It may be surprising to readers of this essay that the research ethics committees established during recent decades to ensure that research is ethical have done so little to influence this research malpractice. Most such committees have let down the people they should have been protecting because they have not required researchers and sponsors seeking approval for new tests to have reviewed existing evidence systematically (Savulescu et al. 1996; Chalmers 2002). The failure of research ethics committees to protect patients and the public efficiently in this way emphasizes the importance of improving general knowledge about the characteristics of fair tests of medical treatments.

This is a reprint from The James Lind Library 2.1 Why comparisons must address genuine uncertainties.

The Cochrane Collaboration: Medicine’s Best-Kept Secret

By Alan Cassels, Foreward by Sir Iain Chalmers, 2015

The-Cochrane-Collaboration book cover omage
To celebrate its 20th anniversary, writer and professional medical skeptic Alan Cassels was selected to draw back the curtains on the Cochrane Collaboration. His extensive interviews of Cochranites allowed him to document their achievements, resulting in a book that its leadership ultimately refused to publish.

” In the complex, ever-evolving realm of modern medicine, how can you even begin to understand what’s hocus-pocus and what really works? Best-selling author and researcher Alan Cassels answers with a single word: Cochrane.

Though largely unknown to the public, the Cochrane Collaboration is made up of more than 30,000 medical researchers and consumer representatives from more than 100 countries – unbiased experts and investigators who parse the science of modern health care and delve deep into the evidence (or lack thereof) to determine what works and what doesn’t.

In this frank, factual and entertaining volume, Cassels draws from more than 160 interviews to shed light on this international cadre of medical truth-seekers whose rigorous work helps prevent medical misjudgement, reduce unnecessary suffering, preserve lives and circumvent the squandering of billions of dollars. “

More books from Alan Cassels

CEBM All Trials session

Ben Goldacre on the importance of the All Trials campaign

Introduced by Sir Iain Chalmers, Ben Goldacre, Sile Lane and Carl Heneghan voice the importance of AllTrials.

More information

Pharma Industry fear a Boycott more than anything else

” I don’t think industry are concerned about patient confidentiality except in so far as they are concerned to avoid being sued for injuries in clinical trials. ”

There is a very informative and fascinating debate going on between Dr David Healy and Ben Goldacre regarding options, thoughts, strategies for better clinical trials access and transparency.
Dr. David Healy posted “fucked” here and I posted a summary here.
Dr Ben Goldacre responded here and here. I republished his reply here.
Dr. David Healy then clarified here – see below – I only added few related links (in the original text response) with the purpose to bring more clarity and/or references to the readers.

Again let me invite you to read “fucked” post 22 comments – about clinical trial data access and pharmaceutical industry transparency.

image of David Healy
David Healy is professor of psychiatry and co-founder of data based medicine, operating through RxISK.org , working towards making medicines safer.

” The first point to make is this post isn’t about AllTrials.
AllTrials is a footnote.
It’s about the dismay that many felt at EMA backsliding. It’s about how it was obvious that something like this was on the cards. Against this background uncritical endorsement of industry looked like a bad idea. There was a desperate need to stay awake. It looks like too many of us have been asleep.
Ben offers an outline of the AllTrials strategy here. It’s helpful to have this.
His accusation that these posts misrepresent campaigns, smear people, shout abuse, and hector from the sidelines looks like a description of posts by others elsewhere. With very few exceptions any comments to the various posts on this blog that in any way fail to support Ben or AllTrials have been deleted.
The post repeated an alternate analysis – that the main thing industry wants to hide are adverse event data.
In a post 18 months ago I outlined how to achieve this industry would in public deploy the issue of patient confidentiality as a main justification for hiding data. In this it seems to me they have been assisted by Iain Chalmers editorial with Patrick Vaillance and now by Ben.
The historical evolution of the confidentiality issue is that the first informed consent forms said nothing about not showing your data to anyone else. Unnoticed industry have slipped in a “we will of course show your data to no-one clause”.
At the EMA conference on data access in November 2012, I made two points. The second was that industry would assert the notion of their privacy rights – which they have done. The other was that no one signs to have their data sequestered. Afterwards, Iain Chalmers congratulated me on the point – I thought we were on the same page.
Whether adverse event data is key or not, Peter Gotzsche through the European Ombudsman and Tom Jefferson and Peter Doshi through Tamiflu and RIAT seem to me to have done more in practical terms to move the issues forward than anyone else. It leaves me wondering why there is an endless call to celebrate Ben and not Peter or Tom.
Some of us have been working the GSK system and can see what the pitfalls are. Even if not redacted, this is a system that will make it close to impossible to analyse CSRs properly. But if it’s not proclaimed by AllTrials first it seems like such insights are unwelcome.
In several posts before the latest debacle I outlined how in my opinion there was a real chance that magnificent though he has been and clearly morally right, Peter Gotzsche’s efforts may do more harm than good. Even without taking GSK’s preposterous data access system into account, pushing for data adds to the undue premium being put on RCTs Twenty years ago the moral case for access was as strong and the risks consequent on failing were much less in that we were less hypnotized by RCTs than we are now.
Far from responding shrilly, Peter Gotzsche recognized the risk and we have been collaborating ever more closely since. The issues are so complex we might all be making mistakes – the only people unwilling to concede this seem to be AllTrials.
The push for data access remains morally compelling but there are other things that can be done that might be more effective.
As the BBC program a week ago on Thalidomide, and previous posts here, make clear, industry fear a boycott more than anything else. It is the only thing they have ever responded to.
At the moment the focus is on a bunch of bureaucrats in EMA, who aren’t there with a brief to protect us other than by regulating the wording of advertisements.
The focus should be on doctors who treat patients. We could refuse to use drugs where there is no access to the data. It shouldn’t even take courage to do this. In my opinion, this is the call that’s needed now rather than a call to support more of what AllTrials have been doing. But who will lead such a call?
Along with colleagues I put forward a softer version of a boycott – an AbbVie – which encouraged doctors and patients to use drugs but to report on the adverse events which would in fact make these chemicals better medicines. It would be difficult for government or anyone else to gainsay this win-win option in the way they might come out against the lose-lose of a boycott.
There is a conflict of interest here. RxISK.org has a stake in this idea. It was set up before AllTrials to move ideas like this forward. I suspect those of us working on RxISK in the evenings and at weekends have been putting far more hours into the effort than the AllTrials team have.
At the end of the day, I may well be wrong on this, but I personally think AllTrials have been naïve. I don’t think industry are concerned about patient confidentiality except in so far as they are concerned to avoid being sued for injuries in clinical trials.
Recent decades have seen industry put Litigation Support Defences in place. As outlined a decade ago in Let Them Eat Prozac, putting a premium on clinical trials has been a key element in their litigation support strategy. Seen from this vantage point AllTrials offers Pharma a lot – all without the effort of having to conspire or fund a conspiracy.
Playing straight into industry’s hands is a hazard for all of us. Good intentions aren’t enough to save us. I’d rest more comfortably if the key players in AllTrials had a track record in bringing adverse events to light or even a record of supporting those trying to do so – if they’d really antagonized industry good and proper. It’s not that partnership isn’t nice but perhaps after playing hard to get first. ”

Find all our posts about the AllTrials campaign, Clinical trials, Dr Ben Goldacre, Dr. David Healy, GlaxoSmithKline, Peter Gøtzsche and Sir Iain Chalmers.

Testing Treatments Interactive free Book, Better Research for Better HealthCare

This book should be in every school, and every medical waiting room says Ben Goldacre

image of Testing Treatments Interactive Book
Click for free download.

The 2nd edition of Testing Treatments was published in 2011. It urges everyone to get involved in improving current research and future treatment, and outlines practical steps that patients and health professionals can take together to do this.

How do we know whether a particular treatment really works? How reliable is the evidence? And how do we ensure that research into medical treatments best meets the needs of patients? These are just a few of the questions addressed in a lively and informative way in the free book written by Imogen Evans, Hazel Thornton, Sir Iain Chalmers and Paul Glasziou.

The full text of the 2nd edition is available for free download, and the paperback and ebook editions can be purchased from the publisher or from online booksellers. The text is already available in some other languages, and translations into additional languages are being prepared.

Sources:

Introducing “Testing Treatments Interactive”

Promoting Better Research for Better HealthCare, 2014

Sir Iain Chalmers explains why fair tests of the effects of treatments can improve healthcare.

More information

The Important Differences between Free and Open Access Publications

More than 10 years after Open Access publishing became widely available, lack of awareness persists concerning its potential to reduce waste, as does confusion over the difference between Open and Free Access

Among the important topics that Paul Glasziou and colleagues address in the Waste in Research Lancet Series:

the key issue of access to published knowledge receives little mention.

Reducing waste from incomplete or unusable reports of biomedical research

the waste in research Lancet series
Keep in touch with The Lancet, the world’s leading general medical journal.

Abstract:
Research publication can both communicate and miscommunicate. Unless research is adequately reported, the time and resources invested in the conduct of research is wasted. Reporting guidelines such as CONSORT, STARD, PRISMA, and ARRIVE aim to improve the quality of research reports, but all are much less adopted and adhered to than they should be. Adequate reports of research should clearly describe which questions were addressed and why, what was done, what was shown, and what the findings mean. However, substantial failures occur in each of these elements. For example, studies of published trial reports showed that the poor description of interventions meant that 40-89% were non-replicable; comparisons of protocols with publications showed that most studies had at least one primary outcome changed, introduced, or omitted; and investigators of new trials rarely set their findings in the context of a systematic review, and cited a very small and biased selection of previous relevant trials. Although best documented in reports of controlled trials, inadequate reporting occurs in all types of studies-animal and other preclinical studies, diagnostic studies, epidemiological studies, clinical prediction research, surveys, and qualitative studies. In this report, and in the Series more generally, we point to a waste at all stages in medical research. Although a more nuanced understanding of the complex systems involved in the conduct, writing, and publication of research is desirable, some immediate action can be taken to improve the reporting of research. Evidence for some recommendations is clear: change the current system of research rewards and regulations to encourage better and more complete reporting, and fund the development and maintenance of infrastructure to support better reporting, linkage, and archiving of all elements of research. However, the high amount of waste also warrants future investment in the monitoring of and research into reporting of research, and active implementation of the findings to ensure that research reports better address the needs of the range of research users.

The Important Differences between Free Access Publications and Open Access Publications

Sir Iain Chalmers
Sir Iain Chalmers

by Ginny Barbour, Medicine Editorial Director, PLOS Medicine

” More than 10 years after Open Access publishing became widely available, lack of awareness persists concerning its potential to reduce waste, as does confusion over the difference between open and free access. Sir Iain Chalmers inadvertently highlighted this in his tweet about the series. We thought it would be useful to clarify the differences illustrated by this example.

Free access means: the article is free to read; it may not be reused (including translated) without permission; authors and readers may be charged for copying the article, and authors may be prohibited from posting their article on an institutional server. “Free” rights may be withdrawn at any time by the publisher, as occurred with the QUOROM paper on reporting of meta-analyses Moher D, 1999. This was published in the Lancet in 1999, was originally made free but was subsequently placed behind a paywall (but which since the time of the writing and submission – and rejection – of this comment to the Lancet has become freely available again, with no indication if it is a permanent state).

Open Access is defined as: free, immediate access online; unrestricted distribution and re-use rights in perpetuity for humans and technological applications; author(s) retains rights to attribution; papers are immediately deposited in a public online archive, such as PubMed Central. These principles, backed up by internationally accepted licenses from Creative Commons, means in practice that anything published Open Access can be read and reused in perpetuity by both humans and machines.

The Lancet Series is “Copyright © 2014 Elsevier Ltd All rights reserved”. At the time of publication it was necessary to login to read the articles. It is, therefore, free, not Open Access.
We hope that future articles in this important endeavour will address the need for open, not just free, access, and will themselves be Open Access. “

Sources

  • NCBI PMID: 24411647 Lancet. 2014 Jan 8; 383(9913):267-76. doi: 10.1016/S0140-6736(13)62228-X. Epub 2014 Jan 8.
  • Learn about the All Trials Campaign

The UK Clinical Trials situation is NUTS: unEthical, unScientific, unEconomic

House of Commons, Science and Technology Committee, 2013

Written evidence submitted by Sir Iain Chalmers
Sir Iain Chalmers says that the current situation is indeed “nuts”—unethical, unscientific and uneconomic nuts

Clinical trials are the experimental foundation on which modern medicine is built. Trials also make a significant contribution to the UK economy and can provide patients with an important means of accessing the most exciting and innovative new treatments, before they reach the market.

Here you can browse the House of Commons Science and Technology Committee report together with the Proceedings of the Committee.

Sir Iain Chalmers submitted this written evidence:

” 3.9 – I hope that the Science and Technology Committee will agree with Jeremy Paxman that the current situation is indeed “nuts”—unethical, unscientific and uneconomic nuts. “
” 3.10 – My efforts to prompt improvement in clinical trial transparency over most of the past 30 years have manifestly failed. However, it is becoming clear that Sense about Science’s recently launched public campaign (www.alltrials.net) and Ben Goldacre‘s bestselling book Bad Pharma may be “game changers”. For the first time in over 30 years I feel that there is reason to hope for substantive progress. I think that those who continue not to take under-reporting of research seriously will find themselves on the wrong side of history. I hope that the Committee will see to it that, after decades of inadequate action, something substantial will be done to deal with the current, indefensible situation. ”

Read:

Related posts: