COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time

First empirical study of major academic journals’ willingness to publish a cohort of comparable and objective correction letters on misreported high-impact studies

Abstract

Background
Discrepancies between pre-specified and reported outcomes are an important source of bias in trials. Despite legislation, guidelines and public commitments on correct reporting from journals, outcome misreporting continues to be prevalent. We aimed to document the extent of misreporting, establish whether it was possible to publish correction letters on all misreported trials as they were published, and monitor responses from editors and trialists to understand why outcome misreporting persists despite public commitments to address it.

Methods
We identified five high-impact journals endorsing Consolidated Standards of Reporting Trials (CONSORT) (New England Journal of Medicine, The Lancet, Journal of the American Medical Association, British Medical Journal, and Annals of Internal Medicine) and assessed all trials over a six-week period to identify every correctly and incorrectly reported outcome, comparing published reports against published protocols or registry entries, using CONSORT as the gold standard. A correction letter describing all discrepancies was submitted to the journal for all misreported trials, and detailed coding sheets were shared publicly. The proportion of letters published and delay to publication were assessed over 12 months of follow-up. Correspondence received from journals and authors was documented and themes were extracted.

Results
Sixty-seven trials were assessed in total. Outcome reporting was poor overall and there was wide variation between journals on pre-specified primary outcomes (mean 76% correctly reported, journal range 25–96%), secondary outcomes (mean 55%, range 31–72%), and number of undeclared additional outcomes per trial (mean 5.4, range 2.9–8.3). Fifty-eight trials had discrepancies requiring a correction letter (87%, journal range 67–100%). Twenty-three letters were published (40%) with extensive variation between journals (range 0–100%). Where letters were published, there were delays (median 99 days, range 0–257 days). Twenty-nine studies had a pre-trial protocol publicly available (43%, range 0–86%). Qualitative analysis demonstrated extensive misunderstandings among journal editors about correct outcome reporting and CONSORT. Some journals did not engage positively when provided correspondence that identified misreporting; we identified possible breaches of ethics and publishing guidelines.

Conclusions
All five journals were listed as endorsing CONSORT, but all exhibited extensive breaches of this guidance, and most rejected correction letters documenting shortcomings. Readers are likely to be misled by this discrepancy. We discuss the advantages of prospective methodology research sharing all data openly and pro-actively in real time as feedback on critiqued studies. This is the first empirical study of major academic journals’ willingness to publish a cohort of comparable and objective correction letters on misreported high-impact studies. Suggested improvements include changes to correspondence processes at journals, alternatives for indexed post-publication peer review, changes to CONSORT’s mechanisms for enforcement, and novel strategies for research on methods and reporting.

Medicine is Broken > we need Your Help to Fix it

Why clinical trial transparency matters to patients in the US

Why patients and patient advocates like AnneMarie Ciccarella and Gregg Gonsalves support AllTrials, a patient-led campaign seeking to have all clinical trials registered, and their results reported.

Video published on 17 Mar 2016 by Sense About Science USA.

About All Trials

The BMJ requires data sharing on request for all trials

How we expect researchers to make all their data available

clinical-trial-capsules
From 1 July The BMJ will extend its requirements for data sharing to apply to all submitted clinical trials,
not just those that test drugs or devices. Image by Esther Dyson.

The movement to make data from clinical trials widely accessible has achieved enormous success, and it is now time for medical journals to play their part. From 1 July The BMJ will extend its requirements for data sharing to apply to all submitted clinical trials, not just those that test drugs or devices.

The BMJ’s Elizabeth Loder explains what this means for authors, and how we expect researchers to make their data available.

Press Play > to listen to the recording.

Sources and more information

  • The BMJ requires data sharing on request for all trials,
    BMJ 2015;350:h2373, 07 May 2015.
Our SoundCloud Playlists

CEBM All Trials session

Ben Goldacre on the importance of the All Trials campaign

Introduced by Sir Iain Chalmers, Ben Goldacre, Sile Lane and Carl Heneghan voice the importance of AllTrials.

More information

Imagining the Future of Medicine

Ben Goldacre was speaking in the home of the BBC Proms on recent progress in promoting transparency in clinical trials

As part of its mission to promote access to the arts and sciences, the Royal Albert Hall hosted an historic event on Monday 21 April 2014, curating a spectacular and provocative programme of talks from world experts celebrating innovation, imagination, inspiration and their passion for a better future in health and medicine.

More information
  • Ben Goldacre, former TED and TEDMED speaker, is a doctor, academic, campaigner and bad science writer whose work focuses on uses and misuses of science and statistics. He is the founder of the All Trials campaign and a keen advocate for clinical transparency.
  • Video published on 10 Jun 2014 by Royal Albert Hall.
  • Watch more research videos on @DES_Journal YT channel.

Ben Goldacre discusses Clinical Data Transparency at the Clinical Data Live!

Quanticate hosted a free symposium, which took place at the Royal College of Surgeons, September 2013

Dr Ben Goldacre, author of ‘Bad Pharma‘ presented a keynote speech at the Clinical Data Live! symposium. In this video hear Ben Goldacre present ‘ALLTrials: Transparency is moving forwards, industry can benefit from doing the right thing‘. This symposium was hosted by Quanticate who support Pharma and Biotech companies with statistical consultancy services.

More information

2014 Cosmic Genome Science Advent Calendar

Here is 2014 with Dr Ben Goldacre

From a successful year for the All Trials campaign, to new book ‘I Think You’ll Find It’s a Bit More Complicated Than That‘, here’s 2014 with Dr Ben Goldacre.

Selective Reporting of Analyses and Discrepancies in Clinical Trials

Evidence for the Selective Reporting of Analyses and Discrepancies in Clinical Trials: A Systematic Review of Cohort Studies of Clinical Trials

Abstract

PLoS_logo image
To ensure transparency, protocols and statistical analysis plans need to be published, and investigators should adhere to these or explain discrepancies.

Background
Most publications about selective reporting in clinical trials have focussed on outcomes. However, selective reporting of analyses for a given outcome may also affect the validity of findings. If analyses are selected on the basis of the results, reporting bias may occur. The aims of this study were to review and summarise the evidence from empirical cohort studies that assessed discrepant or selective reporting of analyses in randomised controlled trials (RCTs).
Methods and Findings
A systematic review was conducted and included cohort studies that assessed any aspect of the reporting of analyses of RCTs by comparing different trial documents, e.g., protocol compared to trial report, or different sections within a trial publication. The Cochrane Methodology Register, Medline (Ovid), PsycInfo (Ovid), and PubMed were searched on 5 February 2014. Two authors independently selected studies, performed data extraction, and assessed the methodological quality of the eligible studies. Twenty-two studies (containing 3,140 RCTs) published between 2000 and 2013 were included. Twenty-two studies reported on discrepancies between information given in different sources. Discrepancies were found in statistical analyses (eight studies), composite outcomes (one study), the handling of missing data (three studies), unadjusted versus adjusted analyses (three studies), handling of continuous data (three studies), and subgroup analyses (12 studies). Discrepancy rates varied, ranging from 7% (3/42) to 88% (7/8) in statistical analyses, 46% (36/79) to 82% (23/28) in adjusted versus unadjusted analyses, and 61% (11/18) to 100% (25/25) in subgroup analyses. This review is limited in that none of the included studies investigated the evidence for bias resulting from selective reporting of analyses. It was not possible to combine studies to provide overall summary estimates, and so the results of studies are discussed narratively.

Conclusions
Discrepancies in analyses between publications and other study documentation were common, but reasons for these discrepancies were not discussed in the trial reports. To ensure transparency, protocols and statistical analysis plans need to be published, and investigators should adhere to these or explain discrepancies.

Sources and Full Report
  • Evidence for the Selective Reporting of Analyses and Discrepancies in Clinical Trials: A Systematic Review of Cohort Studies of Clinical Trials, PLOS one Collections, DOI: 10.1371/journal.pmed.1001666, June 24, 2014.

UK Faculty of Pharmaceutical Medicine backs Greater Transparency in Clinical Trials

The FPM shows overwhelming support for the principles of clinical trial transparency

Faculty Pharm Medicine logo image
Faculty of Pharmaceutical Medicine shows overwhelming support for the principles of clinical trial transparency.

Members of the UK Faculty of Pharmaceutical Medicine have expressed backing for greater transparency in clinical trials in their first survey on the topic. They showed clear support for registration of all clinical trials, earlier publication of the summary results, and increased access to trial data.

  • 95% of survey respondents believe that all clinical trials should be registered
  • 89% believe that increased publication of clinical trial results (including negative results) will ultimately lead to better medicines and better healthcare for patients
  • 73% of respondents thought that clinical trials, irrespective of phase of development, should be published within 1– 2 years of completion (not linked to market authorisation or discontinuation of the project)
  • 87% believe that, overall, increased scrutiny of clinical trial data will result in a stronger science base and enhance medical research
  • Only 10% believe that increased publication and dissemination of clinical trial results will harm the commercial environment in which companies operate
  • 69% supported the requirement for retrospective release of clinical trials data , with the most commonly agreed time frame being data from ~5 years ago

Sources and More Information:

  • Clinical trials transparency and access to data,
    Faculty of Pharmaceutical Medicine, update, 28.8.14.
  • Faculty of Pharmaceutical Medicine survey of members on transparency in clinical trials, FPM, Analysis Report, 28.8.14.
  • Survey of pharmaceutical physicians shows overwhelming support for the principles of clinical trial transparency, FPM press release.
  • Faculty of Pharmaceutical Medicine backs greater transparency in clinical trials, BMJ, 349/bmj.g5381, 01 September 2014.

It is Time to Make Clinical Trials Count!

All trials past and present should be registered, and the full methods and the results reported

It is time all clinical trial results are reported. Results from around half of clinical trials have never been published. Information on what was done and what was found in these trials could be lost forever to doctors and researchers, leading to bad treatment decisions, missed opportunities for good medicine, and trials being repeated. The contributions of the hundreds of thousands of patients who took part in those trials remain unused and unusable.

More info and Videos