First empirical study of major academic journals’ willingness to publish a cohort of comparable and objective correction letters on misreported high-impact studies
Discrepancies between pre-specified and reported outcomes are an important source of bias in trials. Despite legislation, guidelines and public commitments on correct reporting from journals, outcome misreporting continues to be prevalent. We aimed to document the extent of misreporting, establish whether it was possible to publish correction letters on all misreported trials as they were published, and monitor responses from editors and trialists to understand why outcome misreporting persists despite public commitments to address it.
We identified five high-impact journals endorsing Consolidated Standards of Reporting Trials (CONSORT) (New England Journal of Medicine, The Lancet, Journal of the American Medical Association, British Medical Journal, and Annals of Internal Medicine) and assessed all trials over a six-week period to identify every correctly and incorrectly reported outcome, comparing published reports against published protocols or registry entries, using CONSORT as the gold standard. A correction letter describing all discrepancies was submitted to the journal for all misreported trials, and detailed coding sheets were shared publicly. The proportion of letters published and delay to publication were assessed over 12 months of follow-up. Correspondence received from journals and authors was documented and themes were extracted.
Sixty-seven trials were assessed in total. Outcome reporting was poor overall and there was wide variation between journals on pre-specified primary outcomes (mean 76% correctly reported, journal range 25–96%), secondary outcomes (mean 55%, range 31–72%), and number of undeclared additional outcomes per trial (mean 5.4, range 2.9–8.3). Fifty-eight trials had discrepancies requiring a correction letter (87%, journal range 67–100%). Twenty-three letters were published (40%) with extensive variation between journals (range 0–100%). Where letters were published, there were delays (median 99 days, range 0–257 days). Twenty-nine studies had a pre-trial protocol publicly available (43%, range 0–86%). Qualitative analysis demonstrated extensive misunderstandings among journal editors about correct outcome reporting and CONSORT. Some journals did not engage positively when provided correspondence that identified misreporting; we identified possible breaches of ethics and publishing guidelines.
All five journals were listed as endorsing CONSORT, but all exhibited extensive breaches of this guidance, and most rejected correction letters documenting shortcomings. Readers are likely to be misled by this discrepancy. We discuss the advantages of prospective methodology research sharing all data openly and pro-actively in real time as feedback on critiqued studies. This is the first empirical study of major academic journals’ willingness to publish a cohort of comparable and objective correction letters on misreported high-impact studies. Suggested improvements include changes to correspondence processes at journals, alternatives for indexed post-publication peer review, changes to CONSORT’s mechanisms for enforcement, and novel strategies for research on methods and reporting.
How we expect researchers to make all their data available
The movement to make data from clinical trials widely accessible has achieved enormous success, and it is now time for medical journals to play their part. From 1 July The BMJ will extend its requirements for data sharing to apply to all submitted clinical trials, not just those that test drugs or devices.
The BMJ’s Elizabeth Loder explains what this means for authors, and how we expect researchers to make their data available.
Ben Goldacre was speaking in the home of the BBC Proms on recent progress in promoting transparency in clinical trials
As part of its mission to promote access to the arts and sciences, the Royal Albert Hall hosted an historic event on Monday 21 April 2014, curating a spectacular and provocative programme of talks from world experts celebrating innovation, imagination, inspiration and their passion for a better future in health and medicine.
Ben Goldacre, former TED and TEDMED speaker, is a doctor, academic, campaigner and bad science writer whose work focuses on uses and misuses of science and statistics. He is the founder of the All Trials campaign and a keen advocate for clinical transparency.
Quanticate hosted a free symposium, which took place at the Royal College of Surgeons, September 2013
Dr Ben Goldacre, author of ‘Bad Pharma‘ presented a keynote speech at the Clinical Data Live! symposium. In this video hear Ben Goldacre present ‘ALLTrials: Transparency is moving forwards, industry can benefit from doing the right thing‘. This symposium was hosted by Quanticate who support Pharma and Biotech companies with statistical consultancy services.
Evidence for the Selective Reporting of Analyses and Discrepancies in Clinical Trials: A Systematic Review of Cohort Studies of Clinical Trials
Most publications about selective reporting in clinical trials have focussed on outcomes. However, selective reporting of analyses for a given outcome may also affect the validity of findings. If analyses are selected on the basis of the results, reporting bias may occur. The aims of this study were to review and summarise the evidence from empirical cohort studies that assessed discrepant or selective reporting of analyses in randomised controlled trials (RCTs). Methods and Findings
A systematic review was conducted and included cohort studies that assessed any aspect of the reporting of analyses of RCTs by comparing different trial documents, e.g., protocol compared to trial report, or different sections within a trial publication. The Cochrane Methodology Register, Medline (Ovid), PsycInfo (Ovid), and PubMed were searched on 5 February 2014. Two authors independently selected studies, performed data extraction, and assessed the methodological quality of the eligible studies. Twenty-two studies (containing 3,140 RCTs) published between 2000 and 2013 were included. Twenty-two studies reported on discrepancies between information given in different sources. Discrepancies were found in statistical analyses (eight studies), composite outcomes (one study), the handling of missing data (three studies), unadjusted versus adjusted analyses (three studies), handling of continuous data (three studies), and subgroup analyses (12 studies). Discrepancy rates varied, ranging from 7% (3/42) to 88% (7/8) in statistical analyses, 46% (36/79) to 82% (23/28) in adjusted versus unadjusted analyses, and 61% (11/18) to 100% (25/25) in subgroup analyses. This review is limited in that none of the included studies investigated the evidence for bias resulting from selective reporting of analyses. It was not possible to combine studies to provide overall summary estimates, and so the results of studies are discussed narratively.
Discrepancies in analyses between publications and other study documentation were common, but reasons for these discrepancies were not discussed in the trial reports. To ensure transparency, protocols and statistical analysis plans need to be published, and investigators should adhere to these or explain discrepancies.
Sources and Full Report
Evidence for the Selective Reporting of Analyses and Discrepancies in Clinical Trials: A Systematic Review of Cohort Studies of Clinical Trials, PLOS one Collections, DOI: 10.1371/journal.pmed.1001666, June 24, 2014.
The FPM shows overwhelming support for the principles of clinical trial transparency
Members of the UK Faculty of Pharmaceutical Medicine have expressed backing for greater transparency in clinical trials in their first survey on the topic. They showed clear support for registration of all clinical trials, earlier publication of the summary results, and increased access to trial data.
95% of survey respondents believe that all clinical trials should be registered
89% believe that increased publication of clinical trial results (including negative results) will ultimately lead to better medicines and better healthcare for patients
73% of respondents thought that clinical trials, irrespective of phase of development, should be published within 1– 2 years of completion (not linked to market authorisation or discontinuation of the project)
87% believe that, overall, increased scrutiny of clinical trial data will result in a stronger science base and enhance medical research
Only 10% believe that increased publication and dissemination of clinical trial results will harm the commercial environment in which companies operate
69% supported the requirement for retrospective release of clinical trials data , with the most commonly agreed time frame being data from ~5 years ago
Sources and More Information:
Clinical trials transparency and access to data,
Faculty of Pharmaceutical Medicine, update, 28.8.14.
Faculty of Pharmaceutical Medicine survey of members on transparency in clinical trials, FPM, Analysis Report, 28.8.14.
Survey of pharmaceutical physicians shows overwhelming support for the principles of clinical trial transparency, FPM press release.
Faculty of Pharmaceutical Medicine backs greater transparency in clinical trials, BMJ, 349/bmj.g5381, 01 September 2014.
All trials past and present should be registered, and the full methods and the results reported
It is time all clinical trial results are reported. Results from around half of clinical trials have never been published. Information on what was done and what was found in these trials could be lost forever to doctors and researchers, leading to bad treatment decisions, missed opportunities for good medicine, and trials being repeated. The contributions of the hundreds of thousands of patients who took part in those trials remain unused and unusable.