The James Lind Library’s Introduction to Fair Tests of Treatments

JLL Book of Essays, About Fair Tests, May 2019

Introduction

At various times in our lives and to varying levels of intensity, we all use, provide or pay for health and social care. As we decide what to do, take, offer or buy, we need evidence that is reliable, robust and trustworthy about different options. Even before James Lind’s experiment comparing possible treatments for scurvy on HMS Salisbury people had recognised that getting this evidence requires strenuous efforts to reduce bias –but that achieving this is often not straightforward. This book of essays from the James Lind Library is our attempt to illustrate some of the challenges encountered and how to overcome them.

We will take you on a journey through the sometimes stormy waters of why treatments need to be tested, rather than being based on assumptions that “it must work” before the treatment has even been tried, or based on impressions after it has been used a few times, through to the need for fair tests comparing alternative treatment options. We will show why genuine uncertainties must be identified and addressed, and how research to find the most effective and appropriate treatments need to build on research to identify the most effective and appropriate methods for doing that research. We will navigate through the reasons why comparisons need to be fair at the outset, and then kept fair as the treatments being tested are given;outcomes are measured;and results are analysed, reported, and combined in systematic reviews of all the relevant, trustworthy evidence.

We have not cluttered the chapters with references to all the source material on which we have drawn. For that level of detail, please follow the links to the fuller essays on the James Lind Library website. Instead, where we know of reviews of methodology research which are relevant to a topic, we have listed these at the end of each chapter.

By the end of the book, we hope that you will recognise how, to bring benefits of research to patients and the public, systematic reviews of fair tests are needed to provide key elements of the knowledge needed to inform decisions about health and social care, while taking into account other important factors, such as values, preferences, needs, resources and priorities. We also hope that, as you finish the book, you will share the sense of enlightenment, education and enjoyment that we have gained from preparing it.

Finally, we dedicate this book to England’s National Institute for Health Research. Without the Institute’s 16-year-long support for the James Lind Initiative, the home of the James Lind Library during that time, neither the Library nor these essays would have been possible. And we also wish to acknowledge the role the Institute plays in recognising the vital contribution of research to the delivery of health and social care that is effective and efficient, and the Institute’s leadership in ensuring that the research itself is effective, efficient and reliable, with minimal waste.

Abstract

1.3 Why treatment comparisons must be fair

Untrustworthy treatment comparisons are those in which biases, or the play of chance, or both result in misleading estimates of the effects of treatments. Fair treatment comparisons avoid biases and reduce the effects of the play of chance.

It is not only failure to test theories about treatments in practice that has caused preventable tragedies. They have also occurred because the tests used to assess the effects of treatments have been unreliable and misleading. In the 1950s, theory and poorly controlled tests yielded unreliable evidence suggesting that diethylstilboestrol (DES) helped pregnant women who had previously had miscarriages and stillbirths. Although fair tests suggested that DES was useless, theory and unreliable evidence, together with aggressive marketing, led to DES being prescribed to millions of pregnant women over the next few decades. The consequences were disastrous for the women and their children, who experienced infertility and cancers as a result. The lesson is that a treatment that has not been reliably shown to be useful should not be promoted.

Problems resulting from inadequate tests of treatments continue to occur. Again, because of unreliable evidence and aggressive marketing, millions of women were persuaded to use hormone replacement therapy (HRT). It was claimed that, not only could it reduce unpleasant menopausal symptoms, but also the chances of having heart attacks and strokes. When these claims were assessed in fair tests, the results showed that in women over 60, far from reducing the risks of heart attacks and strokes, HRT increases the risks of these life-threatening conditions, as well as having other undesirable effects.These examples of the need for fair tests of treatments are a few of many that illustrate how treatments can do more harm than good. Improved general knowledge about fair tests of treatments is needed so that –laced with a healthy dose of scepticism –we can all assess claims about the effects of treatments more critically. That way, we will all become more able to judge which treatments are likely to do more good than harm.

DES DiEthylStilbestrol Resources

The Primodos Issue ; debated by Lord Alton of Liverpool

UK Parliament, House of Lords Hansard, 28 February 2019

“They have faced the implacable determination of regulatory bodies spending huge amounts of public money on ad hoc scientific reviews to cast doubt on the work of highly reputable scientists. Those who have suffered so grievously deserve much better than this.”

Safety of Medicines and Medical Devices, House of Lords Hansard Debate, 28 February 2019.

More information

Primodos, Sodium Valproate, Surgical Mesh : Baroness Cumberlege Talks

UK Parliament, House of Lords Hansard, 28 February 2019

“For the families involved, it is life-changing and extremely distressing. For those women who took Primodos and sodium valproate, there is an intense feeling of guilt. They took the medication and they blame themselves. However hard one tries to persuade them that it was not their fault, the guilt remains.”

“That tells me something is seriously wrong; the system is not working as it should. People who have been harmed should not have to fight to be heard or to access the care they need.”

More information

Lords debate Safety of Medicines and Medical Devices

UK Parliament, House of Lords Hansard, 28 February 2019

Lords debates medicinal safety in ref to the public health scandals involving the hormone-based pregnancy test drug Primodos, the use of vaginal mesh implants and the anti-epilepsy drug sodium valproate.

  • Interventions from Lord O’Shaughnessy, Lord Hunt of Kings Heath, Baroness Walmsley, Baroness Masham of Ilton, Baroness Cumberlege 36:28 , Lord Brennan, Lord Carrington, Lord Bethell, Baroness Bryan of Partick, Lord Alton of Liverpool 1:17:29 , Lord Suri, Baroness Finlay of Llandaff, The Earl of Dundee, Baroness Jolly, Baroness Thornton, The Parliamentary Under-Secretary of State, Department of Health and Social Care (Baroness Blackwood of North Oxford).
  • Read the Lords Hansard transcript.
  • Parliamentary news, research briefing.
  • Parliament news, press release.
  • Video source, Parliament Tv.

More information

EWG Primodos review criticised for ‘not assessing risks properly’

Sky News’ Exclusive, 4 Apr 2019

“Somebody, somewhere, has performed a meta-analysis.
At some point, there was a decision not to include that in the EWG report.”

Oxford University professor Carl Heneghan has told Sky News the evidence against the pregnancy drug Primodos suggests a significant link between it and birth deformities (see below).

On 23 April 2019, during the Westminster Hall debate on the Expert Working Group report on hormone pregnancy tests, some MPs also questioned the methodology used and asked why not use meta-analysis to assess primodos evidence ? Some others clarified the difference between correlation vs causation (in ref to primodos link to births defects), or said : with pharma funding the MHRA, can the EWG report be fully independent ?

More information

  • 2018 Studies :
    • Oral hormone pregnancy tests and the risks of congenital malformations: a systematic review and meta-analysis, F1000Research, 31 Oct 2018, 7:1725, DOI:10.12688/f1000research.16758.1.
    • The Primodos components Norethisterone acetate and Ethinyl estradiol induce developmental abnormalities in zebrafsh embryos, Nature, 13 Feb 2018, DOI:10.1038/s41598-018-21318-9.
  • Westminster Hall debate pack : CDP-2019/0095, 18 April 2019.

With Pharma funding the MHRA, can the EWG Report be Fully Independent ?

Westminster Hall debate, 23 April 2019

” Yes we had a (Primodos) review, but I’m not sure we can call that ‘independant’ “

Yasmin Qureshi MP calls into question the independence of the MHRA in its central role on the expert working group (EWG), for their report on the use of hormone pregnancy tests (HPTs) and adverse effects relating to pregnancy including possible birth defects.

Westminster Hall debate, 23 April 2019. Video reference.

More information

Vulvar Cancer is on the Rise in the UK, study shows

Relationship between vulvar symptoms and incidence of vulvar cancer in women referred to a rapid access clinic

Abstract

Objective
We performed a study to estimate incidence of vulvar cancer in women with vulvar symptoms (irritation, pain, bleeding +/− presence of lesion) referred to a secondary care, rapid‐access clinic.

Methods
Prospective data collection of all direct referrals from a primary to a secondary care gynecological oncology clinic from 2011 to 2016, for women with suspicious vulvar symptoms.

Results
32/393 (8.1%) women had vulvar cancer, and 24/393 (6.1%) had a premalignant lesion. Multivariate logistic regression showed that women referred without a specific lesion had considerably lower odds of a diagnosis of vulvar cancer than those with a lesion (OR=0.11, 95% CI: 0.03–0.49). In total, 30/234 (12.8%) women with a vulvar lesion (mass or ulcer), had vulvar cancer, compared with 2/159 (1.3%) of those referred without a lesion (these patients had vulvar irritation and bleeding but had a visible lesion on examination). None of the 140 women with irritation alone, in the absence of a visible lesion or bleeding, had pre‐invasive disease or cancer.

Conclusion
Presence of a vulvar lesion, especially if painful/bleeding, has a high positive predictive value for vulvar cancer and 12.8% of women presenting with any vulvar lesion to secondary care had cancer.

Discussion

Overall, 32/393 (8.1%) women referred with vulvar symptoms were diagnosed with a malignancy and a further 24/393 (6.1%) were diagnosed with a pre‐malignant lesion. The presence of a suspicious vulvar lesion, especially if symptomatic, was associated with a cancer diagnosis in 30/234 (12.8%) women. Unless the lesion is obviously benign (which in this series included: sebaceous inclusion cyst; urethral caruncle; inspissated sebaceous material trapped under the clitoral hood; and ecchymosis on a background of lichen sclerosus), a rapid‐access clinic referral is appropriate to exclude a malignancy in these women. A larger cohort would be required to stratify risk by age. However, as the incidence of vulvar cancer is increasing, especially in younger women, it would be important not to dismiss suspicious symptoms based on age alone.

Women without a visible lesion were extremely unlikely to have cancer, based on this cohort of patients (<1.3% depending on whether presence of lesion was defined by patient/primary care [0/159] vs secondary care (2/159). However, these findings would need to be applied to a larger population to test this hypothesis and with wider generalization to other healthcare populations. One strength of this dataset is that the population is well‐defined and relatively stable, so re‐referrals, subsequently diagnosed with a cancer, would have been referred back to the same clinic. In the present series, six women were re‐referred during the study period (one woman was seen three times and five women were seen twice). None of these women had a cancer diagnosis. Unfortunately, we were not able to define an average lead time from onset of symptoms to diagnosis from our data, since this was not reliably recorded.

Many women with symptoms of vulvar soreness and irritation, in the absence of a specific lesion, were diagnosed with an inflammatory condition; half had lichen sclerosus or lichen planus. Primary care physicians should be reassured that, in the absence of a suspicious lesion, a cancer diagnosis or a pre‐malignant condition was unlikely. As per dermatology guidelines, if there are classical signs of lichen sclerosis, a diagnostic biopsy is not needed if there is a response to high‐potency topical steroids. Women whose symptoms do not start to improve after 2–3 weeks of treatment, should be re‐assessed and referred if symptoms persist or a lesion develops.

The overall incidence of vulvar cancer in this series was slightly lower than in those in the literature. This is perhaps surprising, as the local population is relatively elderly; in the 2011 census 29.1% of West Somerset were aged 65 years or over compared with a UK average of 16.4%. However, this series is nearly 10‐fold larger than those previously published, so may reflect the increased statistical power of this study. It may also reflect a difference in local referral patterns and criteria, since women may be referred to a general gynecology clinic or a dermatologist in other areas. One study from an urban area in the South East of England looked at cancer diagnosis rates in women referred to a gynecologic fast‐track clinic. They included a total of 335 women referred to secondary care. Only 18 women had symptoms suggestive of vulvar cancer, of whom only two (11.1%) were diagnosed with vulvar cancer. A similar study, also from the South East of England, included only 13 women with vulvar symptoms and found a positive predictive value of a cancer diagnosis of 15.4% in this subset. A further series of women with suspected gynecologic malignancy, also from the South East of England, included 1105 women referred to secondary care. Forty‐four of these women were referred with suspected vulvo‐vaginal cancer and 13.6% were diagnosed with a malignancy.

These data help to differentiate between those who should be referred via a fast‐track clinic and those who could be treated more conservatively for vulvar symptoms, and could help to inform national guidance—including future updates of NICE guidelines. Further data would be required to determine whether these data had wider applicability in other, more diverse populations, and these data are limited by the secondary care focus of this study. It would be interesting to compare rates of women presenting with vulvar symptoms in a primary care population with secondary care referrals and cancer incidence in that population over the same time period to provide evidence for triage of those requiring rapid assessment. This is important in the face of the increasing incidence of vulvar cancer, due to changes in demographics and HPV prevalence.

Finally, many women delay presentation and may have significant symptoms for many months, due to fear, embarrassment or lack of awareness of vulvar conditions generally, and vulvar cancer in particular. Many women in our series had inappropriate treatment for vulvar skin conditions or suspicious lesions with low potency steroids, topical estrogens, anti‐fungal agents, antibiotics, or no treatment at all for prolonged periods. This may be due to ‘home remedy’ self‐treatment (possibly fueled by lack of knowledge and advertisements for ‘intimate itching’), avoidance of doctors, or reluctance to use adequate courses of high potency topical steroids, by both physicians and the women affected. Research is also needed to inform us about barriers to presentation, especially in older women, who are most at risk of vulvar cancers, and to improve health education for this under‐resourced area.
Feature image credit @DiggerGraham (see below).

Primodos Link to Births Defects : Correlation vs Causation

Westminster Hall, 23 April 2019

“There is an association. The Minister must go from this room knowing that, because that is what the science shows.”

Edward J Davey MP clarifies how to correctly interpret Carl Heneghan’s study (see below). Video (above) reference.

More information

Expert Working Group Methodology Used for Hormone Pregnancy Tests Report

Westminster Hall, 23 April 2019

“When the data was pooled together properly it shows a clear association with several different forms of malformations. …
…The Expert working Group completely failed to use the right approach for systematic reviews”. …

Yasmin Qureshi MP questions the technique used by the expert working group, for their review, on the use of hormone pregnancy tests (primodos drug) and adverse effects relating to pregnancy including possible birth defects. Westminster Hall debate, 23 April 2019. Video reference.

More information

Primodos hormone pregnancy test drug tragedy : what exactly is being covered-up ?

Westminster Hall, 23 April 2019

Is it the legal side involving the NHS, is it the pharma industry influence on the official report, on the MHRA, is it the government wrongdoing, or is it a bit of everything that is being covered-up ?

Sir Mike Penning MP argues that the Expert Working Group review on Primodos was a “cover-up”. Reference.

More information