P2P Draft Evidence Review

Can a Process that is Inappropriate for ME/CFS Inform the Research Path Forward?P2P_title

Summary Overview

As a research focused organization, the Solve ME/CFS Initiative (SMCI) understands the impact a program like the Pathway to Prevention workshop (P2P) can have on the research landscape.  Because of its importance, we have utilized the collective brainpower of our Research Advisory Council, led by our scientific director, Suzanne D Vernon, PhD, to perform a careful review and response to the Evidence-Based Practice Centers’ draft evidence-based review for ME/CFS. (Read our official submission HERE)  We have concluded that:

  • It confirms what we and others have suspected: the Pathway to Prevention (P2P) process is not appropriate for ME/CFS.  Among other reasons, it uses comparative effectiveness methodology to review the evidence to inform healthcare; in the case of ME/CFS the evidence base is much too slim for this method to be effective.
  • The P2P systematic review of the ME/CFS evidence base illustrates the lack of rigor in crucial research design elements and the absence of high quality clinical trial data. The review shines a bright light on the need for well-designed, adequately powered studies to identify diagnostic gold standards and safe and effective treatments.

Furthermore, it is important to note that the vast majority of ME/CFS research studies were omitted from this evidence review due to their inability to meet the rigorous standards required for inclusion. Of all of ME/CFS research studies funded by the NIH since 1991 ($191.5 million spent), less than 1% of those studies were included. So, not only has the NIH egregiously underfunded this illness (based on prevalence), but the research NIH has funded has not contributed to an evidence base that has moved the science forward.

This review substantiates the negative cycle currently in play with ME/CFS research – Because there is little funding, there is little research. And because there is little research, there is little evidence to prompt additional funding. But the P2P process may be what is needed to break through that negative cycle and move us to a day when ME/CFS is appropriately funded. It’s existence demonstrates that there is increasing recognition of the clinically devastating nature of ME/CFS and recognition of the disease as an urgent area of future study.


Background – How did we get here?
To review what has led up to this NIH P2P workshop for ME/CFS:

  • The 2011 NIH State of the Knowledge Workshop for ME/CFS concluded that there were problems with definitions, gaps in study design, and a lack of studies on co-morbid conditions, biomarkers, or genetics.
  • A year later, at the CFS Advisory Committee meeting, Dr. Sandra Kweder, FDA Deputy Director of the Office of New Drugs tells us there are so few applications to the FDA because there is no accepted ME/CFS definition, no accepted method for measuring how patients feel or function and no accepted biomarker to provide a simple, quantitative measure of disease presence or activity.
  • In 2013, ME/CFS is the first “patient-focused drug development” workshop held by the FDA. The outcome of that meeting, as expressed in the Voice of the Patient report, is that ME/CFS is a serious disease with significant unmet medical need that requires comparative effectiveness research to generate the evidence to assess the effectiveness, benefits, and harms of different treatment options.
  • In 2014, Dr. Susan Maier, Deputy Director of the Office of Research on Women’s Health at the NIH, who oversees ME/CFS research at NIH, successfully garnered a Pathways to Prevention workshop titled, “Advancing the Research on Myalgic Encephalomyelitis/Chronic Fatigue Syndrome”.

For much of this year, the Solve ME/CFS Initiative (SMCI), as well as many other individual advocates and patient groups, has been talking about the National Institutes of Health (NIH) Pathways to Prevention (P2P) for ME/CFS workshop. In April, we first reported on the possibility of the P2P workshop, then in the approval process. Once it was approved, the NIH worked with the Agency for Healthcare Research and Quality (AHRQ) and their Evidence-Based Practice Centers to conduct this evidence-based review.

In June, SMCI expressed concerns that the search criteria used for the evidence-based literature review would bias the review toward studies on CBT and GET. On Sept. 22nd AHRQ published the draft systematic evidence review on the Diagnosis and Treatment of ME/CFS for comment.  Several members of our Research Advisory Council together with Suzanne D. Vernon, PhD reviewed the draft and have provided peer review comment directly to AHRQ. SMCI and our research advisors believe that this systematic review has identified important and significant gaps in research that have contributed to the dearth of evidence needed to diagnosis and treat ME/CFS. Indeed, it is almost all “gap” and almost no substance.


Dearth of Evidence
From 1991 to 2014, NIH Research Portfolio Online Reporting Tools (RePORT) indicates a total of $191.5 million has been awarded to investigators to directly study ME/CFS or conduct research relevant to ME/CFS.  Spending for other medically unexplained disorders that often occur with ME/CFS, like Fibromyalgia and irritable bowel syndrome (IBS), is about $400 million for each in the same time frame. This is in contrast to multiple sclerosis (MS) – a condition with objective diagnostic biomarkers and treatments – where NIH spending in the same period exceeds $4 billion.

Further, a search of PubMed.gov for anything published on ME/CFS finds 5,450 peer reviewed scientific articles; there are 8,000 articles for Fibromyalgia and 9,000 for IBS.  This is in contrast to the 55,500 articles found in PubMed that make up the MS evidence-base.  There is a direct correlation between funding levels and the breadth of the published evidence-base.


The One Percentexamine
Now, the NIH P2P draft systematic review is out. All told there were 5,901 papers considered by the Evidence-Based Practice Center in the comparative effectiveness systematic review. In the end, less than 1% of these 5,901 paper met criteria for inclusion in the systematic review process. Why? 

  • Because comparative effectiveness systematic reviews are based on evidence-based facts. The P2P uses a rigorous process to empirically identify methodological and scientific weaknesses in each published paper.
  • The investigators conducting these evidence-based reviews are experts in this review process and come to the task at hand without any bias or preconceived notion about the topic they are reviewing.
  • The procedures for these reviews are possible because there are standard methods and procedures in place for conducting research; especially human research intended to impact clinical care. They systematically reviewed the ME/CFS papers using specific research design criteria known as PICOTS – Populations, Interventions, Comparators, Outcomes, Timing, and Setting.

While the draft comparative effectiveness review on the Diagnosis and Treatment of Myalgic Encephalomyelitis/Chronic Fatigue Syndrome (ME/CFS) may be a hard pill to swallow for patients and for the many ME/CFS scientists whose research studies were excluded from this review, it identifies vast research gaps and provides important information needed to chart the way forward.

The rationale for evidence-based reports makes sense:

  • “Systematic reviews are the building blocks underlying evidence-based practice; they focus attention on the strength and limits of evidence from research studies about the effectiveness and safety of a clinical intervention. In the context of developing recommendations for practice, systematic reviews can help clarify whether assertions about the value of the intervention are based on strong evidence from clinical studies.”

And …

  • “This review is not intended to address the question of etiology nor underlying factors that lead to the onset or perpetuation of ME/CFS but rather to focus on the diagnosis and treatment of this syndrome.”

With this is mind, the review identified only 64 studies that met evidence-based criteria to address the key questions; 1) ME/CFS diagnosis and 2) ME/CFS treatment. For a diagnosis study to be included it had to discriminate ME/CFS patients from healthy controls and (ideally) from other diseases with similar symptoms and if the study used biochemical marker it had to be in the context of treatment versus etiology. Twenty-eight studies addressed various aspects of diagnosis. Most of the studies pertaining to diagnosis were rated as fair quality (on a scale of “good”, “fair”, “poor”). These diagnosis studies showed that certain measures that assess function are able to distinguish between ME/CFS and healthy people reasonably well. The studies were rated “fair” because they were small and diseases similar to ME/CFS were seldom included. The lack of a gold standard for diagnosis of ME/CFS limits how broadly the limited evidence base can be generalized to the ME/CFS community as a whole. The review notes that clinical experts identify post-exertional malaise (PEM) as a cardinal feature of ME/CFS yet current methods of testing, comparing, and monitoring PEM are lacking.


So what few studies WERE included and do they shed light?
There were 36 randomized clinical trial studies for ME/CFS treatment; 9 for medications, 14 for counseling or behavioral therapies, 7 for complementary and alternative medicine, 6 for exercise, and 5 comparing therapies:

  • Seven were rated as “good-quality” trials: 5 counseling and behavioral therapies, 2 complementary and alternative medicine therapies
  • 24 were rated “fair-quality”
  • 5 were “poor-quality”
  • The phase III Ampligen trial was rated “fair to good-quality” and it was the only medication trial to get this rating; all other medication trials were fair and poor-quality.
  • The PACE trial was rated as “good-quality” for both counseling and behavioral and for exercise intervention but the strength of the evidence was moderate to low for cognitive behavioral therapy (CBT) and graded exercise therapy (GET).
  • The other 5 studies that used an exercise intervention were “fair-quality” and overall, exercise intervention trials showed moderate strength effect in function and well being and low-strength effect on fatigue. Importantly, the high rate of refusal in one study may indicate that patients are concerned the possible adverse effects of repeat exercise testing (e.g., the first exercise test may have caused PEM prohibiting the ability to do the second test). This deserves further study and consideration as a possible outcome.

All of the clinical trials were compromised because of small and heterogeneous patient populations, differences in inclusion criteria – including use of different case definitions, and measuring different outcomes (e.g., fatigue, function, etc). This is exactly what Dr. Peter Rowe reported at the 2013 FDA workshop on Development of Safe and Effective Drug Therapies for CFS and ME (http://www.tvworldwide.com/events/fda/130425/). Measuring the effect of an intervention was likely diminished by the above factors. Ways to better detect the effect of an intervention are to measure certain clinical features present before treatment starts compared to after treatment, such as severity and frequency of specific symptoms like brain fog or PEM. This is called subgroup analysis; not one of the intervention studies did a subgroup analysis.


Is there a Silver Lining?
Despite the lack of an evidence-base, this systematic review shines a light on important areas that must be pursued to build the ME/CFS comparative effectiveness research evidence base:

1.  Determine diagnostic criteria that can be used as the gold standard for ME/CFS diagnosis.

  • We note that the work of determining ME/CFS diagnostic criteria is currently underway at the Institute of Medicine (IOM)

2. Test gold standard diagnostic criteria in other populations with diseases similar to ME/CFS where diagnostic uncertainty exists so that the treatment is specific to ME/CFS vs another similar condition.

  • Determine the outcomes that are clinically meaningful and ensure standardized assessment so that the effect of interventions can be measured
  • Post-exertional malaise is a cardinal feature of ME/CFS and potentially one of the most important outcome measures; research should be conducted to determine what it is and how to measure it

3. The biomarker research and clinical trials conducted to date provide clues to the possible causes of ME/CFS and can serve as “disease models” for further study of ME/CFS pathophysiology

  • All basic and intervention research should use methodological standards to  ensure that it meaningfully contributes to the ME/CFS evidence-base

This analysis illustrates the lack of coherence in the field and the absence of high quality clinical trial data.  NIH, through the P2P workshop, set out to identify gaps in the research, and they found more gaps than substance. Yet the existence of this effort shows increased recognition that ME/CFS is an urgent area of future study and clearly implies that more resources need to be focused on well designed, adequately powered studies. We believe that it is NIH’s responsibility to address their own finding.


What Happens Now?
The P2P panel will review this evidence-based report, including the comments and feedback now being submitted, in advance of the meeting that will take place on December 9 & 10, 2014. At that two-day workshop in Washington DC, the P2P panel will hear from the expert speakers and be able to ask clarifying questions in a town-hall-like Q&A that will take place after each session.

Experts in ME/CFS are being invited to address each agenda item. They will speak to their personal experience and expertise as a patient, caregiver, researcher, etc. The slate of speakers has not yet been released.

The day after the P2P meeting, the P2P Panel will write a draft report which will be published and the public will have time for comment on the final report, just as we are now commenting on the draft evidence review. The comment period is 30 days; the deadline should be around January 12th, 2015. Once the comment period closes, the report will be finalized and NIH will organize a plan to disseminate it widely.

The goal, as we understand it, is a report that contains a set of recommendations based on the totality of the evidence, in the hopes of having said recommendations carried out by Federal partners in 2015. These recommendations are aimed at improving the nature of the research being conducted in ME/CFS. The Solve ME/CFS Initiative will pass along additional information as it is received and when the opportunity for public comment is scheduled, we will report that and share the means by which you can participate.

For far too long, ME/CFS has been underfunded, misunderstood, and even disbelieved. Our community has been pushing a very heavy fly-wheel aimed at increasing understanding, acceptance and progress toward treatments and cure. Now there is federal activity, all aimed at moving the needle for ME/CFS – FDA Voice of the Patient, IOM activity on creating clean diagnostic tools and other much-needed tasks, and this P2P. What this evidence based report shows is that the status quo must change. We hope P2P will illuminate the clear need for more funding, increased research activity and faster progress.


What Can You Do?
Registration is now open for the Pathways to Prevention workshop for Advancing the Research on ME/CFS taking place on December 9 & 10. You can register to attend live or participate via webcast. It is our hope that many stakeholders will participate in this process in order to ensure the patients have a strong presence and a voice.

To register to attend live click HERE 

To register for the webcast click HERE



Many other patients and advocates are also weighing in on the P2P and the draft evidence review. Read what just a few others are saying:

Medscape Medical News – a log in is required to view the article, but it is free and anyone can create an account.

Jenny Spotila, Mary Dimmock & others via Occupy  CFS

Cort Johnson via HealthRising