Aren’t You Curious?

From the CEO’s Desk
February 2012

The last time I visited my parents, we took a walk through their neighborhood. A house around the block was obviously the site of some kind of home improvement project. A stack of brick, lumber and Sheetrock was piled near the driveway and a portable storage pod was parked at the end of the driveway closest to the garage. There was a Porta-John in the side yard and, just beyond it, a backhoe.

“What are they building?” I asked. “We haven’t figured it out yet,” my dad responded. (He’s an electical engineer by training and a do-it-yourself’er by choice, so I knew he was piecing together the information.) “We noticed all the different contractors’ trucks — plumbers, electricians, painters, landscapers — passing our house months ago, and then there was no activity for weeks,” my stepmother added. (She’s very observant.) “They were probably getting bids and finishing the plans. The town takes forever to issue permits. This week there have been several deliveries,” my father continued. “But you don’t know what it is they’re building?” I asked again. “Well, we could make some educated guesses based on the trucks I’ve seen go by and what other neighbors have told us, but nobody seems to know.”

Aren’t you curious?

The reason for this little story is that it’s like the work we’ve been doing to launch the next phase of our research program.

On April 6, 2011, we broadly issued a Request for Applications (RFA) to the research community, inviting their ideas and proposals for research projects that would advance diagnostics and treatments for CFS. {To help illustrate the process, think of a homeowner who has saved money for a home improvement project. She has a certain amount of money she can spend, but doesn’t know whether to add a pool, a home theatre, a new master bedroom suite or to remodel a kitchen or family room. Think of our RFA as a notice placed in the local paper to outline a range of home improvement projects on this particular home and the need for contractors to bid on the projects so she can decide how best to invest her money in upgrades.}

On June 3, 2011, we received 37 letters of intent that were screened for alignment with the specifics of our request. {Think of those as cursory bids to do a particular job within the range of possibilities listed in the ad.}

On June 27, 2011, we invited 26 of those investigators to make full applications. {Think of this step as identifying preferred contractors who would get the chance to pitch their plans in greater detail. She might discard the pitches for projects she wasn’t interested in, like a screened porch or custom cabinetry for the garage, or for things that exceeded her stated budget, like adding another story to the house.}

Over the summer months, the 26 teams worked to complete their detailed applications and also had to agree that, if funded, they would comply with a new set of policies issued by the Association to guide the expectations and performance during the grant period. {Think of this time as the period during which contractors would complete their blueprints and determine what materials to use, how many hours of labor it would take to complete the project and what their total costs would be. They would also have to work the project into other commitments and evaluate whether this job would help them get other jobs in the neighborhood or larger projects in the future.}

During this same time, our scientific director, Dr. Suzanne Vernon, was contacting experts from all the subject areas and disciplines that would be needed to evaluate the proposals. She used the letters of intent to identify the types of experts needed and scoured the literature for top people working in related areas. After securing their voluntary assistance and verifying that they didn’t have any conflicts of interest with any of the proposals expected, she obtained signed non-disclosure agreements and oriented them to the review process. Suzanne had to contact about three times as many experts as needed, due to attrition of the pool to accommodate scheduling issues or conflicts of interest. {Now this step probably wouldn’t happen with your average home improvement project, but for the sake of our example here, let’s say the homeowner solicited input from other people who could have done the work but didn’t submit a bid. They are experts in their trades who can help her assess the bids and make some recommendations about strengths and weaknesses, or things she should make sure another contractor could do well.}

On September 30, 2011, the full applications were received and checked for completeness and conformity to the requirements. They were then mailed to the 48 reviewers with score sheets that outlined the 10 scientific merit factors and the nine-point rating scale they were to use to evaluate the proposals.

Over a six-day period in November, Suzanne and I met by conference call with 26 “pods” of reviewers to discuss each proposal for 60 to 90 minutes. The international composition of the review teams meant accommodating time zones across the planet and also addressing cultural differences in rating practices. A clear delineation of the nine points on the rating scale, thorough discussion of the proposals and a scoring algorithm developed for our 2008 review process helped identify each proposal’s strengths and weaknesses, as well as ways that good proposals could be made even better. {Returning to our home improvement project analogy, you can think of this step as the homeowner getting expert advice from other tradesman on the techical and engineering sound-ness of the various contractors’ proposals, including quality of materials, skill of the people to be used on the job, performance based on past jobs and likelihood that the finished project will meet her specifications.}

Next, each of the proposals that scored a 4.0 or better was rated for its strategic merit using the same nine-point scale, applied to eight different strategic factors. {This would be like asking designers and realtors for their input on the various projects — which ones are likely to enhance her enjoyment of the home, improve its aesthetics and add resale value should a sale occur later?}

The final step in the review process was new for this round: we call it “collaborative refinement.” Just before the holidays, we scheduled conference calls with each of the principal investigators for proposals that rated the best for scientific and/or strategic merit. During these calls we discussed the reviewers’ comments and the strengths and weaknesses identified. The applicants then addressed these issues in writing and amended their proposals to adjust specific elements. {Now she calls back a select group of the contractors and asks them to tweak their proposals based on the advice of the various experts consulted.}

At long last, it was decision time. Research funding decisions rest with the Executive Committee of our board of directors — the chairman, vice chairman, secretary, treasurer and CEO (me). The proposals and amendments, factor scores and detailed executive summaries were absorbed by the committee. Over two evening meetings in mid-January, the members met with Suzanne and our chief financial officer to make decisions that optimized scientific promise, strategic alignment and shrewd financial investment. {The homeowner looks over all the proposals, all the input she has requested and received, her bank statement and the calendar, and makes the final decision about which of the projects to contract for.}

All of the applicants to the RFA were notified about the outcome of the review on Jan. 20, 2011. Those who were not successful received the summaries of reviewers’ comments to help them improve the proposal for submission to other funding opportunities, like the NIH program announcement. The investigators whose proposals were selected for funding immediately began the first steps of compliance during which they (and any collaborators from inside or outside their institution) and institutional officials must agree to the terms of the award. This step also involves securing approval from institutional review boards (IRBs) and planning for the first in-person meeting of all the investigator teams. {Now that the homeowner has decided which home improvement project and which contractor(s) will perform the work, they have to secure the necessary building permits, agree on terms of payment, arrange the financing and purchase supplies before work can begin.}

We are also completing some tasks on our end and will soon be ready to announce the new funding awards and describe how they fit into our integrated research strategy. In fact, we’re aiming to make an announcement to the community and the press by Feb. 24. And we’re ready to break ground!

Aren’t you curious?

K. Kimberly McCleary served as the Association’s chief staff executive from 1991-2013.

 

February 7, 2012