Peter Bates > The 37 Test

The 37 Test

For more than twenty years, the National Institute of Health Research (NIHR) has promoted the idea of Patient and Public Involvement (PPI), encouraging research teams to include Public Contributors in every stage of research production. I recently came across a filenote I had written back in 2014, when I had checked out a sample of completed application forms submitted by research teams requesting NIHR funding. The note was brief, simply saying that 37 successful application forms had been checked, and only two of them had made any reference to published work on PPI.

So in an idle moment in December 2021, I repeated the exercise. In my small sample, half of the research teams made an effort to build some PPI into their bid and more research money was awarded to proposals that lacked PPI than those that tried to include Public Contributors. So this evidence suggests that the NIHR community can briefly celebrate its achievements before setting out on the next part of the journey. Reasons for this conclusion are given in the paragraphs below.

Thirty seven successful applications again, just to be neat. But this time I was a little bit more rigorous. I used the NIHR Funding and Awards Search Website, typed ‘protocol’ into the search box, selected ‘outputs’ in the tab below the search box and filtered to select ‘protocol’ and ‘2021’ and looked at research grants worth in excess of £1million. I reasoned that recent, high value research was more likely to have good PPI. Where a research study had uploaded more than one version of their protocol, I looked at the most recent. I simply worked down the search list until I had reviewed 37 of them by searching the full text of the protocol for ‘PPI’, ‘public’ and ‘involvement’ and pasting all relevant text into a document that I could analyse. Here’s what I found.

  • Seventeen (46%) had a PPI section, 8 had written fewer than 150 words in the PPI section, 7 had written 150-300 words and 2 protocols had 300-450 words. It is hard to detect evidence of a clear strategy for coproducing research with Public Contributors if there is no separate PPI section in the application form or if the content of that section is very brief.
  • The 37 protocols represented a combined budget in excess of £87 million, with over £50 million (58%) given to studies that have no PPI section in their protocol. Official guidance advises “Do not under budget for involvement when applying for research funding. Reviewers will check whether the costs reflect the reality of effective public involvement or the support/training you are planning.” This statement has led to the widespread belief that PPI is so important that NIHR reject applications from research teams that pay scant attention to it, but these figures suggest otherwise.
  • Seven protocols (19%) referred to guidance documents or academic work on PPI. Three of them referred to guidelines (two on payments and one on general PPI), one of these also mentioning a James Lind Alliance (JLA) Priority Setting exercise that informed their choice of research topic, making three references to JLA in all. One protocol committed the team to use of GRIPP2 to shape their reporting of PPI. No references were made in any of the 37 protocols to any other academic publications that underpin PPI activities. It is odd that the flurry of research activity investigating PPI has not been reflected in a scholarly approach to the PPI words used in research bids.
  • Nineteen protocols (51%) named one or more individual Public Contributors in their submission. Eight named one, nine mentioned two and two protocols named three Public Contributors. We might wonder how often other vital contributors to the research team (statisticians, for example) remain unidentified and anonymous until the project has begun. Of the 32 named individuals, seven (22%) appeared to be employed professionals working for a relevant third-sector patients’ organisation. Such individuals may bring considerable expertise, but not the challenge delivered by a rank outsider.
  • Seventeen protocols (46%) planned to assign at least one place on their Trial Steering Committee to a Public Contributor, including two protocols that assigned two places to Public Contributors and two more that committed to offer three places.
  • Five protocols (13%) indicated that Public Contributors would help to determine the interview questions (topic guide), although none of them involved Public Contributors in data collection. Two protocols planned to involve Public Contributors in data analysis and interpretation.
  • Three protocols (8%) offered training to Public Contributors to enable them to fulfil their role.
  • No protocols committed to offer opportunities for Public Contributors to co-author their publications or to speak at conferences.

Perhaps there are specialisms where these percentages would be higher – mental health research, perhaps, or Participative Action Research – or perhaps my sample was too small to be taken seriously. But what might happen if other people repeated the “37 Test” on different dates? Who knows, we might even trigger some action!