Search This Blog

Wednesday, July 30, 2008

Peer review

Peer review has been mentioned a lot within the discussions surrounding the STFC funding crisis. Notably it was mentioned during the House of Commons debate a few weeks ago, though much of the discussion came across to me as somewhat fuzzy in its logic.

In the debate there was much talk of whether peer review is the best way to go, how no system is ever perfect and how dangerous it could be for the select committee or anyone else to attack peer review.

I think this missed the point completely; the underlying assumption throughout the debate was that peer review describes a single process, whereas we all know that peer review is something of an umbrella.

This bothered me before when there were admonishments from some that the community should be careful when attacking peer review. It is a false answer; who in the community has really attacked the concept of peer review? Anyone? I have heard plenty of people attack how STFC implemented a process that they claimed was peer review. You see this is just more spin to deflect a real question and it bothers me that we have not heard more about this. So what is the real problem with 'peer review'?

In my view the issue with peer review in the STFC was not that 'peer review' as a concept was fatally flawed, rather that the practical implementation of peer review was rubbish in the eyes of many of us.

With the best will in the world a small panel with no advisory structure could not be considered to be effective peer review for the wide-ranging programme that STFC was responsible for. PPAN did the best they could but let's face it they are only human.

Arguments that these are intelligent people and so they were perfectly capable don't hold water; yes they are intelligent but that is not enough in a situation where community confidence is essential and where people's livlihoods may be impacted. You need intelligence and breadth of experience and familiarity; a proper balance.

When someone argues that smaller committees are better because they are efficient and don't try to please everyone you know they have not really thought through the implications of what they are saying with regard to a smaller panel. Yes, a smaller panel might reach a decision quickly but one lacks the confidence that they might reach the correct decision.

I note that peer review came up again in the letter from Phil Willis MP to John Denham:

It is not clear why the Government goes on to discuss peer review in this section, since peer review was not mentioned in recommendation 6 or its preceding text and was discussed later in the report. Be that as it may, the Government’s assertion that we criticised “the outcome of STFC’s peer review process” and “those researchers who have undertaken it” is an inaccurate paraphrasing of the serious concerns we raised in relation to STFC’s peer review system and decisions made by STFC. We did not criticise the outcomes of STFC’s peer review. Specifically:
(a) on the International Linear Collider, we did not comment on the scientific justification for withdrawal, but raised some concerns that had been put to us during the inquiry;
(b) on Gemini, we did not consider the merits of STFC’s decision, but the way it went about making its decision, or as it turned out, indecision, public;
(c) on solar-terrestrial physics, we questioned Professor Mason’s explanation for the withdrawal of funding, and suggested that STFC renege on that decision until its community had been properly consulted.
Neither did we criticise the members of the peer review panels. On the contrary, we acknowledged that STFC’s peer review committees “have a difficult job to do” and that “we do not doubt the integrity of the individuals who make up those Committees” (p 32 of our report).
Sadly the Secretary of State's reply is less than promising and seems to boil down to a game of statement of 'I'm not going to debate this with you, game over'.

I suppose he has been too busy running around letting everyone know that Gordon Brown is still great and we should stop attacking him because the conservatives are much worse.

The letter is worth reading in full as is the one to Prof. Keith Mason, who to his credit (though not much) gave a more fulsome response to the points raised.

No comments: