Reviewing the Reviewers?

The idea of “who is reviewing the reviewers?” has been on my mind for some time.

Then yesterday this particular article showed it to the extreme. It’s by a Canadian author critiquing a film and also critiquing some of the reviewers of the film. It’s a really interesting review of how 3 layers of white men look at how colonial aftereffects are played out for indigenous populations (context is Innuit people). First white man is the film maker who NEVER actually visited the places or met the people the film is supposedly about. He used YouTube videos taken by those people, picked some pretty horrible ones for the meat of the film, added some footage from other locations (!) and called it anticolonial. The reviewer mentioned in the article NEVER watched the film but still had an opinion on it. The writer of the article critiques them both for being white men presenting the Innuit people in biased ways. The author seems himself to be white, but maybe he’s part Innuit, lived with Innuit people before, or is married to an Innuit woman. I wouldn’t know.

My point is this: the author is reviewing the review, holding the reviewer accountable.

Now think of an academic equivalent of this (and I have heard it said before): scholars who research and speak on behalf of indigenous people when they themselves are white. I have heard of entire panels of these in Australia. And think of all these experts on it who would be reviewing all the output on that. I understand the responsibilities dominant people have to advocate for justice. I just don’t understand how they can do so while continuing to represent other people’s voices rather than empower those voices to be heard on their own terms.

Think of this: how academic research is sometimes using sources in the “researched subject’s” voice (whether from existing writing, from interviews, from observations or open ended interview questions) and piecing it all together from the researcher’s perspective, presenting it to the world as knowledge, as scholarship.

Non-participatory ethnographic research is quite like that film that’s just been reviewed. Except maybe the scholars are clearer on their values and ethics (but this doesn’t necessarily mean the filmmaker lacked them – i am by no means qualified to critique that film, knowing nothing of Innuit people or of the film itself – I am just using the article as an example of a critique of a critique). It also doesn’t mean that our ethical standards or procedures actually prevent us from doing harm. Many of those standards had been developed by healthcare professionals and we are social scientists. The kinds of harm we do in (almost inevitable) misrepresentation of others can be huge. I am not saying that someone outside a community isn’t capable of researching it or making valuable observations or claims. I am not saying that insider views are not partial; they are. I am saying, however, that we need to question the authority of non-participatory research when the voices of the people being researched has no equal power (and often it does not).

Now moving onto the reviewers. Who reviews the reviewers?

First of all, I review articles for a lot of different journals and I review proposals for conferences. I find it a privilege and honor and learning opportunity. I love doing it because I learn a lot doing it, especially about what’s new in the field, and as someone who writes a lot, it’s important for me to take time to read deeply and critically and be able to support other people’s scholarship. Because I do think of peer review as a supportive thing we can do. I am really careful NOT to accept to review something I understand little about. Or at least to put a disclaimer to author and editors about parts that I feel unqualified to review. But I have seen some of the feedback to myself and to others that shows utter lack of knowledge about the subject. I have also seen reviews that lack basic courtesy. I assume a good editor would notice this and possibly not contact that reviewer in future. I wonder if they ever give that reviewer feedback? Ha, I wonder if reviewers get reputations for being fast or thoughtful or such?

I was also thinking that with conference reviews…some stuff proposed at conferences is really cutting edge – it’s entirely possible that a conservative academic rejects it because of that. And it would be a loss.

What bothers me most about reviews is that I often DON’T get to see the feedback of other peer reviewers. Wouldn’t it make me a better reviewer if I could see other people’s feedback on the same piece I wrote? This happens at Hybrid Pedagogy where the peer review process is open and dynamic, but not elsewhere.

Also, at conferences, I don’t usually get to know whether the things I reviewed got accepted. This bugs me. I know I will find out once the conference schedule comes out, but seriously, by then I have forgotten which ones I had reviewed and I don’t wanna wade through hundreds of presentations searching for the 5 or 10 I reviewed. Why is it difficult to BCC reviewers on decisions the way it’s done sometimes for journals? Especially that reviews of conferences are sometimes single-blind so I already know who the authors are.

Just like our written work gets reviewed and we comfort ourselves that it’s “good enough” once it’s published, our reviewing needs to be reviewed in some way – we need feedback on how appropriate our reviews are in certain contexts. Feedback from editors and authors alike. I have sought feedback of editors before when I haven’t been sure. Hybrid Pedagogy authors often give us positive feedback on the process of working with them (though I should seek the feedback of my co-reviewer, managing editor, and also actively ask authors if there are ways we can improve and not just take the praise they give us). For authors who aren’t in an open review process, what kind of power do they have to give feedback? I occasionally used to discuss things like this with editors and they were often understanding. But I doubt any of it ever got back to the reviewers.

What do you think?