Useful Resources
There is a useful summary of reviewing by Allen Lee at:
https://allenslee.com/referee.htm
In particular, I agree with the tips to give “actionable advice” (Action 3), to give general impressions, major comments, and then minor points (Action 5), and to be positive and constructive (Actions 5 and 9). Some of the other advice is more of the “take-it-or-leave-it” variety. It can be useful to put a summary at the beginning (Action 1), though I myself find that this can be largely uninformative, especially if it’s just a rehash of the abstract. For this class, you are not expected to do any additional research (Action 11).
Additional suggestions
As a reviewer, you are a gatekeeper. Your responsibility is to ensure that papers that get published are of adequate quality for the journal or granting agency. Unfortunately, in today’s competitive environment, reviewers must make tough decisions, and unequivocally, positive reviews for the sake of being nice may not be useful. As one presenter said in a grant-writing seminar, “Grants are like eggs… everyone’s a large egg, so you have to be an extra-large egg.” One criterion I use myself is to ask, “If I read this article in this journal would I be excited or disappointed? Or would I wonder, ‘Who was the idiot reviewer who let this through?’?”
In addition to the gatekeeper function, reviewers also have an opportunity to “raise the bar” on the quality of science that is done. Many of my papers have been vastly improved by suggestions from reviewers, and as a reviewer, I have seen numerous manuscripts that have gotten much better after a revision. For grants, the benefits of getting peer feedback on studies before they are started can be particularly valuable.
Be explicit in the problems and suggest how they can be solved. Don’t just write “The introduction was boring,” or “The study was poorly designed”; rather, state what the problem is and how it might be solved. (See also Action 3 of Lee’s paper).
Be aware of the constraints of the author. Perhaps the author didn’t put in a comprehensive introduction because the journal has strict word limits. Perhaps the grant writer didn’t have space in 5 pages to give secondary experimental details.
When citing references, give enough data that the authors can track the item down without too much headache. Don’t just cite, for example, “Smith et al., 2000.”
Not everyone will agree with me on this matter, but I try to give as much information to the authors as possible, and to not cloak too many hidden comments to the editor that the author doesn’t see. I do this because, when I am an author myself, I find it very frustrating to have seemingly good reviews and still get rejected (possibly because the reviewer gave more harsh comments/ratings to the editor than to me).
If you didn’t understand something, it’s often better to admit it. It is frustrating to get a review from a reviewer who didn’t get a concept. I will sometimes say things like, “I didn’t understand this, but I think they meant X, in which case…”
For graduate students and post-docs, I recommend co-reviewing a few articles with your research advisor during your training. In my own lab, I have found this to be symbiotic. I get a second opinion on a paper. My trainees learn how to review (and how to navigate the review process themselves).
I always write a draft of a review and then proofread and edit it the next day. While proofreading, I pay attention to my tone (constructive, not snotty) and my clarity.
Wherever possible, “cross-calibrate.” I really like journals where the editor makes all reviews and the outcome available to all reviewers. It is sometimes reassuring to see that other reviewers had the same concerns as you or surprising to see that they had a very different opinion.
Maintain confidentiality.