When indexing services review the quality of a journal, we learned the hard way that they look at who is on the board of reviewers. Therefore, from time to time, the Board of Governors will send you suggestions on promoting, training, or removing members of your board. There are three main categories:

  1. Non-performing reviewers and editors. 
    Some individuals on your board will consistently display poor performance. These reviewers (or editors) need to be mentored by you. You should initially add them to the watchlist and, if necessary, prune them from your review board. You can sort your reviewers’ ratings on the Reviewers tab (click “Order by Score”). Click the X box to the right of the reviewer’s name to remove the reviewer from your board. You (or, if you prefer, the Managing Editor) should email such reviewers or editors a “Goodbye” letter (template available in Your Notepad) with an invitation to rejoin the review board when they have time to be actively engaged.

It is important to prune away non-performing reviewers for two reasons. First, the editors of papers are quite unhappy when they are left with only one or two quality reviews to use when creating their development letter. Second, including on the review board colleagues known to be low-quality reviewers diminishes the reputation of your journal.

You can use the reviewer data that the Board of Governors sends you to prune your review board of non-performing reviewers. You can use your own dashboard’s data on reviewers to examine the performance of your board members. For example, you could look at the number of times a reviewer didn’t respond using the link at https://www.informingscience.org/Member/Reviewers?Tab=All&SubTab=Reviewers&UnavailableTab=&BoardRoleFilter=&Search=&OrderBy=TotalNonResponsive 

  1. Reviewers and editors who need more guidance from you 
    Email a copy of the Reviewer Manual to these reviewers, and the Editor Manual to these editors, with a letter expressing your wish for more helpful and mentoring reviews (or Decision Letters) in the future. You are their mentor, and they need your guidance.
  2. Excellent reviewers and editors 
    Send excellent reviewers a thank you letter. Good work deserves to be acknowledged. You may wish to acknowledge consistent good work as a Reviewer with a promotion to Senior Reviewer (an honorary advancement).

If you can use help as EiC, offer a promotion to Associate Editor-in-Chief. This job is not for every excellent editor since it requires a higher level of commitment and additional skills in mentoring editors.

For your letters, you need not re-invent the wheel. The Board of Governors provides you with Best Practice templates for each of these categories. You will need to copy them to your email and personalize your emails. Pay attention that you will need to fill in the placeholders manually with actual names and such.

Figure 23. Best Practices letters for mentoring and promoting reviewers and editors

You can also monitor reviewers and editors by clicking on their name in the Editorial Board menu. There you will find the data shown in the reviewers' tab.  This will assist you in acknowledging and promoting the excellent reviewer, in training reviewers who need training, and in saying goodbye to those reviewers who are not contributing. We show you not just wording (Outstanding, Superior, Average, More Training Needed, and Unhelpful); we also show average rating numerically. The actual number allows you to note, for example, that Superior (3.50/5) is not that different from Average (3.33/5).

The EiC is able to view statistics regarding editors’ past performance of accepting/rejecting, how reviewers’ acceptance differs from the average, and A/EiC tardiness. The reviewer scores in the Details view and in the Dashboard are normalized to a percentage, so the average scoring deviation is expressed in points out of 100 (Figure 27).



Figure 24. Checking data and promoting reviewers/editors.
The reviewer above writes excellent reviews (average rating by the editors of 4.13/5)
 and rates papers very similar (less than 1% higher) than the average of each paper’s other reviewers.

You can see what editors wrote to their reviewers (Figure 28). This allows you to mentor them if need be, and this will help you improve your ratings of editors. You should review both the ratings and letters to confirm that your editors are following ISI policy. Typically, the reviewers’ rating should be between 3 and 4, and the feedback to individual reviewers should be customized to that reviewer’s performance. These comments to reviewers by editors are stored for future viewing in the reviewer’s profile (Figure 29).

Figure 25. Check editors’ comments to their reviewers



Figure 26. Comments to reviewers are stored in the system

Figure 30 is an example of really bad feedback by an editor.

Figure 27. Example of poor feedback to a reviewer

Here are some suggested words for editors to use in feedback to a good reviewer: 

  • I appreciate and thank you for the positive way you worded feedback for the author on how to improve the paper. Your phrasing builds up the author through supportive and constructive language. This is the way I would like suggestions on my papers to be worded. Great job!
  • Your review provided the author with concrete ideas on how the submission can be made even stronger.
  • It is obvious that you shared generously with your valuable time and considerable skills to help your colleague improve the paper.
  • Your insights provide the author with ideas for new ways to take the paper.
  • Thank you for assisting the author by suggesting specific additional references to strengthen the paper.
  • Your suggestions on organizing the paper are great.
  • I appreciate your using line numbers to show the author where to make improvements.

And some suggested words for editors to use in feedback to a reviewer who needs mentoring:

  • You should write your review as if you were writing directly to the author and not to me. 
  • Aim to phrase your comments in a way to help and build up the author through supportive and constructive language. Phrase your review to criticize the paper, not the author, perhaps using phrases like “Your submission will be even stronger by …”.
  • We understand that your time is valuable; we ask that you be even more generous with your time and considerable skills to help your colleague improve the paper. We ask for well-formed paragraphs of suggestions, so please use complete sentences to share your ideas with the author.
  • Your future reviews will be more useful to the author and me by including concrete ideas on how the author can make the submission even stronger.
  • The role of the reviewer for this journal is that of the coach, not the referee. Share your thoughts on how to improve the submission if the submission has problems.

The system lists the Editor’s rating as editors as well as their ratings of a reviewer (Figure 31).


Figure 28. The system stores your ratings of each editors’ performance

More about promoting excellent reviewers to editors. The process is first to offer promotion since not every excellent reviewer wants to become an editor. Serving as editor is not suited for every excellent reviewer since the job requires a higher level of commitment and additional skills in writing mentoring letters.

Adding new reviewers to your Board. The system also eases the burden of signing up new reviewers.  The system doesn’t decide whom to accept or not for your board, only you do, but it helps speed you through the process.

As time goes by, you will lose reviewers. Some you will promote from reviewer to editor, some will retire from academia, and some you will let go for poor performance. The system allows journals to accept (or not) new requests to join the review board. By default, ISI journals do allow new “requests to join the review board,” but you can turn off listing your journal as accepting new reviewers.

Your best reviewers will come from authors of papers you publish. They already have some familiarity with the ISI Paper Review System, and you know that they have published at least one paper suitable for your journal. The publisher invites them to join your board in her acceptance letter, but you can also follow up with your personal request to join your journal’s board.

When someone requests to join your review board, the system requires them to fill out their profile, upload their CV, and select topics on which they have interest and expertise. The system then notifies you that there is a reviewer request pending. Click on the link in the email to go directly to the request. You can also see outstanding reviewer requests by clicking on the Reviewers tab on the left menu (from your dashboard) and then the Pending option on the right. The request link will have a copy of the applicant’s CV as well as other useful information, such as whether the applicant is a reviewer for other ISI journals or conferences.

You decide whether to accept the applicant into your board. Read the CV to assess the applicant’s suitability for your journal. Read any notes about this individual, particularly if they are on the Watchlist. If they have any statistics in the system as a past reviewer, see how prior editors rated their performance. Also, see if the applicant’s self-selected topics fit within your journal’s scope and if the applicant has scholarly publications. The system makes it easier to evaluate the fit of the applicant to your journal by showing you the number of topics on the reviewer’s list of topics of interest and expertise that match to your journal.  You want to select people for your review board who have published in your field and who have selected a large number of topics matching your journal’s topics.

Use the system to select your decision, “Accept” or “Reject,” and fill in the comments for the applicant. We have placed some templates in your Notepad that you can insert and modify to fit the situation. The templates are: (i) reject as not a match; (ii) reject for lack of publications; or (iii) accept (welcome). You can preview the letter to ensure the correct information has been inserted in the placeholders. Note, in particular, the signature, as the default is to insert the current user-c344, who may be the AEiC on behalf of the EiC.

We recommend that you keep track of new reviewers for the first few papers with an eye toward mentoring them. When you are satisfied with the new reviewer’s performance, promote them to regular reviewers.  Everyone enjoys being promoted.