Reviewing Motivations in the HCI Community

A while ago Kelly Booth, Gary Hsieh, and I ran a survey to find out what motivates reviewers to review papers in the HCI community. This is a very brief summary.

307 of CHI 2011 reviewers participated (106 females).

part

Encouraging high quality research, giving back to the research community, and finding out about new research were the top motivations for reviewing.

Relevance of the submission to a reviewer’s research and relevance to the reviewer’s expertise were the strongest motivations for accepting a request to review, closely followed by a number of social factors.

Gender and reviewing experience significantly affected some reviewing motivations, such as the desire for learning and preparing for higher reviewing roles.

Based on our survey, we identified 10 motivational factors:

  1. Learning
    • “Reviewing encourages one to be critical which feeds back into one’s own work.”
    • “I certainly have gained perspective by ‘seeing how the sausage is made’.”
  2. Reputation
    • “The person collecting reviews may notice my high quality review.”
    • “I hope to be an AC at some point”
  3. Quality control and influence:
    • “I see the role of reviewers as helping to shape the field.”
    • “I do not want students to get the wrong impressions about human factors in computer systems or about the profession’s standards for acceptable research practices.”
  4. Prestige/significance of the review
    • “It increases my motivation since I know that the paper will [be] read [by] a wider audience if accepted.”
    • “When I know my review will be read by someone I greatly respect and they will know that I wrote it, that is a strong motivator.”
  5. Social Obligation
    • “In the economy of reviewing, it means I may be able to armtwist that researcher to review a paper for me in the future.”
    • “Turning down review requests can also get you branded,”
  6. Scientific ability and match with research interests
    • “It also greatly increases my guilt to say no to a review request when I know I am one of the most appropriate people who could have been asked.”
    • “Close relation of the paper and my ‘future or ongoing’ research will increase my motivation”
  7. Convenience
    • “If it’s too long relative to papers submitted for that venue, I tend to assume it’s poorly written or premature. I don’t like authors using reviewers as editors (or collaborators!).”
    • “Certainly, reviewing is one of the first things to be dropped when pressed for time.”
  8. Content benefit
    • “I feel that regularly reviewing keeps one’s critical skills sharp.”
    • “Being a program committee member gives an even better vantage point to see what’s new in the field.”
  9. Recognition of contribution
    • “When my reviews are ignored in the final decision, I think twice about reviewing for that conference”
  10. Improving papers
    • “The authors took my comments seriously, responded appropriately, and the revised version was excellent, it made me feel useful.”
  11. Other social motivations
    • “Because some reviews I’ve got are very helpful and improve my work, and I want to provide good thoughtful reviews back.”
    • “Reviewing helps me feel connected to the research community.
    • “It can be exciting to get a first look at what may be very good research, based on someone’s reputation, and to have the opportunity to contribute meaningful feedback to them.”

We asked participants to rank their top three reasons for reviewing from a list of potential reasons. The bars are sorted based on how frequently a reason was considered to be the top reason for reviewing (length of the blue bars).

If we look at the participants’ top reason (the blue bar) we can see that there’s a high diversity in people’s primary reason for reviewing.

fig

Awareness of new research which is the highest here, is the primary reason for less than 14% of the participants, and preparing for higher reviewing roles which is one of the least important reasons, is the primary reason for 7.5%.

It suggests that we really need to consider many or all of the potential motivations in designing peer review processes and support systems, to be able to cater to a large portion of researchers. In the paper we discuss various possibilities for designing for each motivation, some technology-dependent and some not. Here’s the full paper.

 

+ Nobarany, S., Booth, K. S. and Hsieh, G. (2016), What motivates people to review articles? The case of the human-computer interaction community. J Assn Inf Sci Tec (JASIST), 67: 1358–1371. doi:10.1002/asi.23469