Empirical study desiderata

Here are Jim Herbsleb’s desiderata for reading and evaluating an empirical paper, from a fall 2014 class assignment at Carnegie Mellon:

The literature review gives a clear picture of the state of research, and identifies a clear gap or next step, and shows why it is important to address it.

The research questions flow in a convincing way from the literature review, as more concrete and explicit expressions of the gap or next step.

The research method is appropriate to the current state of knowledge.  For example, if almost nothing is known, a wider net such as a qualitative, grounded theory approach is often appropriate. If much more is known, and specific, informed hypotheses flow from the theories or principles being investigated, a more focused quantitative approach is often appropriate.

The method is described in sufficient detail and with sufficient clarity that you can judge how well the method was applied. For a statistical model, for example, the choice of models should be explained, and there should be tests of the assumptions underlying the model (e.g., normally distributed variables, absence of problematic multicollinearity).

The results are described clearly, in sufficient detail, without extraneous and irrelevant findings.  Alternate explanations of the findings should be explored as appropriate, and weaknesses and ambiguities should be made as clear as possible rather than glossed over.

The findings and their implications should be described accurately and precisely without overgeneralizing or stating them with a certainty not warranted by the evidence.

The literature should be addressed again, showing how and to what extent the study fills the gap or takes the next step described in the introduction.

New research questions that arise from the work should be identified, and their significance explained.


“We were the first to do X.”  It doesn’t matter unless X needs to be done. The literature review should show this.  Authors claiming to be first to do X should arouse your suspicion.  This is not a substitute for saying that X is important.

‘We are interested in . . .”  Who cares. What does the field need to know?  The authors’ personal interests are irrelevant.

“In our future work, we plan to . . .”  Again, who cares?  Future work is not your personal plan, it is trying to help the reader understand the state of play, the new and important research questions, in the research area as perturbed by your new results.

“We show that always . . . by an empirical . . .”

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.