Validation proto-pattern

Software engineers offer several kinds of evidence in support of their research results. It is essential to select a form of validation that is appropriate for the type of research result and the method used to obtain the result. As an obvious example, a formal model should be supported by rigorous derivation and proof, not by one or two simple examples. On the other hand, a simple example derived from a practical system may play a major role in validating a new type of development method.

Does your result fully satisfy your claims? Are the definitions precise, and are terms used consistently?

Authors tend to have trouble in some specific situations. Here are some examples, with advice for staying out of trouble:

  • If your result ought to work on large systems, explain why you believe it scales.
  • If you claim your method is “automatic”, using it should not require human intervention. If it’s automatic when it’s operating but requires manual assistance to configure, say so. If it’s automatic except for certain cases, say so, and say how often the exceptions occur.
  • If you claim your result is “distributed”, it probably should not have a single central controller or server. If it does, explain what part of it is distributed and what part is not.
  • If you’re proposing a new notation for an old problem, explain why your notation is clearly superior to the old one.
  • If your paper is an “experience report”, relating the use of a previously-reported tool or technique in a practical software project, be sure that you explain what idea the reader can take away from the paper to use in other settings. If that idea is increased confidence in the tool or technique, show how your experience should increase the reader’s confidence for applications beyond the example of the paper.

The table below lists the types of research validation that are used in software engineering research papers and provides specific examples. In this table, the examples are keyed to the type of result they apply to.

Type of validation Examples
Analysis I have analyzed my result and find it satisfactory through rigorous analysis, e.g. …
For a formal model                             …
rigorous derivation and proof
For an empirical model                      …
data on use in controlled situation
For a controlled experiment              …
carefully designed experiment with
statistically significant results
Evaluation Given the stated criteria, my result…
For a descriptive model                      …
adequately describes phenomena of
interest …
For a qualitative model                      …
accounts for phenomena of interest…
For an empirical model                      …
is able to predict … because …,
or …                                                      generates results that fit actual data …Includes feasibility studies, pilot projects
Experience My result has been used on real examples by someone other than me, and the evidence of its correctness/usefulness/
effectiveness is …
For a qualitative model                      …
narrative
For an empirical model or tool         …
data, usually statistical, on practice
For a notation or technique               …
comparison of systems in actual use
Example Here’s an example of how it works on
For a technique or procedure            …
a “slice of life” example based on a
real system …
For a technique or procedure            …a system that I have been developing …
For a technique or procedure            … a toy example, perhaps motivated by realityThe “slice of life” example is most likely to be convincing, especially if accompanied by an explanation of why the simplified example retains the essence of the problem being solved. Toy or textbook examples often fail to provide persuasive validation, (except for standard examples used as model problems by the field).
Persuasion I thought hard about this, and I believe passionately that …
For a technique                                    … if you do it the following way, then …
For a system                                         … a system constructed like this would …
For a model                                          … this example shows how my idea worksValidation purely by persuasion is rarely sufficient for a research paper. Note, though, that if the original question was about feasibility, a working system, even without analysis, can suffice
Blatant assertion No serious attempt to evaluate result. This is highly unlikely to be acceptable

The program committee looks for solid evidence to support your result. It’s not enough that your idea works for you, there must also be evidence that the idea or the technique will help someone else as well.

Analysis, actual experience in the field, and good use of realistic examples tend to be the most effective ways of showing why your result should be believed. Careful narrative, qualitative analysis can also work if the reasoning is sound.

Guidance on specific types of validation

Is the paper argued persuasively? What evidence is presented to support the claim? What kind of evidence is offered? Does it meet the usual standard of the subdiscipline?

Is the kind of evaluation you’re doing described clearly and accurately? “Controlled experiment” requires more than data collection, and “case study” requires more than anecdotal discussion. Pilot stud­ies that lay the groundwork for controlled experi­ments are often not publishable by themselves.

Is the validation related to the claim? If you’re claiming performance improvement, validation should analyze performance, not ease of use or generality. And conversely.

Is this such an interesting, potentially powerful idea that it should get exposure despite a shortage of concrete evidence?

Authors tend to have trouble in some specific situations. Here are some examples, with advice for staying out of trouble:

  • If you claim to improve on prior art, compare your result objectively to the prior art.
  • If you used an analysis technique, follow the rules of that analysis technique. If the technique is not a common one in software engineering (e.g., meta-analysis, decision theory, user studies or other behavioral analyses), explain the technique and standards of proof, and be clear about your adherence to the technique.
  • If you offer practical experience as evidence for your result, establish the effect your research has. If at all possible, compare similar situations with and without your result.
  • If you performed a controlled experiment, explain the experimental design. What is the hypothesis? What is the treatment? What is being controlled? What data did you collect, and how did you analyze it? Are the results significant? What are the potentially confounding factors, and how are they handled? Do the conclusions follow rigorously from the experimental data?
  • If you performed an empirical study, explain what you measured, how you analyzed it, and what you concluded. What data did you collect, and how? How is the analysis related to the goal of supporting your claim about the result? Do not confuse correlation with causality.
  • If you use a small example for explaining the result, provide additional evidence of its practical use and scalability.

Leave a Reply