The language includes patterns at several levels: overall architecture of the scientific argument of a paper, essential components of the scientific argument, intermediate patterns that refine the essential components, and localized patterns for specific elements of a paper.
At present, these are mostly just sketches, or even just titles. Here’s the encoding:
- [[bracketed]] title: pure placeholder, just a hint at what might go there
- plain text title: narrative sketch, sometimes called proto-pattern when it gets some meat
- title ending in “pattern”: an actual draft of a pattern
Here is a canonical template for patterns in this language.
This is potentially a large undertaking. We state some principles to guide the development so that it remains tractable and useful.
Overall architecture of the scientific argument
These patterns capture the main thread of the scientific argument — the “plot” or “story line” of the paper — and typical overall paper organizations. [[At the moment, they are named for the places where they are common, but this list should be reorganized into a classification based on the type of result (e.g., analysis technique, system organization, generalization/theory) or the research method (e.g., empirical study, formal analysis, systematic literature review). This decision should be made early, as it may be hard to refactor. Thus a question to answer early on is which dimension is better at clustering similar papers.]]
Empirical studies are used to derive conclusions from information in the world. They may be quantitative or qualitative, and they may be used to frame hypotheses or to test them. The general empirical study pattern is supported by patterns for specific empirical methods, including both qualitative and quantitative methods; patterns for these techniques are collected in the “Intermediate Techniques” below.
Following Runeson et all (from Resources section), we identify four objectives of empirical studies [[it would also be useful to distinguish qualitative from quantitative methods]]:
- Exploratory—finding out what is happening when the domain of interest has not been fully characterized, seeking new insights and generating ideas and hypotheses for new research; see Exploratory study method pattern
- Descriptive—portraying a situation or phenomenon.
- Explanatory—seeking an explanation of a situation or a problem, mostly but not necessary in the form of a causal relationship.
- Improving—trying to improve a certain aspect of the studied phenomenon
Typical ICSE paper
The thread of the argument is: (a) I set out to do X; (b) I produced result Y (and perhaps a tool T); (c) I validated the result by Z
ESEC/FSE papers ca 2009
The thread of the argument is: (a) I set out to analyze X; (b) I created a technique Y and sometimes a tool T; (c) I validated the result by Z
In terms of the ICSE 2002 study, this is an analysis question, a procedure result, and an example or evaluation validation. Often a tool is a supporting result
EASE papers ca 2009
The thread of the argument is: (a) I wanted to understand P; (b) I designed experiment Q; (c) I learned R, with confidence S and caveats T
In terms of the ICSE 2002 study, this is an instance evaluation question, a specific solution result, and an analytic validation.
Essential components of the scientific argument
The problem or question
A number of these were discussed in the ICSE2002 study. This should include the specific claim about the contribution of the research. The claim is about the new knowledge established by the research; it is usually more abstract than the specific result of the research. There are two principal forms. In the “problem” form, the argument is “we set out to solve this problem; here’s the solution; here’s why it works.” In the “question” form, the argument is “We set out to find out more about this phenomenon; here’s what we learned and why we believe it”
A number of these were discussed in the ICSE2002 study
The validation argument
A number of these were discussed in the ICSE2002 study
The scientific argument must hang together. This includes patterns such as [[traceability]] from claim to conclusions and anti-patterns such as “claims that apply ‘always’ can rarely be validated with examples”
Intermediate patterns that refine the architecture and essential components
Types of empirical studies
The general class of empirical studies includes a variety of specific empirical methods. Each has its own applicability and its own rules. There’s source material for this in the resources section. From Filipe Correia we have a list of methods from the 2012 Workshop on User Evaluation for Software Engineering (USER2012) [[establish a clear connection to the quantitative and qualitative method patterns in the “Overall architecture” section]]:
- “Complete” methods
- Controlled Statistical Experiment
- Controlled Qualitative Study
- Case Study
- Cognitive Modeling
- Participatory Action Research
- Data collection methods
- Direct Observation
- Instrumentation, Automated Measurement
- Audio/Video Recording
- Data Generation Methods
- Interview (Semistructured or Structured)
- Contextual Inquiry
- Data Analysis Methods
- Statistical Evaluation
- Discourse Analysis (Content Analysis)
- Grounded Theory
including issues of overfitting
Localized patterns for specific elements
The paragraph at the end of the introduction that lays out the argument of the paper, including the dreadful rendition that simply echos the section titles.
Decide what audience you’re writing for, or what venue you intend the paper for. It will affect your choice of overall architecture, your decisions about how much explanation is needed, and more [[Is this a pattern, or is it generic advice?]]
Certainly, funding agencies. Often, colleagues who provided insights and significant feedback. In some research areas (notably medicine) there’s a section that identifies the specific contributions of each of the authors — but software engineering has not come to this.
Follow your publication venue’s style guide, including layouts for tables, graphs, figures, references, etc. This pattern language is not about document design, typesetting, bibliographic formats, and the like