Experiments and comparison tests
Controlled experiments and comparison tests are similar in design and practice. We will discuss their design together, but first we review the differences between the two:
Controlled experiments
- Evaluate a scientific theory
- Test a hypothesis that comes from the scientific theory
- Generate results that apply to a wide range of situations (generalizable results)
Comparison tests
- Compare two products that perform the same tasks
- Indicate which product is better for some situations
- Do not normally generate results that apply to other products and situations
Controlled experiments need to create results from at least two conditions. Comparison tests generate results from at least two products. For both kinds of studies, the practitioner designs the study to ensure that the situations (e.g. tasks, environment, participants, instructions) are the same (or at least comparable) among the different conditions or products.
In contrast, the observational study does not have controlled conditions. For this kind of study, the practitioner selects situations or products, observes their usage and compares their results.
Issues for discussion
- Examples of each
- Shortcomings of observational studies