1 of 9

SPRING WEEK 2

How to do evaluation in research

Maya Cakmak

SPRING 2024

CSE 492 R: UNDERGRADUATE RESEARCH IN COMPUTER SCIENCE & ENGINEERING

2 of 9

Research, papers, evaluation

3 of 9

Evaluation is a key part of research

  • evaluation methods
  • findings, results, or arguments
  • significance or implications

4 of 9

Contribution types & evaluation

A “new” or “novel”:

  • Algorithm
  • Formulation, metric
  • Proof
  • “Approach”, “technique”, “method”
  • System
  • Framework
  • Empirical data (from experiments or user study)
  • Dataset, benchmark
  • Literature review, survey
  • Vision

5 of 9

Formative vs. summative evaluation

  • Formative: Learn more about the problem, figure out what the right questions are

​

  • Summative: Try to answer a clear question, make a conclusion

“Ideation” “exploration”

6 of 9

The goal of evaluation: Substantiate claims

Convince the reader (e.g., an expert reviewer) that:

  • Your new method/system works [within reasonable bounds]
    • It achieves its goals
    • It achieves satisfactory/desirable performance [Metrics]
    • It is generalizable across tasks, environments, users
  • Your new method/system is better than a baseline
    • Better in what ways? [Metrics]

​

…and explain why.

7 of 9

Metrics

Measurement of performance

8 of 9

Examples

9 of 9

Class exercise: Characterize/plan your evaluation

  • Is it formative/summative?
  • What are you evaluating?
  • Is it observational or comparative?
    • What is your baseline?
  • Is it quantitative or qualitative or mixed?
  • What is the context of evaluation? What are tasks for evaluation?
  • Does it involve “users” or “human subjects”?
  • What are your objective/subjective metrics?
  • Do you have a hypothesis?