We Should Never Stop BELIVing:

Reflections on 10 Years of Workshops on the “Esoteric” Art of Evaluating Information Visualizations

Enrico Bertini

BELIV Workshop - Oct 23, 2016

Wow, I can’t BELIV it’s been 10 years!

No! That was actually Venice!

How did we start and why? We wanted to go to Venice!

How BELIV was born ...

Oh … Next AVI will be in Venice! ... I need to find a way to go there ...

Oh … Let’s organize a workshop!

Enrico! ... Good news ... We organize a workshop!!!

Uh … ?!@#%$@ What’s a workshop???

… No worries I’ll invent the acronym, you’ll do the rest ...


Beyond time and errors: novel evaluation methods for Information Visualization.

Italians create weird acronyms

Beppe carefully reinforcing the stereotype that Italians always move their hands when they speak :)

I’ll never miss a single BELIV workshop!

In 2012 we finally managed to bring BELIV to IEEE VIS (a big plus was more inclusion of the SciVis “cousins” in the program)

How did BELIV impact Visualization?

Hard to tell, but ...

  • Impact: on me and my work ...

Great studies generate great questions!

Grounded Theory / Qualitative Research

Qualitative research can be done using a systematic/principled approach!

Systematic Exploration of Design Spaces

Comparing A to B is not enough … there’s a whole design space!

2) Impact: putting some order and providing guidance for researchers and practitioners ...

3) Impact: highly influential papers ...

What have we achieved during these 10 years?

Steady Growth / Evaluation Is Expected

Tobias Isenberg, Petra Isenberg, Jian Chen, Michael Sedlmair, and Torsten Möller. A Systematic Review on the Practice of Evaluating Visualization. IEEE Transactions on Visualization and Computer Graphics, 19(12):2818–2827, December 2013.

User Experience + User Performance

Visual Data Analysis + Environments and Work Practices

And slowly moving to a situation in which it’s not just done “because you have to”

Much Better Characterization of the Process

Munzner, Tamara. "A nested model for visualization design and validation." IEEE transactions on visualization and computer graphics 15.6 (2009): 921-928.

Sedlmair, Michael, Miriah Meyer, and Tamara Munzner. "Design study methodology: Reflections from the trenches and the stacks." IEEE Transactions on Visualization and Computer Graphics 18.12 (2012): 2431-2440.

Nested Model

Design Study Methodology

Much Better Use of Stats

Replication + Extension

Highly-Cited Solid Qualitative Research

Jessica’s paper + Narrative vis paper ...

Future directions?

Developing research that is useful and used.

How do we promote and accelerate this kind of process?

Going beyond studying visual representation.

Understanding data analysis and communication processes better.

Building more on top of each other’s work.

Time for a new research agenda?

More systematic probing/evaluation of the visualization design space.

What’s next for BELIV?

(Whole panel taking place on this topic later on today …)

Things I learned organizing BELIV ...

  • Throwing yourself into something bigger than you.
  • Working with people smarter than you.
  • Deeply caring about giving.
  • Not being afraid to push the boundaries.
  • Organizing events is a fantastic catalyzer.


Adam, Heidi, Petra, Tobias, Michael.


Authors felt they could talk and write about whatever came to their mind.

That ended up being so liberating and so much fun!

Also … many tried to put some order, and that was very useful for me.

Repeating every two years

Give priority to discussions …

Bring new people out / and stepping down

Not defining things too rigidly

Publishing in the ACM DL

Long discussions how to have an impact in the community …

Trying to go beyond the workshop … Post-workshop publications …

Printing high-quality proceedings

Challenges / Hurdles

Adoption (how do we make our research adopted? How does one design research in a way that maximizes its potential adoption?)

Beyond time and error? Models?

Understanding data analysis / beyond visual representation

Systematically probe the vis design space

More educational material

Evaluating interaction with automated models?

Cross-triangulation / ensembles methods

Gleicher’s gist paper / Steve Haroz list of evaluation papers!

Two good outcomes: suprising/unexpected outcomes that make people think or very practical guidelines.

Adoption / practical use. Email from practitioners. (what goes in the book?)

Understanding practitioners’ work/needs

Beyond time and error?

From A/B testing to models and theories

Beyond the desktop

Beyond (untested) dogmatism / beyond efficiency

More crazy ideas?

Beyond effectiveness of visual representation

Usability and use of generated knowledge / guidelines

Studying with representative samples (beyond AMT?) / WEIRD studies?

From hunting for differences to understanding whether these differences are meaningful/impactful

Papers born at BELIV … (probably more!)

Major Milestones and Recent Trends ...

Grounded theory / Qualitative type of work
Nested model
Eye-tracking / sensors?
Surveys and overviews (7 scenarios …) / books
Reproduced experiments
Better stats / CIs (!)
SciVis Evals
From perceptual to cognitive
Bold and controversial research
Confirmation of common knowledge in the wild (Borkin’s paper)

Keynote BELIV 2016 - Google Slides