Mimi.jl��Building Software for Climate Change Research &�a Usability Analysis
Lisa Rennels
University of California, Berkeley
Spring 2022
Mimi.jl: An Integrated Assessment Modeling Framework�
Co-authored with Professor David Anthoff , Cora Kingdon, and Richard Plevin
2
2017 report by the US National Academies of Sciences, Engineering, and Medicine (NASEM):
current SC-CO2 estimates no longer reflect the latest research in these disciplines and provided a series of recommendations for improving the scientific basis, transparency, and uncertainty characterization of SC-CO2 estimates
3
Mimi.jl: SC-CO2 and the NASEM Report
computational platform (Mimi.jl)
Socioeconomics module
climate module
damage module
discounting module
uncertainty and sensitivity analysis
fragmented multitude of hosting platforms, programming languages, and APIs used by IAM authors makes the task of tackling analysis of one, let alone several, IAMs daunting for even experienced researchers. This setup also forces researchers and interested parties to write their own code to carry out crucial steps like uncertainty and sensitivity analysis, leading to an all-too-often cursory take, or outright omission, of such work
…
“an open source, computationally efficient, publicly accessible, and clearly documented computational platform” to remedy these problems (NASEM, 2017)
4
computational platform (Mimi.jl)
Socioeconomics module
climate module
damage module
discounting module
uncertainty and sensitivity analysis
Mimi.jl: SC-CO2 and the NASEM Report
5
We needed a platform that … | Which lead to … |
Is computationally fast | macro based embedded domain specific language |
is simple enough for novice programmers & works with the pre-existing ecosystem of models | a readable, easy-to-use API that leverages the full power of Julia including significant employment of multiple dispatch |
enables a modular work style for distributed, loosely coordinated teams | use of a custom Julia registry |
creates a transparent framework enabling easy replication of computational experiments | leveraging of packages and project environments |
Mimi.jl: Design and Impact
6
Mimi.jl: Design and Impact
7
Mimi.jl: Modular Design
8
Mimi.jl: Visualization
9
Mimi.jl: Uncertainty and Sensitivity Analysis
��Contextual Inquiry into Programmer’s Use of Mimi: An Embedded Domain Specific Language�
Co-authored with Professor Sarah Chasins
10
11
Usability Analysis: Context
We needed a platform that … | Which lead to … |
Is computationally fast | macro based embedded domain specific language |
is simple enough for novice programmers & works with the pre-existing ecosystem of models | a readable, easy-to-use API that leverages the full power of Julia including significant employment of multiple dispatch |
enables a modular work style for distributed, loosely coordinated teams | use of a custom Julia registry |
creates a transparent framework enabling easy replication of computational experiments | leveraging of packages and project environments |
Contextual Inquiry
“The core premise of Contextual Inquiry is very simple: go where the programmer works, observe the programmer as he or she works, and talk to the programmer about the work.” (Beyer and Holtzblatt, 1999)
12
Usability Analysis: Data and Methods
Contextual Inquiry
“… feels like coding therapy where I'm being very transparent about what I’m doing and the shameful parts of everything” (Participant 2)
13
Usability Analysis: Data and Methods
Thematic Analysis (Inductive, Semantic)
“Thematic analysis is a method for identifying, analyzing and reporting patterns(themes) within data. It minimally organizes and describes your data set in (rich) detail.” (Braun and Clarke, 2006)
Open Coding
“Open coding is the interpretive process by which data are broken down analytically. Its purpose is to give the analyst new insights by breaking through standard ways of thinking about or interpretaing phenomena reflected in the data.” (Corbin and Strauss, 1990)
14
Usability Analysis: Data and Methods
15
298 individual codes grouped into an evolving hierarchy
Usability Analysis: Results
16
298 individual codes grouped into an evolving hierarchy
Abstractions: Composite Data Types and Macros
The use of abstractions, specifically in the form of custom, mutable composite data types and macros, provide beneficial simplicity for users in most cases. However, they face frustration, confusion, and significant time loss when such constructs are misused or throw low-level error messages referring to concepts of the host language.
Usability Analysis: Results
17
298 individual codes grouped into an evolving hierarchy
Personal Engagement with Community
The Mimi language was designed to promote collaboration of the research community, both on projects and on models themselves. This motivated the modular design and a well-defined, simple API for model modification and augmentation. Users highly value personal engagement with the Mimi community, both for (1) ease of collaboration and contribution and (2) receiving assistance. This key finding echoes those of Meyerovich and Rabkin (2020), who find that extrinsic social language features, including use in a group, team familiarity, and existing code to be among the top reasons for language adoption, even when compared directly to intrinsic language features.
Usability Analysis: Results
18
298 individual codes grouped into an evolving hierarchy
Usability Analysis: Results
Mapping to a Mental Model
One clearly successful aspect of the Mimi eDSL is closeness of representation to domain. The ability to carefully match domain concepts to their representations in language constructs is a key advantage of DSLs.
19
298 individual codes grouped into an evolving hierarchy
Usability Analysis: Results
Host Language is Highly Consequential for the User
One of the most dominant themes that came out of our work was the multitude of ways that the choice of host language (Julia) effected not only the work of the embedded DSL developers, but the usability of the language from the perspective of the user themself. In one telling quote, P9 stated that "my frustration with Julia and my frustration with Mimi run together a lot". This user described finding error messages with high levels of detail about the type-system to be confusing.