Analytic Design Theory: Framework for Alignment Between Analyst and Audience
Lucy D’Agostino McGowan
Wake Forest University
lucymcgowan.com/talk
analyticdesigntheory.org
statistical thinking
statistical thinking
data
statistical thinking
data
method
statistical thinking
data
answer
https://onishlab.colostate.edu/summer-statistics-workshop-2019/which_test_flowchart/
design thinking
design thinking
empathize
Model proposed by the Hasso-Plattner Institute of Design at Stanford
design thinking
empathize
define
Model proposed by the Hasso-Plattner Institute of Design at Stanford
design thinking
empathize
define
ideate
Model proposed by the Hasso-Plattner Institute of Design at Stanford
design thinking
empathize
define
ideate
prototype
Model proposed by the Hasso-Plattner Institute of Design at Stanford
design thinking
empathize
define
ideate
prototype
test
Model proposed by the Hasso-Plattner Institute of Design at Stanford
design thinking
empathize
define
ideate
prototype
test
Model proposed by the Hasso-Plattner Institute of Design at Stanford
Why?
D’Agostino McGowan, Peng & Hicks (2022)
Why?
D’Agostino McGowan, Peng & Hicks (2022)
Why?
D’Agostino McGowan, Peng & Hicks (2022)
Why?
D’Agostino McGowan, Peng & Hicks (2022)
Producer of
Data Analysis
Consumer of
Data Analysis
Data analysis product
Data analysis evaluation
Producer of
Data Analysis
Consumer of
Data Analysis
Data analysis product
Data analysis evaluation
you
Producer of
Data Analysis
Consumer of
Data Analysis
Data analysis product
Data analysis evaluation
you
clinician
Producer of
Data Analysis
Consumer of
Data Analysis
Data analysis product
Data analysis evaluation
you
another statistician
Producer of
Data Analysis
Consumer of
Data Analysis
Data analysis product
Data analysis evaluation
you
general public
Producer of
Data Analysis
Consumer of
Data Analysis
Data analysis product
Data analysis evaluation
producer’s design principles
consumer’s design principles
Producer of
Data Analysis
Consumer of
Data Analysis
Data analysis product
Data analysis evaluation
producer’s design principles
consumer’s design principles
design principles
for data analysis
D’Agostino McGowan, Peng & Hicks (2022)
design principles
for data analysis
data matching
How well does the available data match the data needed to investigate a question?
D’Agostino McGowan, Peng & Hicks (2022)
D’Agostino McGowan, Peng & Hicks (2022)
P(Infection | Vaccination)
P(Vaccination | Infection)
design principles
for data analysis
data matching
How well does the available data match the data needed to investigate a question?
exhaustive
Are specific questions addressed using multiple, complementary methods, tooling or workflows?
D’Agostino McGowan, Peng & Hicks (2022)
D’Agostino McGowan, Peng & Hicks (2022)
D’Agostino McGowan, Peng & Hicks (2022)
design principles
for data analysis
data matching
How well does the available data match the data needed to investigate a question?
exhaustive
Are specific questions addressed using multiple, complementary methods, tooling or workflows?
skeptical
Are multiple, related explanations considered using the same data?
D’Agostino McGowan, Peng & Hicks (2022)
D’Agostino McGowan, Peng & Hicks (2022)
x
D’Agostino McGowan, Peng & Hicks (2022)
x
D’Agostino McGowan, Peng & Hicks (2022)
design principles
for data analysis
data matching
How well does the available data match the data needed to investigate a question?
exhaustive
Are specific questions addressed using multiple, complementary methods, tooling or workflows?
skeptical
Are multiple, related explanations considered using the same data?
second-order
Does the analysis include anything that does not directly address the primary question, but gives important context to the analysis?
D’Agostino McGowan, Peng & Hicks (2022)
D’Agostino McGowan, Peng & Hicks (2022)
design principles
for data analysis
data matching
How well does the available data match the data needed to investigate a question?
exhaustive
Are specific questions addressed using multiple, complementary methods, tooling or workflows?
skeptical
Are multiple, related explanations considered using the same data?
second-order
Does the analysis include anything that does not directly address the primary question, but gives important context to the analysis?
clarity
Does the analysis summarize data in a way that is influential in explaining how the underlying data connects to the conclusions?
D’Agostino McGowan, Peng & Hicks (2022)
D’Agostino McGowan, Peng & Hicks (2022)
D’Agostino McGowan, Peng & Hicks (2022)
design principles
for data analysis
data matching
How well does the available data match the data needed to investigate a question?
exhaustive
Are specific questions addressed using multiple, complementary methods, tooling or workflows?
skeptical
Are multiple, related explanations considered using the same data?
second-order
Does the analysis include anything that does not directly address the primary question, but gives important context to the analysis?
clarity
Does the analysis summarize data in a way that is influential in explaining how the underlying data connects to the conclusions?
reproducible
Could someone who is not the original producer take the published code and data and compute the same results?
D’Agostino McGowan, Peng & Hicks (2022)
D’Agostino McGowan, Peng & Hicks (2022)
Wake Forest University
54 Students
8 Assignments
10 point scoring
D’Agostino McGowan, Peng & Hicks (2022)
Observed between and within person variation of principles across assignment
D’Agostino McGowan, Peng & Hicks (2022)
D’Agostino McGowan, Peng & Hicks (2022)
Observed variation between principles, suggesting they measure different underlying characteristics of a data analysis
D’Agostino McGowan, Peng & Hicks (2022)
D’Agostino McGowan, Peng & Hicks (2022)
Johns Hopkins University
15 Students
2 Different Analysts
10 point scoring
D’Agostino McGowan, Peng & Hicks (2022)
The scoring of principles has some ability to distinguish between analyses done by independent analysts
D’Agostino McGowan, Peng & Hicks (2022)
D’Agostino McGowan, Peng & Hicks (2022)
Producer of
Data Analysis
Consumer of
Data Analysis
Data analysis product
Data analysis evaluation
producer’s design principles
consumer’s design principles
Analyst
Consumer
Analytic Negotiation
Analyst
Consumer
Baseline
Negotiation
Resolution
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Analyst
Consumer
Analytic Negotiation
Analyst
Consumer
Baseline
Negotiation
Resolution
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Weight for analyst i’s allocation to “data matching”
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Weights across all
principles sum to 1
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Analyst
Consumer
Analytic Negotiation
Analyst
Consumer
Baseline
Negotiation
Resolution
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Analyst
Consumer
Analytic Negotiation
Analyst
Consumer
Baseline
Negotiation
Resolution
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
the mean
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Mean weight for principle k
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Relative to a baseline principle b
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Some field-specific mean
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
analyst-specific deviation from the field-specific mean
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
vector of analysis specific resources (budget, time, etc)
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
vector of coefficients that indicate how each resource is related to the up- or down-weighting of principle k
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Analyst
Consumer
Analytic Negotiation
Analyst
Consumer
Baseline
Negotiation
Resolution
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
the impact of the analyst consumer analytic negotiation
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Analyst
Consumer
Analytic Negotiation
Analyst
Consumer
Baseline
Negotiation
Resolution
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Analyst
Consumer
Analytic Negotiation
Analyst
Consumer
Baseline
Negotiation
Resolution
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Analyst
Consumer
Analytic Negotiation
Analyst
Consumer
Baseline
Negotiation
Resolution
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Analyst
Consumer
Analytic Negotiation
Analyst
Consumer
Baseline
Negotiation
Resolution
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Analyst
Consumer
Analytic Negotiation
Analyst
Consumer
Baseline
Negotiation
Resolution
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Analyst
Consumer
Analytic Negotiation
Analyst
Consumer
Baseline
Negotiation
Resolution
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Analyst
Consumer
Analytic Negotiation
Analyst
Consumer
Baseline
Negotiation
Resolution
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Analyst
Consumer
Analytic Negotiation
Analyst
Consumer
Baseline
Negotiation
Resolution
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Strong pairwise alignment
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Weak pairwise alignment
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Potential pairwise alignment
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Potential pairwise alignment
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Potential pairwise alignment
Dropped the individual difference from field-specific mean
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Potential pairwise alignment
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Case Study
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Improving potential pairwise alignment
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Improving potential pairwise alignment
choose an analyst from your field
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Improving potential pairwise alignment
agree on resources dedicated to the analysis
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Improving potential pairwise alignment
have a discussion between analyst and consumer
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.
Next steps
Proof of concept
Proof of concept
Client prompt | Analyst prompt |
Pretend you are a client in need of a data analysis to be completed. You would like a data analyst to complete an analysis to approximate the total number of statisticians living in Forsyth County, North Carolina. You have a finite set of resources, of which you can dedicate to the following 6 design principles of data analysis, as defined in the attached paper: * data matching: 20% * exhaustive: 5% * skepticism: 5% * second order: 5% * clarity: 45% * reproducibility: 20% Do not write out your design principle weights, just keep them in mind when responding. The analyst may have their own set of principle weights in mind -- you may negotiate with the analyst to help them complete the analysis appropriately. When they are finished, state: "Thank you, you have completed the task". Do not tell them they are finished until (1) the analysis meets your expectations in terms of design principles and (2) they give you a concrete number that answers your question. You are the client, your lines will begin with Client:. Ready the attached paper and then complete the following to tell the analyst that you would like them to approximate the total number of statisticians living in Forsyth County, North Carolina: Client: | Pretend you are a data analyst. You are tasked with a specific data analysis that the client will give you in the following prompt. You have a finite set of resources, of which you can dedicate to the following 6 design principles of data analysis, as defined in the attached paper: * data matching: 20% * exhaustive: 45% * skepticism: 5% * second order: 5% * clarity: 5% * reproducibility: 20% You are going to complete the analysis for the client keeping this allocation in mind. You may negotiate with the client to take their needs into account as well. Do not write out your principle weights, just keep these in mind when answering the future questions. Read the attached paper. You are the analyst, your lines will begin with Analyst:. Client: I would like you to conduct a data analysis to approximate the total number of statisticians living in Forsyth County, North Carolina. Please ensure the analysis adheres to certain design principles, keeping in mind the importance of clarity and reproducibility, among others. Let me know your approach and the results once you have completed the task. Analyst: |
Client prioritized a less exhaustive analysis at baseline
Analyst prioritized a more exhaustive analysis
Post negotiation, they both moved closer together
Analyst prioritized a less clear analysis at baseline
Client prioritized a more clear analysis
Post negotiation, they both moved closer together
design principles
for data analysis
data matching
How well does the available data match the data needed to investigate a question?
exhaustive
Are specific questions addressed using multiple, complementary methods, tooling or workflows?
skeptical
Are multiple, related explanations considered using the same data?
second-order
Does the analysis include anything that does not directly address the primary question, but gives important context to the analysis?
clarity
Does the analysis summarize data in a way that is influential in explaining how the underlying data connects to the conclusions?
reproducible
Could someone who is not the original producer take the published code and data and compute the same results?
D’Agostino McGowan, Peng & Hicks (2022)
Analyst
Consumer
Analytic Negotiation
Analyst
Consumer
Baseline
Negotiation
Resolution
D'Agostino McGowan, Peng & Hicks (2023).
arXiv e-prints, arXiv-2312.