PL + HCI Grand Tour!
CS294-184: Building User-Centered Programming Tools UC Berkeley 9/27/20 & 9/29/20
Templates for slides 1 - 3 of each mini presentation
one-sentence summary of the problem that the work tackles
Name
Paper (or Project) Title
Paper (or Project) Authors
Illustration of problem, if relevant
one-sentence summary of the solution the paper proposes
Paper (or Project) Title
Paper (or Project) Authors
Presenter Name
Illustration of solution, if relevant
Paper (or Project) Title
Paper (or Project) Authors
Presenter Name
Demo video
Insert your slides after here!
Day 1
“turn abstract statements written in familiar math-like notation into one or more possible visual representations”
Penrose
Ye et al. (CMU graphics crew)
Yifan
https://www.youtube.com/watch?v=OyD4LIv2PDc&ab_channel=KeenanCrane
“the visual representation is user-defined in a constraint-based specification language; diagrams are then generated automatically via constrained numerical optimization”
Penrose
Ye et al. (CMU graphics crew)
Yifan
https://www.youtube.com/watch?v=OyD4LIv2PDc&ab_channel=KeenanCrane
https://www.youtube.com/watch?v=O60RuV2gBMk&ab_channel=ACMSIGCHI
Penrose
Ye et al. (CMU graphics crew)
https://www.youtube.com/watch?v=OyD4LIv2PDc&ab_channel=KeenanCrane
One-liners can be… idiomatic, plus annoying to write code for
J.D. Zamfirescu-Pereira
Small Step Live Programming by Example (SnipPy)
Ferdowsifard, Ordookhanians, Peleg, Lerner, & Polikarpova
Illustration of problem, if relevant
Wants to return “A.A.K”
Build expressions programming-by-
example using a live-updating structured view of code
Small Step Live Programming by Example (SnipPy)
Ferdowsifard, Ordookhanians, Peleg, Lerner, & Polikarpova
J.D. Zamfirescu-Pereira
Background: Projection Boxes
Immediate view of program’s runtime state, for example:
I’d argue that this is the actual HCI contribution...though this paper does offer an interesting evaluation of a new use for Projection Boxes!
SnipPy Example
A video example can be found at https://youtu.be/VqIy4iuSpzI
SnipPy: Multiple demonstrations (examples)
SnipPy: System Diagram
SnipPy: Expression Grammar
* had to be a bit clever for string constants...
SnipPy: Evaluation
Method: standard battery of tasks; SnipPy vs. Projection Boxes conditions; measures of speed, correctness, use of synthesized code; final survey.
SnipPy: Findings
SnipPy: Findings
Help data scientists retrieve information from previous versions of Jupyter Notebooks
Doris Xin
Towards Effective Foraging by Data Scientists to Find Past Analysis Choices
Kery, M. et. al. (CHI ‘19)
An augmentation of JupyterLab that
*
Doris Xin
Towards Effective Foraging by Data Scientists to Find Past Analysis Choices
Kery, M. et. al. (CHI ‘19)
HCI
PL
Doris Xin
Towards Effective Foraging by Data Scientists to Find Past Analysis Choices
Kery, M. et. al. (CHI ‘19)
Doris Xin
Towards Effective Foraging by Data Scientists to Find Past Analysis Choices
Kery, M. et. al. (CHI ‘19)
Doris Xin
Towards Effective Foraging by Data Scientists to Find Past Analysis Choices
Kery, M. et. al. (CHI ‘19)
Conducted the user study at JupyterCon ‘18
20 cell notebook contained over 300 versions
AVG(tasks/person) = 6
Doris Xin
Towards Effective Foraging by Data Scientists to Find Past Analysis Choices
Kery, M. et. al. (CHI ‘19)
“Baseline”: For comparison, data scientists interviewed in [17] reported making many local copies of their notebook files. Imagine giving our participants over 300 files and asking them to answer a series of detailed questions about them. Many participants would have run out of time or given up.
When we create a layout: fix the relative sizes and positions of visual elements with constraints.
Qitian Liao
Programming by Manipulation for Layout
Thibaud Hottelier, Ras Bodik, Kimiko Ryokai
Programming by Manipulation:
Programming by Manipulation for Layout
Thibaud Hottelier, Ras Bodik, Kimiko Ryokai
Qitian Liao
Example Demo
HCI Demo
PL Demo
Programming by Manipulation for Layout
Thibaud Hottelier, Ras Bodik, Kimiko Ryokai
Qitian Liao
Evaluations:
Conclusions:
Programming by Manipulation for Layout
Thibaud Hottelier, Ras Bodik, Kimiko Ryokai
Qitian Liao
An enormous number of non-programmers use Excel, and bugs in their work can be costly, time-consuming, and dangerous in a myriad of domains.
Lisa Rennels
ExceLint: Automatically Finding Spreadsheet Formula Errors
Daniel W. Barowy, Emery D. Berger, and Benjamin Zorn
Produce a tool that uses static analysis to automatically find spreadsheet formula errors using a (a) visualization called ‘global view’ and (b) a proposed fixes tool called ‘guided audit’.
Lisa Rennels
ExceLint: Automatically Finding Spreadsheet Formula Errors
Daniel W. Barowy, Emery D. Berger, and Benjamin Zorn
HCI elements
PL elements
ExceLint: Automatically Finding Spreadsheet Formula Errors
Daniel W. Barowy, Emery D. Berger, and Benjamin Zorn
Lisa Rennels
“The evaluation of ExceLint focuses on answering the following research questions.
(1) Are spreadsheet layouts really rectangular?
(2) How does the proposed fix tool compare against a state-of-the-art pattern-based tool used as an error finder?
(3) Is ExceLint fast enough to use in practice?
(4) Does it find known errors in a professionally audited spreadsheet?”
ExceLint: Automatically Finding Spreadsheet Formula Errors
Daniel W. Barowy, Emery D. Berger, and Benjamin Zorn
Lisa Rennels
“The evaluation of ExceLint focuses on answering the following research questions.
(1) Are spreadsheet layouts really rectangular?
(2) How does the proposed fix tool compare against a state-of-the-art pattern-based tool used as an error finder?
(3) Is ExceLint fast enough to use in practice?
(4) Does it find known errors in a professionally audited spreadsheet?”
ExceLint: Automatically Finding Spreadsheet Formula Errors
Daniel W. Barowy, Emery D. Berger, and Benjamin Zorn
Lisa Rennels
Precision: TP / (TP + FP)
Recall: TP / (TP + FN)
“The evaluation of ExceLint focuses on answering the following research questions.
(1) Are spreadsheet layouts really rectangular?
(2) How does the proposed fix tool compare against a state-of-the-art pattern-based tool used as an error finder?
(3) Is ExceLint fast enough to use in practice?
(4) Does it find known errors in a professionally audited spreadsheet?”
ExceLint: Automatically Finding Spreadsheet Formula Errors
Daniel W. Barowy, Emery D. Berger, and Benjamin Zorn
Lisa Rennels
Precision: TP / (TP + FP)
Recall: TP / (TP + FN)
ExceLint: Automatically Finding Spreadsheet Formula Errors
Daniel W. Barowy, Emery D. Berger, and Benjamin Zorn
Lisa Rennels
ExceLint: Automatically Finding Spreadsheet Formula Errors
Daniel W. Barowy, Emery D. Berger, and Benjamin Zorn
Lisa Rennels
DEMO (if time):
great ~15 minute video from SIGPLAN:
https://dl.acm.org/doi/10.1145/3276518
How should compilers explain problems to developers?
Cristina Teodoropol
How Should Compilers Explain Problems to Developers?
Barik, Titus, et al.
Solution: Follow design principles!�
Cristina Teodoropol
How Should Compilers Explain Problems to Developers?
Barik, Titus, et al.
But first, some background
How Should Compilers Explain Problems to Developers?
Barik, Titus, et al.
Cristina Teodoropol
Problem – Research questions
How Should Compilers Explain Problems to Developers?
Barik, Titus, et al.
Cristina Teodoropol
Approach RQ1
Survey of professional developers asking preferences of specific OpenJDK vs Jikes compiler error messages
→ 5 pairs of error messages:�same problem, different argument structures
�E1 Deficient argument vs. simple argument
E2 Deficient argument vs. extended argument
E3 Claim-resolution vs. extended argument
E4 Different claim, same extended argument
E5 Same claim, same simple argument
How Should Compilers Explain Problems to Developers?
Barik, Titus, et al.
Results RQ1
E1 Deficient argument vs. simple argument�
E2 Deficient argument vs. extended argument�
E3 Claim-resolution vs. extended argument
Preferred a resolution�
E4 Different claim, same extended argument
Content influenced preference: natural language�
E5 Same claim, same simple argument
How Should Compilers Explain Problems to Developers?
Barik, Titus, et al.
Cristina Teodoropol
Approach/Results RQ2 How is the structure of explanations in Stack Overflow different from compiler error messages?
����������
1 Simpson, S. L., Lyday, R. G., Hayasaka, S., Marsh, A. P., & Laurienti, P. J. (2013). A permutation testing framework to compare groups of brain networks. Frontiers in computational neuroscience, 7, 171.
How Should Compilers Explain Problems to Developers?
Barik, Titus, et al.
Cristina Teodoropol
Results RQ2 Cont’d.
How Should Compilers Explain Problems to Developers?
Barik, Titus, et al.
Cristina Teodoropol
How does a compiler writer know what to optimize?
An empirical study of FORTRAN programs (1971)
Donald Knuth
Will Crichton
Quantitative analysis of source code and runtime information
An empirical study of FORTRAN programs (1971)
Donald Knuth
Will Crichton
“We also found that less than 4 percent of a program generally accounts for more than half of its running time. This [...] means that programmers can make substantial improvements in their own routines by being careful in just a few places; and optimizing compilers can be made to run much faster since they need not study the whole program with the same amount of concentration.”
An empirical study of FORTRAN programs (1971)
Donald Knuth
Will Crichton
“A first idea for obtaining ‘typical’ programs was to go to Stanford’s Computation Center and rummage in the waste-baskets and the recycling bins. This gave results but showed immediately what should have been obvious: waste-baskets usually receive undebugged programs.”
“Our next method of obtaining programs was to post a man by the card reader [...] the job was very time-consuming since it was necessary to ask embarrassing questions about the status of people’s programs.”
“Was this sample representative? Perhaps the users of Stanford’s computers are more sophisticated than the general programmers to be found elsewhere; after all we have such a splendid Computer Science Department! [...] But it was distressing to see what little impact our courses seem to be having, since virtually all of the programs we saw were apparently written by people who had learned programming elsewhere.”
Problem: How do we synthesize a SQL query given an input-output pair?
Synthesizing Highly Expressive SQL Queries from Input-Output Examples
Chenglong Wang, Alvin Cheung, Rastislav Bodik
Jerry Song
Solution: System Scythe uses abstract language for queries that makes it easier to synthesize SQL queries
Synthesizing Highly Expressive SQL Queries from Input-Output Examples
Chenglong Wang, Alvin Cheung, Rastislav Bodik
Jerry Song
Language Grammar: similar to SQL but filter predicates are replaced with holes
Synthesizing Highly Expressive SQL Queries from Input-Output Examples
Chenglong Wang, Alvin Cheung, Rastislav Bodik
Jerry Song
Abstract queries synthesized w/ enumerative approach
Synthesizing Highly Expressive SQL Queries from Input-Output Examples
Chenglong Wang, Alvin Cheung, Rastislav Bodik
Jerry Song
Predicate Synthesis uses bit vector encodings and grouping/pruning techniques to reduce search time
Synthesizing Highly Expressive SQL Queries from Input-Output Examples
Chenglong Wang, Alvin Cheung, Rastislav Bodik
Jerry Song
Evaluation: Comparing Scythe to Enum method of program synthesis on stack overflow issues
(50 unsolved cases)
Synthesizing Highly Expressive SQL Queries from Input-Output Examples
Chenglong Wang, Alvin Cheung, Rastislav Bodik
Jerry Song
Evaluation: Does not produce the simplest solution
Synthesizing Highly Expressive SQL Queries from Input-Output Examples
Chenglong Wang, Alvin Cheung, Rastislav Bodik
Jerry Song
HCI
PL
Synthesizing Highly Expressive SQL Queries from Input-Output Examples
Chenglong Wang, Alvin Cheung, Rastislav Bodik
Jerry Song
Why do some programming languages fail and others succeed?
Rolando Garcia
Socio-PLT: Principles for Programing Language Adoption
Leo Meyerovich and Ariel Rabkin
Rolando Garcia
Socio-PLT: Principles for Programing Language Adoption
Leo Meyerovich and Ariel Rabkin
Programmers often have misconceptions about what code actually does and waste time investigating false leads.
Griffin Prechter
Addressing Misconceptions About Code with Always-On Programming Visualizations
Tom Lieber, Joel Brandt, Robert C. Miller
ant
Theseus visualizes a program’s run-time state using code coloring and marginal notes, illuminating how code actually behaves.
Griffin Prechter
Addressing Misconceptions About Code with Always-On Programming Visualizations
Tom Lieber, Joel Brandt, Robert C. Miller
Illustration of solution, if relevant
Griffin Prechter
Addressing Misconceptions About Code with Always-On Programming Visualizations
Tom Lieber, Joel Brandt, Robert C. Miller
Programming Languages
HCI
Understand how written code actually works.
Implemented using instrumentation hooks on JavaScript source code.
Uses a trace-collecting module that is injected into programs being debugged.
Rather than requesting information explicitly, as with other debugging tools, information is always visible.
Number of times a function is called is visible along with syntax highlighting of code that is never called.
An event-oriented summary of program execution with easy navigation of source code.
Griffin Prechter
Addressing Misconceptions About Code with Always-On Programming Visualizations
Tom Lieber, Joel Brandt, Robert C. Miller
Griffin Prechter
Addressing Misconceptions About Code with Always-On Programming Visualizations
Tom Lieber, Joel Brandt, Robert C. Miller
Evaluation 1: Lab Study; 7 participants (all graduate students) were given 5 programming tasks, some to be performed using Theseus, and others with standard debugging tools.
Results: Inconclusive. Due to small number of participants, there was no relationship between using Theseus and debugging success found. 4 of 7 would use; 6 of 7 would recommend. Participants were pleased with how much information Theseus made available.
Evaluation 2: Interviews with 9 professional JavaScript programmers encouraged to use Theseus for 1 week.
Results: Interviews revealed little evidence about perceived or actual time-wasting.4 of 7 subjects expressed interest in more always-on displays.
Evaluation & Results
Griffin Prechter
Addressing Misconceptions About Code with Always-On Programming Visualizations
Tom Lieber, Joel Brandt, Robert C. Miller
Some programmers did enjoy the availability of reachability color and call counts. Some programmers adopted new problem solving strategies.
In the future, work towards increasing Theseus’s omniscience. Capture finer-grained detail of control flow and function invocation to provide even more information through always-on visualization.
Conclusions & Future Work
Translating keyword commands into executable code in the context of the application
Max Yao
Translating keyword commands into executable code
Greg Little, Robert C. Miller
Tokenize and pre-process user command, then recursively match against permutations of possible functions (and parameters) via heuristic evaluation, with limited depth.
Translating keyword commands into executable code
Greg Little, Robert C. Miller
Max Yao
How this came from PL:
Translating keyword commands into executable code
Greg Little, Robert C. Miller
Max Yao
Suppose the user’s input has n tokens, and every substring of tokens matches f functions in the library, each taking a arguments. The first call to the recursive algorithm must try every way to divide the n tokens into a + 1 substrings in any order (one for each argument plus the function name it-self), which is O(ana). And since each substring matches f functions, this gives O(fana) possibilities for each recursive call. ince the function tree for n tokens can have at most n nodes (ignoring function inference), the total search time would be O((fana)n). In practice, f should be small (tokens match few functions), a is small (most functions take few arguments), and n is small (users use few tokens), so this worst case is unlikely to bite.
Translating keyword commands into executable code
Greg Little, Robert C. Miller
Max Yao
Evaluations: Performance
Evaluation: User Study
Translating keyword commands into executable code
Greg Little, Robert C. Miller
Max Yao
Evaluation: User Study Results
The non-programmer group succeeded at 84% of the tasks, and the programmer group succeeded at 95% of the tasks. (We found this difference statistically significant using a two- tailed t-test, with p = 0.04.) Each group averaged 1.7 attempts per task. Non-programmers completed 72% of the tasks on the first try, with only one command. The programmers achieved this for 77% of the tasks. If the system under- stood only JavaScript, and we had offered no instructions, we would have expected a completion rate around 0% for both groups.
Translating keyword commands into executable code
Greg Little, Robert C. Miller
Max Yao
How do you give meaning to a program with expression and type holes?
If evaluation reaches a hole, intuitively we want to keep evaluating, but also track the context and then display the result to the developer.
Contributions
Live Functional Programming with Typed Holes
Cyrus Omar, Ian Voysey, Ravi Chugh, Matthew A. Hammer
Gabriel Matute
Gabriel Matute
Live Functional Programming with Typed Holes
Cyrus Omar, Ian Voysey, Ravi Chugh, Matthew A. Hammer
Day 2
Problem: Data visualization is time consuming and requires expertise in data wrangling and visualization.
Sam
Visualization by Example
Chenglong Wang, Yu Feng, Rastislav Bodik, Alvin Cheung, Isil Dillig
Solution: Program synthesis!
Sam
Visualization by Example
Wang et al.
Output: 2-part candidate programs (wrangling + vis)
Input: Data & Visualization “sketch”
HCI
Sam
Visualization by Example
Wang et al.
PL
Table transformation language
Visualization language
Sam
Visualization by Example
Wang et al.
Evaluation
Sam
Visualization by Example
Wang et al.
Evaluation
Sam
Visualization by Example
Wang et al.
How can we create a (pedagogical) programming environment to encourage novices to plan?
Nate
Pyrus: Designing a Collaborative Game to Promote Problem Solving Behaviors
Shi et. al., Northwestern
Build a “serious game” where players, with asymmetric abilities, take turns to create a program one “construct” at a time
Nate
Pyrus: Designing a Collaborative Game to Promote Problem Solving Behaviors
Shi et. al., Northwestern
Build a “serious game” where players, with asymmetric abilities, take turns to create a program one “construct” at a time
Stack
Do While
For Loop
Conditional
Nate
Pyrus: Designing a Collaborative Game to Promote Problem Solving Behaviors
Shi et. al., Northwestern
Build a “serious game” where players, with asymmetric abilities, take turns to create a program one “construct” at a time
Discrete Actions
Distributed Resources
Failure Condition
Enforced Turntaking
Nate
Pyrus: Designing a Collaborative Game to Promote Problem Solving Behaviors
Shi et. al., Northwestern
Nate
Pyrus: Designing a Collaborative Game to Promote Problem Solving Behaviors
Shi et. al., Northwestern
PL
HCI
New programming environment with significant non-typing interactions
Constructs feel a bit like structured editors or schema/idioms
Levels of abstraction for construction
Serious Games: Games designed not primarily for entertainment
Behavior-Centered Game Design: Obstacle → Desired Behavior → Mechanics
Transcripts, logs, interviews, and surveys
Did it work? Kinda. Compared to pair programming… (n=18)
2x time spent planning solutions in Pyrus� ...including planning around Pyrus
Otherwise, ~1.4x, but not significant
Also ~3x slower to use Pyrus
Nate
Pyrus: Designing a Collaborative Game to Promote Problem Solving Behaviors
Shi et. al., Northwestern
How interactive can we make synthesis? How user-friendly and efficient are these interaction methods?
ameesh
Interactive Program Synthesis
Vu Le et. al., MSR
Sampling for Bayesian Program Learning, Ellis et al. 2018.
How does the synthesizer know which program to choose?
Approaches to interactive Program Synthesis:
Interactive Program Synthesis
Vu Le et. al., MSR
ameesh
1. Incremental (CEGIS + efficient twists)
3. Step-based (user-guided top-down)
2. Feedback-based (clarify ambiguities in solution space with questions)
Refine search space with each iteration!
Have user tell you which next step to take
evaluation
ameesh
Interactive Program Synthesis
Vu Le et. al., MSR
Incremental: leads to speedup over non-incremental methods (with relatively few iterations needed)
Step-based: required less intervention from the user than when all examples must be provided manually
Feedback-based: required fewer user inputs than providing examples manually
takeaways
Interactive Program Synthesis
Vu Le et. al., MSR
ameesh
Data needs to be pre-processed (wrangled) before analysis / visualizing but tools are outside of notebooks
Wrex: A Unified Programming-by-Example Interaction for
Synthesizing Readable Code for Data Scientists
Ian Drosos, Titus Barik, Philip J. Guo, Robert DeLine, Sumit Gulwani
(UCSD, Microsoft)
🦆
Example data wrangling task: create derived field
Generate readable wrangling code via programming by example, in the notebook environment
Wrex: A Unified Programming-by-Example Interaction for
Synthesizing Readable Code for Data Scientists
Ian Drosos, Titus Barik, Philip J. Guo, Robert DeLine, Sumit Gulwani
(UCSD, Microsoft)
🦆
A: User creates data frame
B: Wrex presents an interactive grid view, and user can create a derived column and provide example values
C: Wrex synthesizes the data transformation program, as a new cell
D: Synthesized code can be inserted into a new cell
E: Derived columns can be used
Generate readable wrangling code via programming by example, in the notebook environment
Wrex: A Unified Programming-by-Example Interaction for
Synthesizing Readable Code for Data Scientists
Ian Drosos, Titus Barik, Philip J. Guo, Robert DeLine, Sumit Gulwani
(UCSD, Microsoft)
🦆
Generate readable wrangling code via programming by example, in the notebook environment
Wrex: A Unified Programming-by-Example Interaction for
Synthesizing Readable Code for Data Scientists
Ian Drosos, Titus Barik, Philip J. Guo, Robert DeLine, Sumit Gulwani
(UCSD, Microsoft)
🦆
Qualitative Feedback
Program synthesizers are not very interactive… (but a REPL is).
①
②
.
.
.
Justin
Programming with a Read-Eval-Synth Loop
Hila Peleg, Roi Gabay, Shachar Itzhaky, Eran Yahav
for i in
print("
for i in
??
Synthesizer
examples
for i in
print("
if i <=
else:
for i in
??
Synthesizer
examples
examples
examples
for i in
print("
Evaluator
5
for i in
print("
if i <=
else:
Evaluator
5
8
Solution: Incorporate a program�synthesizer into a REPL to form a
RESL
Justin
Programming with a Read-Eval-Synth Loop
Hila Peleg, Roi Gabay, Shachar Itzhaky, Eran Yahav
Justin
Programming with a Read-Eval-Synth Loop
Hila Peleg, Roi Gabay, Shachar Itzhaky, Eran Yahav
Custom discriminators to create equivalence classes for observational equivalence in enumerative search
Justin
Programming with a Read-Eval-Synth Loop: The PL
Hila Peleg, Roi Gabay, Shachar Itzhaky, Eran Yahav
Justin
Programming with a Read-Eval-Synth Loop: The PL
Hila Peleg, Roi Gabay, Shachar Itzhaky, Eran Yahav
Justin
Programming with a Read-Eval-Synth Loop
Hila Peleg, Roi Gabay, Shachar Itzhaky, Eran Yahav
Evaluation: user study (quantitative analysis)
Justin
Programming with a Read-Eval-Synth Loop: The HCI
Hila Peleg, Roi Gabay, Shachar Itzhaky, Eran Yahav
RQ1: Does RESL reduce the number of edit iterations and the portion of the code written by the user? → YES
RQ2: Does RESL bridge knowledge gaps and reduce the need for documentation? → YES
RQ3: Does RESL reduce task abandonment? → YES
RQ4: Does RESL speed up time to solution? → ¯\_(ツ)_/¯
RQ5: Are RESL users correct? → YES
RQ6: Does RESL improve knowledge of JavaScript? → NO
Justin
Programming with a Read-Eval-Synth Loop: The HCI
Hila Peleg, Roi Gabay, Shachar Itzhaky, Eran Yahav
How can teachers explore the space of student solutions to programming assignments in large computer science courses?
Allen
OverCode: Visualizing Variation in Student Solutions to Programming Problems at Scale
Elena L. Glassman, Jeremy Scott, Rishabh Singh, Philip Guo, Robert C. Miller
Using static and dynamic analysis, OverCode clusters similar solutions together and provides a way to visualize these clusters.
Allen
OverCode: Visualizing Variation in Student Solutions to Programming Problems at Scale
Elena L. Glassman, Jeremy Scott, Rishabh Singh, Philip Guo, Robert C. Miller
Allen
OverCode: Visualizing Variation in Student Solutions to Programming Problems at Scale
Elena L. Glassman, Jeremy Scott, Rishabh Singh, Philip Guo, Robert C. Miller
PL | HCI |
Static analysis to reformat solutions to have consistent line indentation and token spacing, also gets rid of comments | (Iterative design specifically for human readability) Visualization that shows similarity and variation among solutions, with cleaned code shown for each variant |
Lightweight dynamic analysis algorithm that uses variable renaming to cluster solutions whose variables take on the same sequence of values when executed on an autograder test | (Need finding) Teachers find OverCode easy to use; it allows them to read code that represents many student solutions instead of having to tediously go through every single submission |
Resulting algorithm is linear in the number of solutions and the size of each solution (compared to quadratic for pairwise AST methods based on edit distance) | Teachers provide more useful feedback to students and have a better view of students’ understanding and misconceptions |
Allen
OverCode: Visualizing Variation in Student Solutions to Programming Problems at Scale
Elena L. Glassman, Jeremy Scott, Rishabh Singh, Philip Guo, Robert C. Miller
User study 1 and evaluation: Subjects were asked to browse thousands of student submissions and give feedback by writing a forum post. After the task, subjects found OverCode easier to use, more helpful, and less overwhelming when compared to the baseline.
Allen
OverCode: Visualizing Variation in Student Solutions to Programming Problems at Scale
Elena L. Glassman, Jeremy Scott, Rishabh Singh, Philip Guo, Robert C. Miller
User study 2 and evaluation: Subjects were given a fixed amount of time to look at student submissions and to identify the five most frequent strategies to solve a problem. In this study, the subjects were able to look at more student solutions in the given time frame with OverCode when compared to the baseline. Furthermore, the solutions they looked at represented a wider range of student solutions for the compDeriv task, which is the most challenging of the three tasks.
Allen
OverCode: Visualizing Variation in Student Solutions to Programming Problems at Scale
Elena L. Glassman, Jeremy Scott, Rishabh Singh, Philip Guo, Robert C. Miller
Takeaways and Limitations:
Problem:
Mesh decompilers synthesize flat output, which do not clearly capture repetitive patterns (such as the spokes in the image) and make edits tedious and error-prone.
Synthesizing Structured CAD Models with Equality Saturation and Inverse Transformations
Nandi, Willsey, Anderson, Wilcox, Darulova, Grossman, Tatlock
�
Randy
�
Solution:
Szalinksy is a tool that
Synthesizing Structured CAD Models with Equality Saturation and Inverse Transformations
Nandi, Willsey, Anderson, Wilcox, Darulova, Grossman, Tatlock
�
Randy
�
Synthesizing Structured CAD Models with Equality Saturation and Inverse Transformations
Nandi, Willsey, Anderson, Wilcox, Darulova, Grossman, Tatlock
�
Randy
�
Synthesizing Structured CAD Models with Equality Saturation and Inverse Transformations
Nandi, Willsey, Anderson, Wilcox, Darulova, Grossman, Tatlock
�
Randy
�
Synthesizing Structured CAD Models with Equality Saturation and Inverse Transformations
Nandi, Willsey, Anderson, Wilcox, Darulova, Grossman, Tatlock
�
Randy
�
Synthesizing Structured CAD Models with Equality Saturation and Inverse Transformations
Nandi, Willsey, Anderson, Wilcox, Darulova, Grossman, Tatlock
�
Randy
�
This paper shows how formal verification methods can be used to encode correct and appropriate social norms into the interaction design of social robots including how getting the feedback from formal verification increases designers’ ability to accurately find errors within their designs.
Gloria Tumushabe
Authoring and Verifying Human-Robot Interactions by David Porfirio, Allison Sauppé, Aws Albarghouthi, Bilge Mutlu
A scenario that inspires this:
If a robot comes to a hospital emergency room and keeps announcing its arrival, that would be rather disruptive and people would not appreciate that. Such a robot does not align with social norms.
Implementation of Formal Verification
Gloria Tumushabe
to Illustrate the interactions, the authors use LTL (Linear Temporal Logic) to specify correctness properties of software and hardware designs.
Using model operators such as:
G humanSpeaking → ¬ robotSpeaking
(G refers to the global operator: should hold every time)
F Farewell
(F is model operator for the future interactions)
[ robotSpeaking → ( X ¬ robotSpeaking ∨ X humanReady ) ] U humanReady
(X is the next operator and U is the until operator)
Rover
Gloria Tumushabe
On the left is the environment that the designer does the work in.
On the right is the completed implementation of the robert delivering a package to the user implemented in Rover.
The inside such as ask is a micro interaction, the handoff and remark are a group example
Results from the evaluation and study
Results from the user study show that verification assistance decreases error discrepancy and increases ease of finding and interpreting errors.
Gloria Tumushabe
Conclusion
This paper heavily focuses on HCI as it takes into account human factors in the design of the robot. The goal of the authors is to make sure that the robot and the human work together harmoniously which is the a major component in HCI. Another aspect of HCI that is covered is having the participants use Rover and evaluating how easy it is to find and interpret the errors.
In the implementation, the authors use Linear Temporal Logic to enforce formal verification which is an aspect of PL.
Gloria Tumushabe
Qutub
PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations
Toby Jia-Jun et al.
Natural Language Programming is a useful approach for task automation using intelligent agents,
But
“Cold Weather..”
“Heavy Traffic..”
Qutub
“a new multimodal domain-independent approach that combines natural language programming and programming-by-demonstration to allow users to first naturally describe tasks and associated conditions at a high level, and then collaborate with the agent to recursively resolve any ambiguities or vagueness through conversations and demonstrations.”
Qutub
FORMATIVE STUDY FINDINGS-
PUMICE
An agent that supports understanding of ambiguous natural language instructions for task by recursive definition of new/vague concepts in a multi-level top-down process.
Design Features :
System Implementation:
HCI Elements
PL Elements
Qutub
Qutub
EVALUATION & RESULTS
Qutub
PUMICE Short Demo
Professional webpages embed stylesheets that are complex and difficult for novices to understand
Vibhor
Ply: A Visual Web Inspector for Learning from Professional Webpages
Sarah Lim et al.
Ply helps novices learn CSS concepts and design patterns using CSS pruning and dependency map
Ply: A Visual Web Inspector for Learning from Professional Webpages
Sarah Lim et al.
Vibhor
Needfinding Study
Problems
Ply displays Implicit dependencies
Ply prunes Ineffective properties
Problem of Ineffective Properties
Problem of Implicit Dependencies
HCI elements
PL elements
Ply: A Visual Web Inspector for Learning from Professional Webpages
Sarah Lim et al.
Vibhor
Demo: Ply on CalCentral
Results
Ply: A Visual Web Inspector for Learning from Professional Webpages
Sarah Lim et al.
Vibhor
An Empirical Investigation of Programming Language Syntax
Andreas Stefik, Susanna Siebert
Pratyush Mishra
Programming language design is often ad-hoc and based on gut feelings and “best practices” (see: PL flamewars)
How to do evidence-based PL design?
An Empirical Investigation of Programming Language Syntax
Andreas Stefik, Susanna Siebert
Pratyush Mishra
This paper proposes an evidence-based approach to designing one aspect of PL design: syntax.�
Questions 1 & 2
Results
Method�Surveys asking students to rank the intuitiveness of code samples��Languages: Java, Python, ..., Quorum
1) What is the most “intuitive” syntax for a concept?
2) Does syntax from existing languages accurately capture a concept?
Evidence-based PL for novices
Questions 3 & 4
Results
Method�Asked students to look at code samples, and write new code��Languages: Java, Python, Ruby, Quorum, and Randomo
3) Are new “evidence-based” languages better for novices?�
4) Where does existing syntax fall short?
Like Quorum, but with random tokens
How to figure out which parts of syntax are confusing?
Technique: Token Attention Maps
Annotate each token with fraction of participants that used it correctly�
x = 1
for i in 1..50
x = x - i
// line 3 inside loop
end
Example: using TAMs to simplify Quorum
User Interaction Models for Disambiguation in Programming by Example
Mikaël Mayer, Gustavo Soares , Maxim Grechkin, Vu Le, Mark Marron, Oleksandr Polozov, Rishabh Singh, Benjamin Zorn, Sumit Gulwani
Peitong Duan
Problem: Programming by Example (PBE, i.e. synthesizing programs based on input examples specified by user) can be ambiguous
User Interaction Models for Disambiguation in Programming by Example
Mayer et. al.
Peitong Duan
Solution: Two user interaction models to narrow down synthesized program candidates
Program Navigation
Conversational Clarification
User Interaction Models for Disambiguation in Programming by Example
Mayer et. al.
Peitong Duan
User Interaction Models for Disambiguation in Programming by Example
Mayer et. al.
Peitong Duan
HCI
Two novel interaction models of how users could resolve ambiguity in PBE
A new PBE framework that that utilizes both models
User study that evaluated the effectiveness of both models
PL
Utilize version space algebra to succinctly represent the program set based on shared subspaces
Program Navigation: Used templating-based strategy to paraphrase programs into English
Conversational Clarification: Algorithm to iteratively narrow down top subexpressions candidates based on clarifying questions
User Interaction Models for Disambiguation in Programming by Example
Mayer et. al.
Peitong Duan
RQ1: Program Navigation and Conversational Clarification both improve program correctness
RQ2: Conversational Clarification is perceived as more useful than Program Navigation
RQ3: Conversational Clarification increased trust in the PBE system.
Current methods for synthesis of code snippets are inexpressive and limited e.g. autocomplete
Richard Lin
CodeHint: Dynamic and Interactive Synthesis of Code Snippets
Sen et. al. (Go bears!)
Illustration of problem, if relevant
New method for code synthesis that is dynamic, easy to use, and interactive
CodeHint: Dynamic and Interactive Synthesis of Code Snippets
Sen et. al. (Go bears!)
Richard Lin
How?
CodeHint: Dynamic and Interactive Synthesis of Code Snippets
Sen et. al. (Go bears!)
Richard Lin
4:18 - 6:32
Results
HCI: Interactive, less ambiguity
PL: Generate possible candidates using dynamic context, suggest more likely statements, code synthesis
CodeHint: Dynamic and Interactive Synthesis of Code Snippets
Sen et. al. (Go bears!)
Richard Lin