Automating Software Development: Challenges and Solutions
Jiasi Shen
The Hong Kong University of Science and Technology
AutoMates @ IJCAI 2023
1
Aug 2023
Research in programming languages
Historically: Language design, compilers
Now: Reasoning about computations�(e.g., analyzing and proving properties, improving software quality)
My focus: Automating the software development process
“If your research is 100% successful, what will happen in 20-30 years?”
2
Aug 2023
3
Aug 2023
4
Aug 2023
5
Aug 2023
6
Aug 2023
7
Aug 2023
8
Aug 2023
9
Aug 2023
10
Aug 2023
(my copy-and-paste error)
GPT4
GPT3.5
11
Aug 2023
(my copy-and-paste error)
GPT4
GPT3.5
12
Aug 2023
(my copy-and-paste error)
GPT4
GPT3.5
Observations of AI-based code generation
Strengths
Generality
Fixing syntax errors
Suggesting boilerplate code
Suggesting “textbook-ed” solutions
Explaining code with natural language
Weaknesses
Reasoning about sophisticated logic
Less-common functionality
Guarantees
Scenarios that can’t tolerate noise
13
Aug 2023
Example scenario: Automated program repair
A program fails some test cases
A tool suggests patches for fixing the program
14
Aug 2023
J. P. Cambronero, J. Shen, J. Cito, E. Glassman and M. Rinard, "Characterizing Developer Use of Automatically Generated Patches," VL/HCC’19
Example patches
15
Aug 2023
J. P. Cambronero, J. Shen, J. Cito, E. Glassman and M. Rinard, "Characterizing Developer Use of Automatically Generated Patches," VL/HCC’19
Example scenario: Automated program repair
A program fails some test cases
A tool suggests patches for fixing the program
Realistic assumption:
The tool suggests five plausible patches
Is this tool helpful or not?
16
Aug 2023
J. P. Cambronero, J. Shen, J. Cito, E. Glassman and M. Rinard, "Characterizing Developer Use of Automatically Generated Patches," VL/HCC’19
Controlled experiment
Control group:
Treatment group:
Participants:
Findings:
17
Aug 2023
J. P. Cambronero, J. Shen, J. Cito, E. Glassman and M. Rinard, "Characterizing Developer Use of Automatically Generated Patches," VL/HCC’19
Observations of AI-based code generation
Strengths
Generality
Fixing syntax errors
Suggesting boilerplate code
Suggesting “textbook-ed” solutions
Explaining code with natural language
Weaknesses
Reasoning about sophisticated logic
Less-common functionality
Guarantees
Scenarios that can’t tolerate noise
18
Aug 2023
Faster and easier to generate “bad” code
Provide stronger guarantees
Provide insights to facilitate adoption
19
Aug 2023
AI�research
PL/SE�research
Automate�software�development
Generality
Automation
Faster and easier to generate “bad” code
20
Aug 2023
AI�research
PL/SE�research
Automate�software�development
Generality
Automation
Guarantees
Reasoning
POPL 2023, PLDI 2022, OOPSLA 2022 sessions
Algorithmic Verification
Analysis
Assurance
Automated Verification
Automatic Differentiation
Blockchain
Compilation
Compile
Concurrency
Concurrency
Concurrency & Linearizability
Data
Debugging
Distribution
DSLs
Effects
Formal Methods in Compilation & Implementation
Hardware I
Hardware II
Logic & Decidability I
Logic & Decidability II
Logic and Concurrency
Logic and Verification I
Logic and Verification II
Logic Programming
Numbers
Probabilistic
Probabilistic Inference
Program Analysis & Parsing
Program Logics & Resources
Proofs
Quantum
Quantum
Quantum Computing
Relational & Automated Verification
Resource Analysis
Runtime
Security
Security
Semantics
Semantics & Effects
Semantics and Security
Semantics I
Semantics II
Synthesis I
Synthesis I
Synthesis I
Synthesis II
Synthesis II
Synthesis II
Synthesis III
Systems and Verification
Tensors
Testing & Synthesis
Testing and Maintenance
Type Theory
Types
Types I
Types II
Verification
Verification & Optimization
Verification I
Verification II
Verified Compilation
21
Aug 2023
22
Aug 2023
AI�research
PL/SE�research
Automate�software�development
Generality
Automation
Guarantees
Reasoning
Usage scenario may be restrictive
Challenge: Develop new software�rapidly, cheaply, and securely
23
Aug 2023
https://spectrum.ieee.org/facebook-philosophy-move-fast-and-break-things
https://www.akamai.com/content/dam/site/en/documents/state-of-the-internet/akamai-state-of-the-internet-gaming-in-a-pandemic-infographic.pdf
Challenge: Extract, maintain, and reuse the�human knowledge embedded in software
24
Aug 2023
https://www.gao.gov/products/GAO-19-471
25
Aug 2023
https://nj.gov/governor/news/news/562020/approved/20200404b.shtml
https://nj.gov/governor/news/news/562020/approved/20200408c.shtml
https://news.bloomberglaw.com/banking-law/an-ancient-computer-language-is-slowing-americas-giant-stimulus
April 4th, 2020 Coronavirus Briefing Media
April 8th, 2020 Coronavirus Briefing Media
“not only do we need healthcare workers but given the legacy systems we should add a page for COBOL computer skills because that’s what we’re dealing with in these legacies… But literally, we have systems that are 40 years-plus old, and there’ll be lots of postmortems. And one of them on our list will be how did we get here where we literally needed COBOL programmers?”
“Our online portal is now open for people with experience in a variety of technological areas, and for those who wish to join our team at this time, please head to covid19.nj.gov/tech. For all of you COBOL programmers in particular, now is your chance.”
Challenge: Extract, maintain, and reuse the�human knowledge embedded in software
26
Aug 2023
https://www.digitalrealty.com/resources/white-papers/all-change-in-financial-services-as-incumbents-meet-startups
https://www.pdisoftware.com/blog/what-are-the-top-tech-trends-for-petro-retailers/
https://www.pdisoftware.com/wp-content/uploads/2021/05/5-Tech-Trends-Petro-Retailer-Infographic-PDI-2021.pdf
June 3, 2021
“What are financial services firms doing to enhance their IT systems?”
All-Change in Financial Services as Incumbents Meet Startups
A White Paper by IDG Connect on behalf of Digital Realty
(IT decision makers in the UK financial services sector)
Early 2018
Challenge: Adapting software to new contexts rapidly, reliably, and flexibly
27
Aug 2023
https://www.defense.gov/News/Releases/Release/Article/2924187/dod-software-modernization-strategy-approved/
https://media.defense.gov/2022/Feb/03/2002932833/-1/-1/1/DEPARTMENT-OF-DEFENSE-SOFTWARE-MODERNIZATION-STRATEGY.PDF
Manual engineering effort
Development of new software
Detection and elimination of defects and security vulnerabilities in existing software
Maintenance of legacy software
Integration of existing software into more contexts
28
Aug 2023
29
Aug 2023
AI�research
PL/SE�research
Automate�software�development
Generality
Automation
Guarantees
Reasoning
Program Inference and Regeneration
30
Aug 2023
Inference and Regeneration
Old Program
New Program
J. Shen and M. Rinard. "Using active learning to synthesize models of applications that access databases." PLDI’19
J. Shen and M. Rinard. "Active Learning for Inference and Regeneration of Applications that Access Databases." TOPLAS’21
J. Shen and M. Rinard. "Active Loop Detection for Applications that Access Databases." InSubmission
Program Inference and Regeneration
Black box dynamic analysis
Active learning on the program behavior
Guaranteed correct for several domains
31
Aug 2023
Inference and Regeneration
Old Program
New Program
Choose inputs
Observe outputs
J. Shen and M. Rinard. "Using active learning to synthesize models of applications that access databases." PLDI’19
J. Shen and M. Rinard. "Active Learning for Inference and Regeneration of Applications that Access Databases." TOPLAS’21
J. Shen and M. Rinard. "Active Loop Detection for Applications that Access Databases." InSubmission
32
Aug 2023
AI�research
PL/SE�research
Automate�software�development
Generality
Automation
Guarantees
Reasoning