Old Apps,
New Tricks
�How AI can write Automated tests for your Shiny Apps
for
Python
Role:
QA Engineer, Shiny Team
Some things I to do:
Meet Gregory the Grasshopper
introvert
One's a hermit, two's a trigger, three’s chaos
The trigger
Serotonin is released
Dangerous threats can lie dormant until a trigger awakens them
Dormant threat
Dormant threat
prevents
Dormant threat
Dormant threat
Dangerous threat
App
Test
Bug(s)
App
Bug(s)
State of testing in the Shiny ecosystem
Shiny for R
Shiny for Python
Shinytest2
Playwright
Shiny for R
Shinytest2
Shiny for Python
Playwright
Shiny for Python
Playwright
Shiny for Python
Playwright
Shiny for Python
2023
Introduced controllers
Make it easy to write Playwright tests
Introduced shiny add test CLI command
Automatically generate a boilerplate test file for a Shiny for Python app
2024
Enhanced shiny add test
Integrated AI with the CLI command to generate complete E2E test for a Shiny for Python app
2025
Playwright
controllers simplify component interaction within tests
YOU
controllers
components
Playwright
InputCheckbox controller
Methods:
controllers simplify component interaction within tests
Shiny for Python
2023
Introduced controllers
Make it easy to write Playwright tests
Introduced shiny add test CLI command
Automatically generate a boilerplate test file for a Shiny for Python app
2024
Enhanced shiny add test
Integrated AI with the CLI command to generate complete E2E test for a Shiny for Python app
2025
type command
Shiny for Python
How 2024 shiny add test worked
Select App file
Test file is created
type command
Select App file
How 2024 shiny add test worked
Test file is saved
test scaffolding
test_app.py
Shiny for Python
How 2024 shiny add test worked
Test file is saved
test_app.py
Shiny for Python
How 2024 shiny add test worked
Test file is saved
test_app.py
Shiny for Python
What 2024 shiny add test lacked
What we provided
What we wished we provided
What we provided
It's 2025 now
where we are surrounded by AI
Shiny for Python
2023
Introduced controllers
Make it easy to write Playwright tests
Introduced shiny add test CLI command
Automatically generate a boilerplate test file for a Shiny for Python app
2024
Enhanced shiny add test�CLI command
Integrated AI with the CLI command to generate complete E2E test for a Shiny for Python app
2025
How 2025 shiny add test works
How 2025 shiny add test works
off-the-shelf AI
shiny add test
off-the-shelf AI
shiny add test
off-the-shelf AI
shiny add test
HALLUCINATIONS!!!
Enter the of using AI
How we minimize hallucinations within shiny add test
We run 10 evaluation apps through shiny add test to generate tests
How we minimize hallucinations within shiny add test
Run all generated tests
Passing % > 85%
Evaluate tests through inspect-ai scorers
Passing % > 80%
Evaluate tests through inspect-ai scorers
Passing % > 80%
Show me the steps
type command
Enter path to Shiny app
Enter path for test file
DEMO
Some tips to get the most from your shiny add test experience
1. Use premium models from Anthropic or OpenAI for best results
Some tips to get the most from your shiny add test experience
2. The generated tests will only cover Shiny components — they won’t test third‑party APIs or databases.
1. Use premium models from Anthropic or OpenAI for best results
Some tips to get the most from your shiny add test experience
2. The generated tests will only cover Shiny components — they won’t test third‑party APIs or databases.
3. Use ids for every shiny component within your Shiny App
1. Use premium models from Anthropic or OpenAI for best results
Some tips to get the most from your shiny add test experience
2. The generated tests will only cover Shiny components — they won’t test third‑party APIs or databases.
3. Use ids for every shiny component within your Shiny App
1. Use premium models from Anthropic or OpenAI for best results
4. Use caution when sharing code with LLMs
Generate Shiny app tests with AI
Karan Gathani
/Shiny Team
@karangattu
shiny add test CLI command