1 of 16

Beyond Static Test Scripts

An agentic approach to adaptive yet reliable Android testing

Rana Roy · Sasken Technologies

Demo walkthrough

2 of 16

THE PROBLEM

When the application evolves faster than the tests

Beyond Static Test Scripts

2

APPLICATION

Sprint 1

Sprint 2

Sprint 3

Sprint 4

TEST AUTOMATION

selector

wait

coordinate

navigation

assertion

…

UI changes

Test changes

Test maintenance

Test debt

3 of 16

THE PROBLEM

Beyond Static Test Scripts

3

What if the test suite�became the bottleneck?

Not because the app was broken.�Because the tests could not keep up with it.

4 of 16

THE PROBLEM · ROOT CAUSE

Tests are coupled to how, not what

Beyond Static Test Scripts

4

Selectors

UI hierarchy

Navigation path

Timing

Exact assertions

E2E test script

Every UI change is a test change.

Churn

Stale automation

Trust erosion

Application development runs at continuous-delivery speed; test automation runs at rewrite-and-catch-up speed. This slide is why.

A traditional E2E test does not encode intent - it encodes one path through one build of the UI: selectors and resource IDs, the view hierarchy, the exact navigation path, timing and sleeps, and exact-string assertions.

Say this out loud: NONE of those five is what the test is trying to prove. They are incidental - but they are what breaks. A designer moves a button into a bottom sheet and fifty tests go red while the feature works perfectly.

Then the bottom row: repair effort compounds, new coverage stalls, and eventually the team stops running the suite on every build. That is the moment the safety net is gone while everyone still believes it exists. Use your own numbers here if you have them.

5 of 16

PROPOSED SOLUTION

From how to what

Beyond Static Test Scripts

5

how the UI works

what must be true

implementation

intent

6 of 16

PROPOSED SOLUTION · THE ARTIFACT

A test skill is not just a prompt

Beyond Static Test Scripts

6

Test Identity

which test · which feature

Intent & Preconditions

what must be true before you start

Execution Guidance

preferred tools · hard constraints

Adaptive Guidance

read live UI · do not assume

Evidence & Reporting

what actually happened

Stable intent + bounded guidance + evidence contract

7 of 16

PROPOSED SOLUTION · WHY IT HOLDS

One redesign, two outcomes

Beyond Static Test Scripts

7

A redesign lands — the button moves into a bottom sheet

Script

selector gone

path changed

assertion moved

tests go red

Intent skill

same intent

agent re-locates

same outcome

still green

readable by QA and PM

reviewed in git, like code

one intent, many builds

This is the slide that has to earn the previous one. Showing a markdown file proves nothing on its own - the question in everyone's head is 'so what, why is that better than my Espresso test?'

Take one ordinary event: a designer moves a button into a bottom sheet. Nothing about the feature changed, and nothing about what we want to prove changed.

The scripted test experiences three separate breakages from that one change - the selector is gone, the navigation path is different, and the assertion text moved. All three are false failures, and someone spends a day proving the product was fine.

The intent skill experiences none of them. The sentence 'schedule a meeting tomorrow at 10 AM and confirm it lands in the calendar' is exactly as true after the redesign as before. The agent re-locates the control at runtime; the expected outcome is unchanged.

Then the three chips, which are the durable arguments: a PM or manual QA engineer can read and review the test; it lives in git so intent gets review and history like code; and one intent outlives many builds instead of being rewritten by each one.

Say the honest limit too: this does not make tests immune to change. It moves the change from fifty test files to zero, and concentrates it in one agent that adapts.

8 of 16

THE RISK

Probabilistic execution is hard to trust

Beyond Static Test Scripts

8

same intent

pass?

trust

reproduce

diagnose

scale

9 of 16

THE APPROACH

Reasoning where it is needed.�Determinism where it matters.

Beyond Static Test Scripts

9

Agentic

reasoning

Deterministic

controls

trust

10 of 16

PROPOSED ARCHITECTURE

Agentic reasoning on a deterministic substrate

Beyond Static Test Scripts

10

TEST SKILL

AGENT

Observe · Reason · Plan · Act · Validate · Recover

MCP tools

Android actions

tap · swipe · type · launch

Validation

screenshot · UI inspect · app checks

Android App

Observability

Evidence

Verification

Recovery

Classification

11 of 16

SEE IT RUN

A short recording

Beyond Static Test Scripts

11

▶

LIVE DEMO / RECORDED RUN — TO BE INSERTED

Replace this shape with the recording; the slide needs no other change.

12 of 16

MAKING IT TRUSTWORTHY

Five controls around the agent

Beyond Static Test Scripts

12

01

Deterministic�setup & teardown

preconditions are�established, not assumed

02

Complete�execution traces

video, logcat, tool calls,�screenshots, report

03

Adversarial�verification

the agent does not�grade its own homework

04

Bounded�self-healing

retry cap, no-progress�guard, time budget

05

Failure�classification

product defect vs�everything else

The spine of the second half - about four minutes on this slide, roughly 45 seconds each.

The design rule that generated all five: everywhere the agent could be believed without proof, insert a deterministic gate.

01 SETUP/TEARDOWN - deterministic code, not the agent, runs before and after every test: check, establish, re-verify, then run. If a precondition still cannot be met, the run stops BEFORE step one and reports BLOCKED. Real example: three tests were once reported as product failures when the causes were an unconnected calendar connector and memory left dirty by the previous test.

02 TRACES - every run writes a timestamped folder: screen recording, full logcat, the tool-call trace, screenshots and a structured report. A step does not pass without evidence, and the evidence must show what the step claims - we once passed a 10 AM meeting on a frame that started at 3 PM, because the match came from an off-screen node.

03 ADVERSARIAL VERIFICATION - an agent that performs the action and judges it is optimistic by construction. A separate pass is asked to DISPROVE the claim, judging rendered pixels rather than the tree the actor just walked.

04 BOUNDED HEALING - self-healing is the selling point and also how it hangs forever. At most three retries, a no-progress guard, honour declared waits, and an overall time budget.

05 CLASSIFICATION - the next slide.

13 of 16

CONTROL 05

When something fails, who is responsible?

Beyond Static Test Scripts

13

FAILURE

PRODUCT

DEFECT

ENV

ISSUE

INFRA

ISSUE

PRECONDITION

ISSUE

AGENT

ISSUE

Only product defect counts against the app. Everything else is lab, setup or agent.

14 of 16

REASON ONCE

Replay many

Beyond Static Test Scripts

14

FIRST RUN

AGENT REASONS

VALIDATED

Deterministic path?

YES

REPLAY

CI runs

DRIFT?

no

yes

PASS

HEAL → AGENT

NO

AGENTIC EVERY RUN

no replay artifact

compile: nav · forms · asserts

stay agentic: semantic / generated

15 of 16

THE LIFECYCLE

One loop

reason → validate → determinize → replay → heal

Beyond Static Test Scripts

15

Reason

Validate

Determinize

Replay

Heal

LLM cost paid once, per drift

16 of 16

Use AI where reasoning is required.�Use determinism where repeatability matters.

Thank you — questions?