Technical Analysis of Agentic AI Efficacy in Navigating Complex Web Architectures for Accessibility Remediation

The intersection of generative artificial intelligence and digital accessibility has entered a transformative phase with the introduction of "agentic" browser features. These technologies, exemplified by Google’s Auto Browse in Gemini for Chrome and Microsoft’s Copilot Actions in Edge, represent a departure from static screen-reading software toward dynamic, reasoning-based assistants.1 While traditional assistive technologies (AT) rely on the explicit semantic structure of the Document Object Model (DOM) and the Accessibility Object Model (AOM), agentic AI seeks to interpret the visual and functional intent of a web application to perform multi-step tasks autonomously.4 However, as observed in enterprise environments such as the Salesforce Experience Cloud and Trailhead Marketplace, the performance of these agents remains inconsistent. The unreliability of these systems is not merely a byproduct of early-stage development but is rooted in fundamental tensions between autonomous interaction, browser security architectures, and the structural opacity of modern web frameworks.7

The Architecture of Native Browser Agents

To understand the operational hurdles facing these technologies, it is necessary to examine the technical foundations of native browser agents. Unlike standard extensions that interact with the web through high-level APIs, Gemini in Chrome and Copilot in Edge are integrated directly into the browser’s runtime environment.4

Google Gemini and the Chrome Auto Browse Ecosystem

Google’s Auto Browse, launched as a headline feature for Gemini in Chrome on January 28, 2026, utilizes the Gemini 3 multimodal model to bridge the gap between user intent and browser execution.13 The system functions by launching a browsing session within a protected Chrome profile sandbox, leveraging the Chrome DevTools Protocol (CDP) to interact with web elements.4 This native integration allows the AI to "see" the page through a hybrid lens of raw HTML structure and multimodal visual interpretation.4

The capability of Gemini to select from previously inaccessible comboboxes in the Salesforce Trailhead Career Marketplace illustrates its ability to resolve elements that lack traditional accessibility markup.13 Because Gemini can process images and video content in real-time, it can identify the visual "bounds" of a dropdown menu even if the aria-expanded state or role="combobox" attribute is missing or misconfigured.5 However, this reliance on visual reasoning introduces non-determinism; if the layout shifts or the rendering engine delays the display of the combobox list, the agent may fail to locate the target coordinates, leading to the "hit-and-miss" results reported by users.10

Microsoft Edge Copilot and the Actions Framework

Microsoft’s approach through Copilot Actions in Edge focuses on a chat-driven assistant that "sees" the browser through periodic screenshots.2 This visual grounding allows Copilot to provide summaries, compare products across tabs, and automate certain tasks like unsubscribing from newsletters or booking travel.3

The Edge architecture defines three distinct security tiers that govern how Copilot interacts with the DOM. The "Light" mode provides minimal protection, allowing for broad autonomy, while the "Balanced" mode—the recommended default—requires user approval for unfamiliar sites.2 The "Strict" mode enforces a permission request for every site interaction.2 These tiers are critical because they dictate the agent's ability to "accessify" a page. If a site is not on Microsoft’s curated "allow list," the agent is functionally paralyzed unless the user manually intervenes.2

The failure of Copilot to interact with the Salesforce Experience Builder is a direct consequence of these security boundaries. While the user attempted to use Copilot to bridge an accessibility gap, the browser's security model perceived the action—dragging a component to a canvas—as a high-risk operation involving cross-origin manipulation or potential UI redressing.2

The Salesforce Experience Builder Barrier: Technical Constraints

Salesforce Experience Builder is a sophisticated, single-page application (SPA) that relies heavily on the Shadow DOM, iframes, and dynamic rendering.8 These technologies, while beneficial for modularity and style encapsulation, represent an "impenetrable fortress" for many automated agents.20

Shadow DOM and the Visibility Crisis

The Shadow DOM allows developers to encapsulate a component's internal HTML and CSS, preventing it from being modified by global styles or scripts.20 For a native browser agent like Gemini or Copilot, the Shadow DOM creates a two-fold problem. First, traditional selectors like XPath and CSS often stop at the "shadow root," rendering the internal elements (like the drag-handle of a headline component) invisible to the agent's DOM parser.8 Second, even if an agent uses visual perception to identify a button, it cannot easily map those pixels to a specific, interactable DOM node without "piercing" the shadow boundary.21

Modern testing frameworks like Playwright can handle Shadow DOM traversal, but the reasoning engines in Gemini and Copilot often rely on a simplified, flattened representation of the page to save on token costs.8 When a Salesforce page contains nested shadow roots, the agent's internal "map" of the page becomes fragmented.8

The Physics of Drag-and-Drop in AI Reasoning

The user's specific failure with drag-and-drop highlights a "grounding gap" in agentic AI. Unlike a "click" action, which is a discrete event at a single coordinate , drag-and-drop is a continuous state-dependent workflow involving a sequence of events: mousedown, mousemove (potentially involving multiple layout shifts as the ghost image follows), and mouseup.18

For an AI agent to execute a drag-and-drop, it must maintain a real-time, high-fidelity model of the page's geometry. Most vision-based agents, however, operate on a latency-heavy loop where a screenshot is taken, processed by the model (taking 1-2 seconds), and then a command is sent.22 By the time the agent attempts to "drop" the component on the Experience Builder canvas, the underlying JavaScript may have timed out the drag event, or the "drop zone" may have failed to initialize because the agent's simulated movement did not trigger the necessary mouseover events.10

Security Policies and the Refusal to Act

The "security features" cited by Copilot—and the less informative refusal by Gemini—are rooted in the industry-wide concern regarding "Same-Origin Policy (SOP) Collapse" and "Privileged Browser Function" abuse.7

SOP Collapse and the Risks of Autonomy

The Same-Origin Policy is the cornerstone of web security, ensuring that a script from one site cannot access data from another site.7 When an AI agent is granted the power to read content in one tab (e.g., an email with a headline) and act on another tab (e.g., the Salesforce canvas), the agent itself becomes the bridge between origins.9

If an agent were allowed to perform drag-and-drop freely, a malicious website could theoretically "trick" the agent into dragging sensitive data (like a session token displayed on a page) into an attacker-controlled form.9 Consequently, Microsoft and Google have implemented "hard blocks" on complex UI interactions in sensitive applications.2 These blocks are particularly aggressive when the agent detects a "Builder" or "Admin" interface, as the potential for catastrophic data loss or system misconfiguration is significantly higher.27

CVE-2026-0628 and the Gemini Vulnerability

A critical factor in Google's "less informative" response may be related to vulnerabilities like CVE-2026-0628, where attackers found they could tap into the browser environment by hijacking the Gemini browser panel.7 This vulnerability allowed malicious extensions to access the user's camera, microphone, and local files by injecting JavaScript into the privileged Gemini interface.7 To mitigate these risks, browser vendors have restricted the agent's ability to execute arbitrary JavaScript or simulate complex mouse movements in "sensitive" browser windows, which include authenticated sessions in enterprise software like Salesforce.7

The Perception-Reasoning-Action Loop: Quantifying Unreliability

The "hit-and-miss" results are a function of the agent's failure at different stages of the execution loop. If we represent the probability of task success () as the product of the probabilities of successful Perception (), Reasoning (), and Action ():

The current state of technology sees high variance in all three variables when applied to inaccessible web apps.5

Perceptual Failure ()

Agents that rely on DOM scraping fail when the DOM is messy, which is typical of inaccessible sites.10 Agents that rely on vision (screenshots) fail due to "coordinate guessing".18 A vision model might see a "Headline" component and estimate its center at , but if the actual clickable "handle" is at , the simulated click will miss the target.18 Research shows that vision-based agents have an error rate of 10-20% simply in misidentifying similar-looking elements or missing elements obscured by overlays (like Salesforce's floating property panels).22

Reasoning Failure ()

Even if the agent perceives the element, it may fail to reason through the sequence. Salesforce Experience Builder requires the user to click and hold before moving. Many AI agents are programmed for simple click or type actions and do not have a robust "state machine" for long-duration drag events.24 This leads to "reasoning loops" where the agent tries to click the component repeatedly, gets no response, and eventually gives up.10

Action Failure ()

The action phase is where security and infrastructure limits are felt. Chrome's "Auto Browse" has a hard execution limit of 60 minutes per task, but many agents time out much earlier—often after 7 seconds of inactivity from an external API or slow Salesforce server.31

Comparative Analysis of Alternative Technologies

Given the limitations of native browser agents, other specialized technologies may offer more reliable alternatives for "accessifying" web content for blind and low-vision users.

MultiOn: The Autonomous Reasoning Agent

MultiOn is an agentic layer that sits on top of the browser rather than being "baked in".33 It focuses on "what" the user wants rather than the technical selectors of the page.33 MultiOn uses vision-capable models that are specifically trained to adapt to website layout changes.30

• Advantages: High adaptability; strong at interpreting visual cues; does not rely on hardcoded CSS selectors.5
• Disadvantages: Significant latency (due to the extra reasoning layer); high cost; potential for "runaway" loops on complex tasks.33

While MultiOn is excellent for research (e.g., "find the cheapest flight"), its latency makes it poorly suited for real-time interface design in the Experience Builder, where the user needs immediate feedback on component placement.30

Guide: The Specialized AI Accessibility App

The "Guide" application is a specialized Windows desktop tool designed specifically for blind and low-vision users.37 Unlike browser-based agents, Guide takes a screenshot of the entire computer and uses AI (Claude) to interpret the screen content.24

Guide's primary innovation is that it executes commands by taking direct control of the Windows mouse and keyboard.24 This is a critical distinction from native agents like Copilot, which try to act "within" the browser's DOM.2 Because Guide simulates a physical human input at the OS level, it is not subject to the same "Same-Origin Policy" restrictions that block Copilot and Gemini.24

Guide has been specifically highlighted as a tool that allows blind users to "drag and drop components onto canvases in builder apps" that were not designed with accessibility in mind.16 The tool's ability to tell the user the steps it is taking allows the user to learn the workflow and even "record" successful interaction patterns for future use.24

rtrvr.ai and the DOM-Native Intelligence

A different path is taken by technologies like rtrvr.ai, which eschew vision-based models in favor of "DOM-native intelligence".22 By using a Chrome extension to traverse the live, rendered DOM, these tools avoid the OCR hallucinations and coordinate drift that plague vision agents.22 They can natively navigate Shadow DOMs and handle infinite scrolls by understanding the underlying framework logic rather than just the pixels.22

For a user struggling with the Salesforce Experience Builder, a DOM-native tool would likely be more successful at identifying the " Headline" component because it can see the hidden semantic markers that the visual model might miss or misinterpret as decorative text.17

The Economic and Regulatory Driver: EAA 2025

The urgency of solving these unreliability issues is driven by shifting legal requirements. By June 2025, the European Accessibility Act (EAA) will mandate that a wide range of digital products and services, including SaaS platforms like Salesforce, meet strict accessibility standards (WCAG 2.1/2.2).40

The global market for assistive technologies for visually impaired users is projected to reach $6 billion in 2024 and double by 2030.41 However, blind users often find that "stapling an accessibility AI agent on top of a broken UI" is less effective than fixing the underlying semantic issues.41 As organizations like Salesforce face increased legal accountability, the "automated remediation" provided by agentic AI will likely shift from a "helper" role to a "compliance" layer, potentially forcing browser vendors to relax security restrictions for certified accessibility tools.42

The Future of Agentic Accessibility: Synthesis and Conclusion

The current unreliability of Auto Browse and Copilot Actions in "accessifying" web apps like Salesforce is a multi-layered technical failure. At the architectural level, native agents are hampered by their own security sandboxes, which prioritize data protection over the autonomous flexibility required to manipulate complex interfaces.2 At the perceptual level, the probabilistic nature of visual coordinate guessing is too imprecise for the "high-stakes" interactions of a builder canvas, where a miss of a few pixels renders an action void.18

For users seeking to bridge these gaps today, the most reliable approach is not to wait for native browser improvements but to adopt "Computer Using Agents" (CUAs) that operate at the OS level.24

Synthesized Insights for Future Development

1. AOM Integration Over DOM Scraping: For AI agents to become truly reliable, they must move away from raw HTML and toward the Accessibility Object Model (AOM). The AOM provides a filtered, semantically rich view of the page that is natively designed for assistive technology, bypassing much of the structural "noise" that causes context overflow in current LLMs.5
2. Semantic Geometry as a Middle Layer: Developers are increasingly building "semantic geometry" layers (like SentienceAPI) that map visible pixels back to interactable DOM nodes.18 This allows an agent to "know" that the headline it sees visually is tied to a specific <div> with a draggable attribute, reducing the reliance on coordinate guessing.18
3. Role-Based Agent Handoffs: Rather than one "general" agent trying to handle everything, the future of accessibility lies in "multi-agent orchestration".36 A specialized "UI Navigation Agent" could identify elements, while an "Execution Agent" with specific security clearances performs the actual drag-and-drop, mediated by a "Supervisor Agent" that ensures the user’s intent is being followed.29
4. Bypassing the Browser Sandbox: The success of the "Guide" application suggests that for the most difficult "accessification" tasks, the browser is actually the bottleneck. By moving the reasoning loop to the desktop level and simulating HID (human interface device) inputs, AI can provide a more consistent bridge to inaccessible web content without triggering the browser's cross-origin security alarms.24

In conclusion, while Gemini Auto Browse and Copilot Actions represent a major step forward in "smart browsing," they are currently optimized for consumer tasks rather than the rigorous requirements of enterprise accessibility remediation.3 Users with physical or visual disabilities should look toward OS-integrated CUAs like Guide or agentic layers like MultiOn, while advocating for broader adoption of the AOM standard to provide a more stable foundation for the next generation of AI-driven inclusion.5

Works cited

1. Chrome + Gemini Auto Browse agent is Actually Insane, accessed March 28, 2026, https://www.youtube.com/watch?v=6mQb7Tg_uuc
2. Copilot Actions in Edge - Microsoft Support, accessed March 28, 2026, https://support.microsoft.com/en-au/topic/copilot-actions-in-edge-5ed5e17e-42df-40a3-984a-20420eba86e2
3. The AI Browser War: Chrome, Edge, ChatGPT Atlas & Comet | Orange SEO, accessed March 28, 2026, https://www.orangeseo.net/blog/2025/10/23/the-ai-browser-war
4. Auto Browse Explained: How Google's AI Agent Takes Over Chrome : r/AISEOInsider, accessed March 28, 2026, https://www.reddit.com/r/AISEOInsider/comments/1qfme9x/auto_browse_explained_how_googles_ai_agent_takes/
5. Screen Reader AI: A Conversational Web-Accessibility Assistant for Blind and Low-Vision Users - ResearchGate, accessed March 28, 2026, https://www.researchgate.net/publication/396362763_Screen_Reader_AI_A_Conversational_Web-Accessibility_Assistant_for_Blind_and_Low-Vision_Users
6. How Screen Readers Actually Work: From DOM to Speech - TestParty, accessed March 28, 2026, https://testparty.ai/blog/how-screen-readers-parse-dom
7. Taming Agentic Browsers: Vulnerability in Chrome Allowed Extensions to Hijack New Gemini Panel - Palo Alto Networks Unit 42, accessed March 28, 2026, https://unit42.paloaltonetworks.com/gemini-live-in-chrome-hijacking/
8. The three main challenges of testing Salesforce, explained | QA Wolf, accessed March 28, 2026, https://www.qawolf.com/blog/three-main-challenges-of-testing-salesforce
9. The Security Risks of AI Browser Agents for Enterprise - Witness AI, accessed March 28, 2026, https://witness.ai/blog/ai-browser-agent-security-risks/
10. Anyone else noticing agents fall apart the moment they leave clean APIs? - Reddit, accessed March 28, 2026, https://www.reddit.com/r/AI_Agents/comments/1qguqft/anyone_else_noticing_agents_fall_apart_the_moment/
11. AI Browser: Copilot in Edge - Microsoft, accessed March 28, 2026, https://www.microsoft.com/en-us/microsoft-copilot/for-individuals/do-more-with-ai/ai-for-daily-life/ai-browser-innovation-with-copilot-in-edge
12. The Best AI Browsers of 2025 - Navigating the Agentic Shift and the Quest for Trust, accessed March 28, 2026, https://www.sidespace.app/blog/ai-browser-comparison-in-2025
13. Chrome auto-browse: How to use Gemini's new AI agent feature ..., accessed March 28, 2026, https://www.eesel.ai/blog/chrome-auto-browse-how-to-use-geminis-new-ai-agent-feature
14. The Agentic Browser Landscape in 2026: A Complete Guide | No Hacks Podcast, accessed March 28, 2026, https://www.nohackspod.com/blog/agentic-browser-landscape-2026
15. Ask Gemini in Chrome to complete tasks for you with auto browse - Google Help, accessed March 28, 2026, https://support.google.com/chrome/answer/16821166?hl=en
16. Demonstration: Guide Accessifies the Addition of Components to ..., accessed March 28, 2026, https://blindaccessjournal.com/2025/10/demonstration-guide-accessifies-the-addition-of-components-to-salesforce-experience-cloud-site-pages/
17. Leveraging LLMs to Detect and Correct Web Accessibility Violations in HTML Code - arXiv, accessed March 28, 2026, https://arxiv.org/html/2507.19549v1
18. Why Web Agents Fail — and How Semantic Geometry Helps Them Execute - Medium, accessed March 28, 2026, https://medium.com/@rcholic/why-web-agents-fail-and-how-semantic-geometry-helps-them-execute-24d5a6cb950d
19. Release Notes for Microsoft 365 Copilot, accessed March 28, 2026, https://learn.microsoft.com/en-us/copilot/microsoft-365/release-notes
20. Salesforce UI testing challenges: the complete guide to stable automation - Gearset, accessed March 28, 2026, https://gearset.com/blog/salesforce-ui-testing-challenges/
21. Breaking through the Salesforce Shadow DOM - Functionize, accessed March 28, 2026, https://www.functionize.com/blog/breaking-through-the-salesforce-shadowdom
22. rtrvr.ai DOM Intelligence Architecture: Why Screenshots Reduce Performance, accessed March 28, 2026, https://www.rtrvr.ai/blog/dom-intelligence-architecture
23. Agent-Browser: AI-First Browser Automation That Saves 93% of Your Context Window | by Rick Hightower - Medium, accessed March 28, 2026, https://medium.com/@richardhightower/agent-browser-ai-first-browser-automation-that-saves-93-of-your-context-window-7a2c52562f8c
24. Guide - AI assistant for people who are blind or low-vision - AppleVis, accessed March 28, 2026, https://www.applevis.com/forum/windows/guide-ai-assistant-people-who-are-blind-or-low-vision
25. I am among the first people to gain access to OpenAI's “Operator” Agent. Here are my thoughts. : r/ChatGPTPro - Reddit, accessed March 28, 2026, https://www.reddit.com/r/ChatGPTPro/comments/1i8jln3/i_am_among_the_first_people_to_gain_access_to/
26. Agentic Artificial Intelligence (AI): Architectures, Taxonomies, and Evaluation of Large Language Model Agents - arXiv, accessed March 28, 2026, https://arxiv.org/html/2601.12560v1
27. AI Browsers: A Security Nightmare | Flipping the Board on Decades of Security Progress, accessed March 28, 2026, https://www.culture.ai/resources/blog/ai-browsers-a-security-nightmare
28. AI Browser Security Risks: Why Gartner Recommends Blocking Autonomous AI Browsers | CyberMaxx, accessed March 28, 2026, https://www.cybermaxx.com/resources/ai-browser-security-risks-why-gartner-recommends-blocking-autonomous-ai-browsers/
29. AI Agent Architecture: Tutorial and Best Practices, accessed March 28, 2026, https://www.patronus.ai/ai-agent-development/ai-agent-architecture
30. So far all attempts at giving AI agents a browser have failed - but why? : r/AI_Agents - Reddit, accessed March 28, 2026, https://www.reddit.com/r/AI_Agents/comments/1nl6d89/so_far_all_attempts_at_giving_ai_agents_a_browser/
31. Why AI Agents Fail: 3 Failure Modes That Cost You Tokens and Time - DEV Community, accessed March 28, 2026, https://dev.to/aws/why-ai-agents-fail-3-failure-modes-that-cost-you-tokens-and-time-1flb
32. Browse AI - Get Access | Soverin, accessed March 28, 2026, https://soverin.ai/product/browse-ai/
33. 5 Best Web Browsing Tools for AI Agents - 2026 Guide | Fast.io, accessed March 28, 2026, https://fast.io/resources/best-web-browsing-tools-ai-agents/
34. Browserbase vs Multion | Browser Agents Comparison | Respan - Keywords AI, accessed March 28, 2026, https://www.respan.ai/market-map/compare/browserbase-vs-multion
35. The Best AI Agents For 2026: Automate Smarter And Work Faster - Monday.com, accessed March 28, 2026, https://monday.com/blog/ai-agents/best-ai-agents/
36. AI Agent Orchestration Frameworks: Which One Works Best for You? - n8n Blog, accessed March 28, 2026, https://blog.n8n.io/ai-agent-orchestration-frameworks/
37. accessed March 28, 2026, https://www.guideinteraction.com/#:~:text=Guide%20is%20a%20desktop%20application,to%20digital%20tools%20for%20everyone.
38. Guide - AI assistant for people with low vision or blindness, accessed March 28, 2026, https://www.guideinteraction.com/
39. Where AI meets GUI: An overview of computer-using agents | by Anirudh Sundar - Medium, accessed March 28, 2026, https://medium.com/data-science-at-microsoft/where-ai-meets-gui-an-overview-of-computer-using-agents-3085d3bbe332
40. Accessibility | 2025 | The Web Almanac by HTTP Archive, accessed March 28, 2026, https://almanac.httparchive.org/en/2025/accessibility
41. AI accessibility and blind users: a multi-billion dollar market that most AI companies still ignore - Reddit, accessed March 28, 2026, https://www.reddit.com/r/accessibility/comments/1rp0320/ai_accessibility_and_blind_users_a_multibillion/
42. Top Website Accessibility Trends to Watch by the End of 2025 - Brisk Ventures, accessed March 28, 2026, https://briskventures.us/blog/top-website-accessibility-trends-to-watch-by-the-end-of-2025/
43. How to Create a Product Accessibility Case on Salesforce Help, accessed March 28, 2026, https://help.salesforce.com/s/articleView?id=000382999&language=en_US&type=1
44. 6 Accessibility Practices You Need to Know in 2025, accessed March 28, 2026, https://remarkable.global/insights/accessibility-practices-for-2025/
45. What is a DOM, and how does it affect accessibility? - Accessible Web, accessed March 28, 2026, https://accessibleweb.com/question-answer/what-is-a-dom/
46. The Best AI Agents in 2026: Tools, Frameworks, and Platforms Compared | DataCamp, accessed March 28, 2026, https://www.datacamp.com/blog/best-ai-agents