ACENDENCE VII [ GEN I ]
KEYS.AI
AUTONOMOUS AI DESKTOP ASSISTANT
Voice. Vision. Computer control. Autonomous coding.
PROJECT MEMBERS
Harman Sadhwani
Vaishnavi Srivastava
A unified desktop assistant designed to understand, act and build.
PROJECT PRESENTATION
01 / OVERVIEW
Keys.ai at a glance
ACENDENCE VII [ GEN I ]
02
One desktop assistant. Multiple everyday workflows.
Keys.ai brings voice interaction, screen understanding, desktop actions, research and software building into one coordinated application.
40+
tools registered
Lazy-loaded through a central registry
3
build paths
Site • App • Code
4
automation tiers
From UI Automation to mouse/keyboard
2
memory layers
Long-term profile + session context
CORE IDEA
Move beyond a chat-only assistant: connect conversation to verified computer actions and end-to-end project creation.
ACENDENCE VII [ GEN I ]
02
02 / USER EXPERIENCE
From request to verified outcome
ACENDENCE VII [ GEN I ]
03
A practical interaction loop
01
UNDERSTAND
Interpret the user request through chat or live voice.
02
PLAN
Identify tools, requirements and missing details.
03
ACT
Research, create files or control supported desktop apps.
04
VERIFY
Check outputs, run tests and confirm action results.
CONNECTED INPUTS
VOICE
SCREEN
FILES
BROWSER
TOOLS
Outputs can include desktop actions, research findings, generated documents and working software artifacts.
ACENDENCE VII [ GEN I ]
03
03 / WHAT IT CAN DO
A broad capability set
ACENDENCE VII [ GEN I ]
04
LIVE VOICE
Real-time conversation through Gemini Live API, with streaming audio and tool calling.
SCREEN VISION
Capture the screen or webcam, read on-screen text and analyze visual context.
DESKTOP CONTROL
Open apps, interact with windows, type, click, scroll and manage supported settings.
AUTONOMOUS CODING
Crown Code can build, repair, run and verify software projects.
FILE & DOCUMENTS
Create and edit files, including Word, Excel and PowerPoint outputs.
WEB RESEARCH
Search and browse, capture screenshots and collect useful reference material.
MESSAGING
Connect workflows for WhatsApp, Telegram, Gmail and SMS integrations.
SYSTEM & MEMORY
Monitor system resources and retain user preferences and project context.
ACENDENCE VII [ GEN I ]
04
04 / HOW THE SYSTEM IS ORGANIZED
System architecture
ACENDENCE VII [ GEN I ]
05
USER INTERFACE • Qt
Main window • chat • voice controls • capture preview • settings
KEYSAI LIVE ENGINE • main.py
Audio queues • Gemini Live session • tool dispatch • wake word • echo guard
TOOL REGISTRY • core/tool_registry.py
40+ tools • lazy loading • action modules • core tools • plugins
CAPABILITY ENGINES
Voice / Audio • Vision / OCR • Computer Control • Crown Code
MEMORY & STATE
Long-term memory • session summaries • encrypted configuration
CORE TECHNOLOGIES
Python + Qt UI
Gemini Live API
OpenCV + Tesseract
pywinauto / Playwright
PyAutoGUI
Ollama + Node.js
The UI coordinates the live engine; the registry routes model tool calls to focused capability modules.
ACENDENCE VII [ GEN I ]
05
05 / VOICE
Live voice & audio pipeline
ACENDENCE VII [ GEN I ]
06
Designed for natural, low-friction conversation
MICROPHONE
PyAudio capture�16 kHz • mono
AUDIO QUEUE
Chunked async�input streaming
GEMINI LIVE
Native audio�+ tool calling
TTS OUTPUT
24 kHz playback�via async queue
VOICE FEATURES
Wake word
Optional local detection
Echo guard
Prevents self-listening
Barge-in
Voice interruption, opt-in
Phone mic
Pauses PC mic while active
Session resume
Reconnects with session handles
Live model specified in the project documentation: Gemini 2.5 Flash Native Audio preview.
ACENDENCE VII [ GEN I ]
06
06 / PERCEPTION
Screen understanding & vision
ACENDENCE VII [ GEN I ]
07
1
CAPTURE
Full screen • webcam • selected region
2
PRE-PROCESS
PyAutoGUI / OpenCV capture and image handling
3
READ & ANALYZE
Tesseract OCR + Gemini vision analysis
4
RESPOND
Describe, extract or answer using visible context
SUPPORTED CONTEXT
Screen
Read visible application content
Documents
Extract text and inspect information
Camera
Analyze captured webcam frames
Messages
Verify WhatsApp message content
Cooldowns
Control capture frequency
Vision tools include screen capture, camera capture and image-to-text analysis.
ACENDENCE VII [ GEN I ]
07
07 / DESKTOP AUTOMATION
Computer control with layered automation
ACENDENCE VII [ GEN I ]
08
Keys.ai selects an interaction method based on the application and available access.
TIER 1
Windows UI Automation
pywinauto
Read application controls and interact with standard Windows UI.
TIER 2
Native application APIs
Playwright / CDP
Use DOM-level access for browsers and automation-ready apps.
TIER 3
Computer vision + OCR
OpenCV / OCR
Identify visual elements when a native automation API is unavailable.
TIER 4
Mouse & keyboard
PyAutoGUI
Coordinate-based fallback for broad compatibility.
SAFETY & USER CONTROL
Gold control overlay • ESC/STOP • genuine user input auto-pauses • confirmation for high-impact actions
ACENDENCE VII [ GEN I ]
08
08 / SOFTWARE BUILDING
Crown Code: autonomous coding agent
ACENDENCE VII [ GEN I ]
09
The coding workflow is structured to reduce assumptions and verify what it creates.
01
REQUIREMENTS GATE
Analyze brief; ask for missing details; avoid building on assumptions.
02
WORKSPACE SETUP
Create project directory and save a structured brief.
03
BUILD TARGET
Choose Site, App or general Code path.
04
RESEARCH & BLUEPRINT
Collect references/assets and synthesize the implementation plan.
05
AGENT LOOP
Read, write, edit, run commands and inspect results.
06
VERIFY & DELIVER
Run checks, repair failures and open the completed artifact.
Crown Code bridge: Python ↔ Node.js headless agent ↔ local Ollama model or cloud API.
ACENDENCE VII [ GEN I ]
09
09 / BUILD & QUALITY
Three build paths, one verification mindset
ACENDENCE VII [ GEN I ]
10
SITE PATH
Websites • portfolios • landing pages
Plan → Render → Verify
Site builder renders the plan and checks the result in a real browser.
APP PATH
Dashboards • desktop apps • utilities
Recipe → Compile → Test
Recipe-based build flow; checks compilation, runtime and endpoints.
CODE PATH
General coding • tools • complex projects
Read → Write → Run → Fix
Agent loop backed by a local or cloud model, with repair passes.
QUALITY GATES
Project tests
Run pytest when test files exist; up to 2 repair attempts.
Design checks
Contrast, visible copy, broken images, font size and spacing.
Runtime checks
Launch output, inspect console errors and verify images.
ACENDENCE VII [ GEN I ]
10
10 / TOOLING & INTEGRATIONS
A modular tool ecosystem
ACENDENCE VII [ GEN I ]
11
DESKTOP & FILES
open_app
computer_control
computer_use
editor_control
file_processor
computer_settings
WEB & COMMUNICATION
web_search
browser_control
research_browser
send_message
send_email
whatsapp_control
DEVELOPMENT
crown_code
code_helper
repo_inspect
site_builder
app_builder
game_builder
SYSTEM & UTILITIES
system_status
weather_report
reminder
youtube_video
flight_finder
globe
The registry centralizes dispatch and loads tool modules lazily; plugins can extend the system.
ACENDENCE VII [ GEN I ]
11
11 / PERSONALIZATION
Memory, plugins & configuration
ACENDENCE VII [ GEN I ]
12
LONG-TERM MEMORY
• Saves user facts, preferences and project context��• Searchable by keyword��• Auto-trims older entries��• Stored in JSON files
SESSION MEMORY
• Tracks the active conversation��• Generates a summary at session end��• In-memory only��• Starts fresh on each launch
CONFIG & PLUGINS
• Encrypted secrets store for API keys and settings��• Drop-in Python plugin modules��• Plugin discovery at startup��• Fallback dispatch for extra tools
CONFIGURATION NOTE
Provider keys, model choices, voice settings and device configuration are managed through dedicated settings/config files.
ACENDENCE VII [ GEN I ]
12
12 / ENGINEERING FOUNDATION
Technology stack & code organization
ACENDENCE VII [ GEN I ]
13
APPLICATION
Python 3.12+ • Qt interface • async audio queues
AI & MODELS
Gemini Live API • Ollama local models • cloud model support
AUTOMATION
pywinauto • Playwright/CDP • OpenCV • Tesseract • PyAutoGUI
CODING AGENT
Python bridge • Node.js headless agent • terminal execution
DATA & SERVICES
JSON memory • encrypted config • embedded Chromium research browser
PROJECT MODULES
main.py / ui.py
actions/ — action tools
core/ — shared services
agent_control/
plugins/
memory/
dashboard/ • globe/ • tests/
The modular layout separates UI orchestration, tool implementations, shared services and agent control.
ACENDENCE VII [ GEN I ]
13
13 / GETTING STARTED
Setup & development workflow
ACENDENCE VII [ GEN I ]
14
PREREQUISITES
Python 3.12+
Ollama for local models
Git for Crown Code
Project Python dependencies
Gemini API key for live voice
Example local models in the documentation:
qwen3:8b • qwen3-coder:30b
CONFIGURE & RUN
1
Prepare Crown Code
Clone/set up the agent in its configured directory.
2
Configure providers
Add Gemini and local/cloud provider settings.
3
Install a model
Pull a supported Ollama model when local inference is needed.
4
Launch Keys.ai
Run the application entry point: python main.py.
5
Test the system
Run pytest and Crown Code integration checks.
ACENDENCE VII [ GEN I ]
14
KEYS.AI
AUTONOMOUS AI DESKTOP ASSISTANT
PROJECT MEMBERS
Harman Sadhwani
Vaishnavi Srivastava
Thank you
ACENDENCE VII [ GEN I ]