AMD Developer Hackathon

Team Name: TICK TOCK

Problem Statement

Why naive LLM usage doesn't scale

Expensive LLM Calls

Every prompt unnecessarily hits a premium mode leading to massive token usage.

Cloud Dependency

High latency and recurring API costs.

No Confidence Awareness

Models regenerate instead of verifying.

High Cost

Token Waste

Poor Scalability

BROKEN PIPELINE

Query

LLM (always)

Response

02

Solution

​

HybridRouter is a multi-tier task-solving agent that escalates spend only when it has to.

Every incoming task flows through progressively more expensive tiers, gated by confidence:

• Tier 0 : Deterministic solvers (math, parsing, regex): zero tokens, instant, 100% confidence when they apply.
• Tier 1 : Local model (Gemma 3 12B, running on-container via ROCm): zero Fireworks tokens. Produces both an answer and a confidence score.
• Confidence gate: per-category thresholds decide what happens next and accept the local answer outright, cheaply verify it with a single-token Fireworks call, or escalate.
• Tier 2/3 : Fireworks cloud: only reached when local confidence is genuinely too low. Tier 2 uses a cheap model per category; Tier 3 (a stronger model) is the last resort, reserved for the hardest or most uncertain tasks.

​

Architecture Diagram

AI Pipeline

• Input: each task is classified into one of eight categories (math, code, factual, logic, etc.) before any solving begins
• Deterministic solvers attempt first, code-based math and parsing solvers run at perfect accuracy and zero tokens whenever the task allows it
• Unsolved tasks go to the local AMD GPU model, confidence is scored via self-consistency sampling: the model generates multiple independent samples per task, and agreement across samples becomes the confidence score
• Low-confidence results escalate to Fireworks, the cheapest sufficient model is tried first per category; the strongest model is reserved only for high-risk, low-confidence cases
• Every result is logged with full traceability, tier used, model, tokens spent, confidence, and escalation reason, so every answer is auditable end to end
• Answer caching (hash-based dedup) avoids re-processing exact repeated tasks, at zero additional token cost

Five services, one Docker Compose stack

Technology Used

AMD Integration

• The local inference tier : the tier that absorbs the majority of traffic at zero token cost runs entirely on AMD ROCm, using the rocm/pytorch base image for GPU-accelerated inference.
• HSA_OVERRIDE_GFX_VERSION is configurable per target GPU architecture, making the local model server portable across different AMD GPU generations without code changes.
• Because local inference is unlimited and free under the competition's scoring rules, maximizing the local (AMD-GPU) tier's hit rate is directly what makes the system cheap, this isn't a side integration, it's the core lever the whole architecture is built around.
• The confidence-gating system exists specifically to push as much accurate work as possible onto the AMD-hosted model before ever reaching a paid Fireworks token.

Features

• Zero-cost tiers first: cache → deterministic code solvers → local AMD-hosted LLM, all before a single paid token is spent.
• Category-aware confidence gating: eight task categories, each with tuned high/medium/low thresholds and its own confidence-weighting formula (consistency, answer length, hedging language).
• Smart model selection per category: cheap and strong Fireworks models are chosen per task category, not globally, e.g. math/code/logic escalate differently than factual/creative tasks.
• Fail-forward reliability: rate-limit retries, automatic fallback to a stronger model on failure, and a hard-fallback path so the system never crashes or returns nothing.
• Full observability: every task's tier, model, token spend, confidence, latency, and escalation reason is logged for analysis and threshold tuning.
• Built-in threshold tuning: an evaluation harness sweeps different threshold configurations and reports accuracy vs. cost tradeoffs, so thresholds are chosen empirically, not guessed.
• Live dashboard: React front-end visualizing token spend and accuracy over time.

Process Flow Diagram

• Steps 1 to 4 run entirely on free tiers: cache, classification, deterministic solvers, and the local AMD-hosted model.
• Steps 5 & 6 are the only paid tiers, reached only when local confidence genuinely falls short.
• Step 7 logs every decision tier, tokens, latency, confidence so the routing behavior is fully auditable, not a black box.

​

Demo Screenshots

Demo Screenshots

Demo Screenshots

Impact and Market

From our own threshold-tuning sweep (eval/tune_thresholds.js), the best-performing configuration achieved:

• 83.3% accuracy while cutting cost to roughly 13.7% of naive-baseline spend (an ~86% cost saving vs. always calling a strong cloud model).
• 50% of tasks resolved locally (zero Fireworks tokens) with only 33% escalated to cloud models.
• Sub-1.3s average latency, with most of that time in the local-model tier rather than network round-trips to the cloud.
• Local AMD GPU inference reduces dependency on paid cloud APIs, lowering operating cost at scale
• Scoring-formula alignment: Score = Accuracy − Token Penalty; system is purpose-built to maximize this exact tradeoff

​

THANK YOU

https://github.com/Tetra4ge/HybridRouter