Lessons Learned from a

Multi-Model Question

Answering Pipeline

on the Chameleon Testbed

Deshan Wattegama · Nagarjuna Kandimalla · Katelyn Van Dyke · Tanu Malik, PhD

University of Missouri | CHI-251508 | Sixth Chameleon User Meeting — April 15, 2026

Project Overview

The Challenge

No single LLM consistently yields the best answer across diverse question types, domains, or latency constraints.

​

Goal:

Investigate data-centric model routing — dynamically select the optimal model per query.

​

Context:

Semester-long graduate course: Designing End-to- End ML Systems at University of Missouri.

What We Built

• End-to-end containerized QA pipeline
• Embedding-based retrieval & context ranking (ChromaDB)
• Three small LLMs: LLaMA 3 8B, Mistral 7B, Qwen 2.5B
• Three routing modes: manual rules, dataset-aware heuristics, ML classifier
• Evaluated 30,000 questions across HotpotQA, NarrativeQA, PubMedQA

1

Project Formulation

Milestone 1 :

RAG-based Q&A Pipeline

• Generate embeddings for context
• Store embeddings in ChromaDB vector store
• Embed user query
• Retrieve top relevant chunks
• Generate and evaluate answers

​

Milestone 2 :

Containerized RAG-based Q&A Pipeline

• Utilizing a Model Storage (Minio)
• Containerized, Models and other components to independent containers.
• Orchestrate the system as an interconnected application via docker-compose feature

​

2

Pipeline Architecture

Docker on Chameleon

Infrastructure: Docker orchestration · Docker containers · MinIO (S3) artifacts · JupyterLab dev environment · Ceph-backed Block storage

Three Routing Modes:

Manual Rules

Fixed routing policy

Dataset Heuristics

If-then rules per dataset

ML Router

Logistic regression classifier

Milestone 3 :

3

Initial Results

Training Performance – Dataset Based Routing

​

Testing Performance – Dataset Based Routing

​

4

Initial Results

ML-Based Routing: Training Overview

Utility-based classifier to select best model per question

TRAINING SETUP

Dataset | Prompt characters | Context characters

TARGET LABELS

Training Performance – Dataset Based Routing

​

5

Initial Results

Testing Performance – ML Based Routing

​

Performance Comparison – Dataset Based vs ML Based

​

• ML based routing policy showed similar results for hotpotqa and pubmedqa datasets
• Dataset routing policy did well for narrativeqa dataset

6

Dataset-Dependent Performance

No single model dominated. Optimal model selection varied strongly by dataset type.

Dataset-Aware Routing Wins

Simple heuristic routing matched or beat static selection. Qwen 2.5B best on HotpotQA (0.66), Mistral best on NarrativeQA & PubMedQA.

Conclusions from the project

Quality–Latency Trade-off

Longer context → better answers but higher latency. NarrativeQA latency doubled (~250ms → ~500ms) with increased context.

ML Router Needs Richer Features

Logistic regression on (dataset, prompt length, context length) achieved ~0.46 balanced accuracy — semantic embeddings needed for improvement.

7

Hardware Configuration

Instance type: g1.h100.pci.1

GPU:

1× NVIDIA H100 (94 GB) via PCI passthrough

vCPUs:

48 virtual CPUs

Memory:

~250 GB system RAM

Storage:

Ceph-backed SSD, 150 GiB (expandable on demand)

Networking:

Docker network + public port access

Artifacts:

MinIO S3-compatible object storage (boto3)

Why Chameleon Made This Possible

Rapid GPU Access:

Reserved H100s within hours vs. long queues on Hellbender or Nautilus clusters

Full Root Control:

Installed Kubernetes, Ollama, MinIO, custom Python deps — no ticket required

Elastic Storage:

Expanded Ceph volume on demand when ChromaDB index outgrew initial allocation — zero downtime

Long Reservations:

Held GPUs for days, enabling complete 30K-question evaluation runs without interruption

Public Port Access:

Deployed live routing UI accessible externally for demo & latency benchmarking

Resource Usage

8

Honest Account of What Didn't Work

Challenges & Insights

Challenge: Trovi SSH key conflict

Trovi artifacts auto-inject a new SSH key per instance. Students using personal key pairs could not SSH in — their keys were not recognized, stalling onboarding.

Resolution: Chameleon web UI

Switched to the web UI for reservations — accepted existing key pairs, select instance types, attach volumes, and launch experiments within minutes.

Rapid H100 GPU access

Reserved H100s faster than Hellbender or Nautilus. Held GPUs for days vs. Nautilus time-limited slots — essential for 30,000-question evaluation runs.

Full root access — no ticket queue

Installed Kubernetes, Ollama, MinIO instantly. On Hellbender, custom packages or storage increases required submitting tickets with long wait times.

On-demand Ceph volume expansion

When the vector index outgrew initial allocation, we expanded Ceph volumes via dashboard in minutes, restarted ChromaDB, and continued without downtime.

Self-managed Docker & public ports

Deployed our own containers, opened ports for public access, and benchmarked routing latency with a live UI — impossible on shared HPC systems with locked orchestration layers.

PLATFORM COMPARISON

Chameleon Cloud — chameleoncloud.org | Hellbender HPC — University of Missouri | Nautilus — National Research Platform

9

What Worked Well

• H100 GPU access — rapid provisioning, multi-day reservations
• Root access: Kubernetes, Ollama, MinIO installed without barriers
• Ceph volume expansion on-demand without service disruption
• Containerization enabled full reproducibility across team members
• Public port access enabled live UI + latency benchmarking
• Chameleon environment closely mirrors real production cloud setups

What We Wish Existed

• Quick Start templates for multi-container ML deployments (Kubernetes + GPU)
• Clearer Trovi keypair handling issue
• Pre-built Dockerfile templates for common AI stacks (Ollama + FastAPI + ChromaDB)
• Community forum or mentor matching for infrastructure troubleshooting
• Storage quota visibility before hitting limits

Infrastructure Needs & Community Insights

10

Multi-model QA workloads impose heterogeneous, dynamic infrastructure demands not well matched to traditional HPC assumptions.

Chameleon provided essential GPU accessibility, software freedom, and elastic storage for large-scale routing experiments in a graduate classroom.

Data-centric model routing is promising but requires richer features beyond simple metadata for competitive ML-based selection.

Full administrative control and long GPU reservations were critical differentiators vs. Hellbender and Nautilus.

Better onboarding templates and AI-aware monitoring would significantly improve the Chameleon experience for ML education.

Conclusions

1

2

3

4

5

11

Thank You