1 of 43

Running Hermes in production

2 of 43

$ whoami

  • Saurabh Hirani
  • Principal SRE at One2N
  • Infrastructure and reliability engineering
  • Reading and wordplay
  • Joke pronouns: Sed / Awk
  • Sed-istic and Awk-ward

3 of 43

What's init for you?

  • Learnings from running Hermes in prod for 3 months
  • Best practices - apply to other agents

4 of 43

SRE day-to-day tasks

  • Creating / updating infra for services
  • Triaging incidents
  • Adhoc scripts for automation
  • Building / maintaining CI/CD workflows for deployment

5 of 43

Scattered automation

  • Scripts, CI/CD == predictable work
  • Repetitive investigations that deviate from runbooks
  • Operational requests: restart a service, provide logs, etc.
  • SRE responsibility == anything that is not product development

6 of 43

Plan to fix

  • WIP roadmap for deterministic automation
  • Operational reasoning requires context
  • Scattered harness usage to do the following
    • Investigating incidents
    • Intent driven runbooks and infra
    • Onboarding/discovery queries

7 of 43

Orchestration layer

with a human interface

8 of 43

Enter Hermes

  • Interpret requests, discover context, invoke tools
  • Interface == Slack
  • Conversational approach to
    • Investigating incidents
    • Intent driven runbooks and infra
    • Onboarding/discovery queries

9 of 43

It works on my machine

10 of 43

11 of 43

Production setup

12 of 43

13 of 43

Control self improvement

14 of 43

What is skillable?

  • Recurring
  • Deterministic
  • Bounded
  • Known context
  • Example: Kafka connector restart

15 of 43

What is not skillable?

  • One-off
  • Destructive
  • Masks the real fix
  • Success != reuse
  • Example: Prometheus WAL issue

16 of 43

Update skills

17 of 43

18 of 43

Absorb skills

19 of 43

20 of 43

Dogfooding: Infra/CI Gitops

21 of 43

22 of 43

Patterns: IaC vs non-IaC infra

23 of 43

IaC infra

  • The agent got a clean house
  • The context is already there

24 of 43

Non-IaC infra

  • Clickops setup
  • Agent discovered the same context with every request
  • Pinch of higher cost per solution for non-IaC infra
  • Turn manual ops into code / pipelines / templates first
  • Groundwork > Model
  • Less magic, less cost

25 of 43

Usage patterns emerged

26 of 43

27 of 43

More Tools, More Context, More Problems

  • More tools
    • Expand capability but increase cost and security issues
    • Audit what is needed

28 of 43

Before: Every tool reaches the model

29 of 43

30 of 43

After: Selective view

31 of 43

32 of 43

Managing secrets

  • Before Bifrost
    • Init container injects secrets into Hermes pod
    • Hermes invokes MCPs
  • After Bifrost
    • Bifrost manages secrets
    • Hermes goes through an auditable route

33 of 43

Model selection

34 of 43

Model

Performance

Verdict

Claude Sonnet 4.6

Worked for all requests

Cost too much for routine requests

DeepSeek 3.2

Worked for simple requests

Fell short on some multi-turn investigations

Kimi 2.5

Sweet spot

Cost / capability balance

35 of 43

Memory

  • Design decision - Limit memory, expand skills
  • Know what is global
    • Repo list
    • Organizational communication style
    • Production workflow
  • Runbooks == skills, not growing memory

36 of 43

Scaling Hermes horizontally

  • Multiple Hermes -> update skills in same repo
  • Challenges
    • Duplicate PRs
    • Conflicting PRs
    • Review load

37 of 43

Monitoring Hermes

  • Basic K8s monitoring
  • OTEL integration through community plugin
  • Model cost and reliability monitored through Bifrost
  • Bifrost advantage - model fallback in case of failure

38 of 43

Tying it all together

39 of 43

40 of 43

Bring it to your machine!

  • Pending - open source repo - reflects most of the setup

41 of 43

Was it worth it?

  • Capacity: 100-120 users in a slack channel
  • Performance: 20 parallel sessions before queue up
  • BAU Cost: ~70 USD / month LLM API cost
  • Incident cost: ~1-4 USD / typical investigation
  • Savings: ~40-44 engineering hours saved / month

42 of 43

Conclusion

  • Build reliable systems around a probabilistic one
  • Control the learning loop
  • Gitops and AI Gateways help

43 of 43

Twitter - @sphirani

LinkedIn - linkedin.com/in/shirani/

Repo - Pending

​

Questions?