ENTERPRISE GENAI MASTERCLASS

The Enterprise GenAI Playbook

From Prototype to Production

How enterprises bridge the gap between 'AI initiative' and 'meaningful EBIT impact.'

AD

Ayush Dwivedi

Senior Manager, Data Science

Walmart Global Tech

Linkedin : https://www.linkedin.com/in/ayush--dwivedi/

Twitter/X : https://x.com/ayush_dw

Email - dwivedi.r.ayush@gmail.com

ACT 1 · CONTEXT — 2 OF 24

WHAT IS GENERATIVE AI

Traditional AI

PREDICTS

Classifies · Labels · Ranks

Fixed output space · Testable

Generative AI

CREATES

Novel output · Open-ended

Probabilistic · Emergent

→

Text

Reports, emails

Code

Functions, apps

Images

Designs

Audio/Video

Synthesis

Decisions

Routing, approvals

Key shift:

Closed, finite output space → Open, generative output space. This makes GenAI powerful and harder to govern.

2 / 24

ACT 1 · CONTEXT — 3 OF 24

THE HONEST TIMELINE

The transformer is 8 years old.

ChatGPT is 3½ years old.

2017

Transformers

Paper

2020

GPT-3

2K tokens

Jan 2022

InstructGPT

(RLHF)

Oct 2022

LangChain

Launches

Nov 2022

ChatGPT

100M in 2mo

2023-24

GPT-4 / Claude

Agents era

RLHF was the breakthrough

InstructGPT (Jan 2022) made models follow instructions. Not scale — alignment training.

Nov 2022 = accessibility, not technology

A chat box on an existing model. 1M users in 5 days. 100M in 2 months.

Tooling arrived simultaneously

LangChain (Oct) + LlamaIndex (Nov) made RAG systems buildable in days.

3 / 24

ACT 1 · CONTEXT — 4 OF 24

THE 100 MILLION USER RACE

2 months. Every other technology

took years.

ChatGPT

2 months ← unprecedented

TikTok

9 months

Instagram

~2.5 years

Facebook

~4.5 years

Spotify

~7 years

Netflix

~10 years

This explains every board-level AI mandate — and why operational readiness always lags adoption.

4 / 24

ACT 1 · CONTEXT — 5 OF 24

CURRENT ENTERPRISE STATE — CITED STATISTICS

65% use GenAI. Only 6% generate

meaningful business impact.

65%

regularly use GenAI in ≥1 function

(up from 33% in 2023)

McKinsey, early 2024 (n=1,363)

72%

use AI in ≥1 area of operations

McKinsey Global AI Survey 2024

~6%

generate >5% EBIT impact from AI

(the 'high performer' group)

McKinsey State of AI 2025

50%

of GenAI POCs abandoned by end 2025

Gartner, July 2024 (updated 2025)

48%

of AI projects reach production

(avg 8 months POC → prod)

S&P Global Market Intelligence 2025

60%

of projects lacking AI-ready data

will be abandoned by 2026

Gartner, Feb 2025

5 / 24

ACT 1 · CONTEXT — 6 OF 24

WHY 2022 — FOUR SPECIFIC CONVERGING FORCES

None was new. All four reached

a threshold simultaneously.

01 RLHF made models instruction-following

• InstructGPT, Jan 2022 (Ouyang et al.)

• Trained on 13K instruction pairs + human pref

• Before: GPT-3 completed prompts unpredictably

• After: reliable instruction-following → ChatGPT

02 Context windows crossed enterprise thresholds

• GPT-3 (2020): 2,048 tokens

• GPT-4 (2023): 8K–32K tokens

• GPT-4 Turbo (2023): 128K tokens

• Gemini 1.5 (2024): 1,000,000 tokens

03 Production RAG tooling appeared in 8 weeks

• LangChain: Oct 2022 (1M downloads by Jan 2023)

• LlamaIndex: Nov 2022

• Pinecone, Weaviate, Chroma scaled commercially

• 'Build a RAG system': 3 months → 2 weeks

04 Capital at infrastructure scale

• Microsoft: $10B to OpenAI (Jan 2023)

• Google: $300M → $2B to Anthropic

• Amazon: $4B to Anthropic (2023-24)

• H100 GPU production scaled globally

6 / 24

ACT 2 · CONCEPTS — SLIDE 7 OF 24

THE AI LANDSCAPE — WHERE GENAI FITS

Not all AI is GenAI. Not all GenAI is the same. Here is the precise taxonomy.

Traditional ML

Classifies, predicts, labels. Fixed output space.

Spam filters, fraud detection, demand forecasting

Classic NLP

NER, sentiment, intent. Task-specific models.

BERT, fine-tuned classifiers, intent detection

Foundation Models

Pretrained on trillions of tokens. Generalizes to new tasks without task-specific training.

GPT, Claude, Gemini, Llama, Mistral

Multimodal Models

Text + image + audio + video in one unified model. Transformer architecture is modality-agnostic.

GPT-4V, Claude 3.5, Gemini 1.5 Pro

GenAI / LLMs

Generate novel content.

Reason, plan, synthesize.

Use tools. Take actions.

Probabilistic.

Emergent.

Open-ended output.

GPT-4o, Claude 3.5+,

Gemini 1.5, Llama 3.1

7 / 24

ACT 2 · CONCEPTS — 8 OF 24

THE MULTIMODALITY BREAKTHROUGH

One transformer handles all modalities.

This was not obvious. Now it's everywhere.

BEFORE: One model per modality

ResNet / EfficientNet

Images only

BERT / RoBERTa

Text only

Whisper / wav2vec

Audio only

LayoutLM

Documents only

Separate pipeline

Each use case

→

AFTER: Tokenize anything → one transformer

Text

BPE tokens (GPT tokenizer)

Images

16×16 patch tokens (ViT, 2020)

Audio

Spectrogram patch tokens

Documents

Layout + text tokens

One model

Shared transformer weights

Breakthrough chain

• 2020: ViT — images as transformer patches

• 2021: CLIP — joint text-image embeddings

• 2022: Flamingo — few-shot visual reasoning

• 2023: GPT-4V — deployed at scale

• 2024: GPT-4o / Gemini 1.5 — omni-modal

Enterprise use cases unlocked

• Insurance: form scan + photos + notes → decision

• Invoices: PDF layout + table + fields → ERP

• Compliance: contract image + tables → extract

• Medical: imaging + radiology report → summary

8 / 24

ACT 2 · CONCEPTS — SLIDE 9 OF 24

FOUR PROPERTIES THAT INVALIDATE YOUR EXISTING ENGINEERING PLAYBOOK

Every operational challenge in GenAI traces back to one of these four properties.

∼

1. Probabilistic outputs

Same input ≠ same output. The model samples from a probability distribution at inference time. Temperature controls this distribution. By design — not a bug.

Enterprise implication: Breaks every assert(output==expected) test. Requires rubric-based evaluation over output distributions, not exact-match testing. Your QA team needs a new methodology.

✦

2. Emergent capabilities

Capabilities appearing at scale not explicitly trained for: multi-step reasoning, in-context learning, code generation. These emerged unexpectedly from scale alone.

Enterprise implication: The failure surface is open-ended. You cannot enumerate all possible failure modes before deployment. Continuous production monitoring is mandatory, not optional.

⊡

3. Context sensitivity

System prompt + conversation history + retrieved documents + user query together determine the output distribution. One sentence change in a system prompt can shift behavior across the entire output space.

Enterprise implication: Prompts are code with non-local effects. They require version control, peer review, regression testing. 'System prompt in a shared Google doc' is a production incident waiting to happen.

⇄

4. Open-ended output space

Unlike classifiers (label from finite set) or regression (a number), LLMs generate arbitrary text. There is no valid-output enum. Output validation must be semantic, not structural.

Enterprise implication: Validation must ask: 'does this satisfy the intent and grounded in source?' — not 'is this value in the valid set?' This requires a fundamentally different validation architecture.

9 / 24

ACT 2 · CONCEPTS — 10 OF 24

CORE DEPLOYMENT PATTERNS

Exhaust prompt engineering first.

Most teams jump straight to fine-tuning.

Prompt Engineering

First. Always.

System prompt + few-shot examples · 70%+ of tasks · No training cost

RAG

For private/fresh knowledge

Retrieve docs → inject context → generate · Required for proprietary data

Fine-tuning

⚠ Last resort, style only

LoRA/QLoRA on labeled examples · NOT for knowledge · Expensive, brittle

Tool Use

Foundation of agents

Model outputs JSON → runtime executes tool → model sees result

Structured Output

System integration

JSON mode / schema constraint · Validates format, NOT semantics

Guardrails

Non-negotiable

Input: PII detection, injection filter · Output: policy check, hallucination

Correct sequence: Prompt Engineering → RAG (if knowledge gap) → Fine-tuning (last resort, style only). Most enterprise use cases never reach step 3.

10 / 24

ACT 2 · CONCEPTS — 11 OF 24

AGENTS & AGENTIC SYSTEMS

An agent takes actions that change the world.

The R^N math is brutal.

Single-turn LLM

not an agent

Query → response. No memory, no action.

Orchestrated workflow

proto-agent

LLM decides branches. Bounded actions.

ReAct / tool-using

agent

Reasons + acts in loop. Real tool calls.

Multi-agent system

agentic

Multiple agents collaborating. Complex.

R^N Reliability — why agent length matters

R = per-step reliability. End-to-end = R^N

Per-step R

3 steps

5 steps

7 steps

10 steps

R = 0.99

97%

95.1%

93.2%

90.4%

R = 0.95

85.7%

77.4%

69.8%

59.9%

R = 0.90

72.9%

59%

47.8%

34.9%

R = 0.85

61.4%

44.4%

32.1%

19.7%

At R=0.95 → 10-step agent succeeds only 59.9% of the time.

→ Design for bounded autonomy. Every verification node reduces N.

11 / 24

An agent = LLM + memory + tools + planning loop that produces outcomes, not just outputs.

ACT 3 · THE MAP — 12 OF 24

THE FULL TERRITORY — READ THIS BEFORE WE DIVE IN

"Prototype is easy. Production is systems engineering."

WHY PROTOTYPES FAIL

• Hallucination / confabulation

• Prompt brittleness

• Cost explosion at scale

• No evaluation harness

• Governance gaps

ARCHITECTURE EVOLUTION

• Stage 1: Prompt App

• Stage 2: RAG System

• Stage 3: Tool Calling

• Stage 4: Agent Workflows

• Stage 5: Enterprise Platform ←

ENTERPRISE PILLARS

• Reliability + guardrails

• Semantic observability

• Evaluation-first dev

• Governance + audit

• Cost routing + flywheel

"The challenge is not generating text anymore. It's building reliable systems around probabilistic models."

12 / 24

ACT 4 · START WITH PAIN — 13 OF 24

WHY PROTOTYPES FAIL IN PRODUCTION

96% accuracy

= 40,000 broken / day

at 1M queries / day scale [illustrative math]

Evidence

• Gartner 2024: only 48% of AI projects reach production

• Gartner 2025: ≥50% of GenAI POCs abandoned

• McKinsey 2024: 65% use GenAI, only ~6% generate >5% EBIT

• Vectara 2024: 3–27% hallucination rate in RAG systems

• Liu et al. 2023: 'Lost in the Middle' context degradation

Hallucination

Model generates confident wrong output.

NOT 'data drift' — LLMs predict plausible tokens,

not verified facts.

Prompt brittleness

Small input change → large output shift.

Semantically equivalent queries score

very differently.

Calibration failure

High expressed confidence ≠ high accuracy.

LLMs are poorly calibrated.

(HELM, Stanford CRFM 2023)

Lost in the Middle

Docs in middle of long context ignored.

Position matters, not just relevance.

(Liu et al. 2023)

13 / 24

ACT 4 · FAILURE MODES 1–5 — 14 OF 24

SCORE YOUR PROJECT — HOW MANY DO YOU HAVE?

Nine Failure Modes (1 of 2)

01

CRITICAL

Eval debt - built before measuring

​

Diagnostic: How do you know last prompt change helped?

Monday: Curate 100 golden queries

02

CRITICAL

Distribution shift

Diagnostic: Did you test on your imagination or your users?

Monday: Sample 500 real production queries

03

CRITICAL

RAG illusion

Diagnostic: What is your retrieval precision@3?

Monday: Measure retrieval separately

04

SERIOUS

The cost cliff

Diagnostic: Have you run the scale math?

Monday: Instrument per-query tokens

05

SERIOUS

Governance vacuum

Diagnostic: Can you answer all 5 governance questions?

Monday: Name a single owner now

14 / 24

ACT 4 · FAILURE MODES 6–9 — 15 OF 24

SCORE CHECK AT THE END

Nine Failure Modes (2 of 2)

06

SERIOUS

Hidden state accumulation- silent propagation through agentic pipeline

​

Diagnostic: Does a wrong intermediate output halt or propagate?

Monday: Define verification predicates per step

07

SERIOUS

The integration canyon - AI done in week 2, plumbing takes 6 months

Diagnostic: Is production data accessible from day 1?

Monday: Map all data sources before coding

08

SERIOUS

Model compatibility drift - provider update breaks behavior silently

Diagnostic: Do you run eval on a schedule, not just deploys?

Monday: Daily golden set cron job

09

SERIOUS

Multi-agent whack-a-mole :Fix one agent, break three others

Diagnostic: Are inter-agent message schemas versioned?

Monday: Define typed contracts per agent

Having all 9 is normal for a POC. Shipping to production with all 9 unresolved is the problem.

15 / 24

ACT 4 · SHOW EVOLUTION — 16 OF 24

ARCHITECTURE EVOLUTION — 5 STAGES

Most enterprises are stuck at Stage 2–3.

Stages are not skippable.

1

Prompt App

LLM + system prompt · Any dev · Days

Breaks at: private knowledge

→

2

RAG System

LLM + vector DB + retrieval · ML eng · Wks

Breaks at: staleness, chunk size

→

3

Tool Calling

LLM + function calling · Backend+ML · Mos

Breaks at: tool reliability, auth

→

4

Agents

ReAct loop + memory + planner · AI team

Breaks at: R^N, hidden state

→

5

Enterprise

Platform

Gateway+eval+obs+governance · 6-12mo

Quality compounds. Target state.

Most common mistake: Jumping Stage 1 → Stage 4. Stages 2 and 3 build the reliability infrastructure Stage 4 depends on.

16 / 24

ACT 4 · GO DEEP — 17 OF 24

THE PRODUCTION GENAI STACK — REQUEST FLOWS TOP TO BOTTOM

7 layers. Clear responsibilities.

Request in at top, response out at bottom.

1. Client & Ingestion

REST / gRPC / WebSocket

Kafka / SQS async queue

Webhook / file upload

Rate-limited per client

2. Security & Compliance

OAuth2 / JWT / RBAC

PII detect → redact before LLM

Prompt injection classifier

Immutable audit log

MOST SKIPPED

3. Orchestration Runtime

LangGraph / Temporal DAG

Tool dispatcher

Checkpoint state

Max-iteration guard

4. Context & Memory

Versioned prompt registry

Token-budget sliding window

RAG: hybrid search → reranker

Long-term memory store

MOST SKIPPED

5. LLM Inference & Routing

Complexity classifier → model select

Semantic cache (before API call)

Provider fallback chain

Structured output parser

6. Tools & Knowledge

Vector DB (Pinecone/Weaviate/pgvector)

Redis session/state cache

External APIs + code sandbox (e2b)

Circuit breaker per tool

7. Observability (all layers)

OpenTelemetry → Langfuse/LangSmith

Faithfulness + hallucination rate

Cost attribution per user/team

Continuous eval pipeline

MOST SKIPPED

17 / 24

ACT 4 · GO DEEP — 18 OF 24

EVALUATION-FIRST DEVELOPMENT

Build the eval harness before

you build the application.

1

Golden Dataset

• 200–500 queries · Stratified

• Hard-weighted (easy tests tell you nothing)

• Sample from real production distribution

• Built BEFORE writing the application

2

LLM-as-Judge

• G-Eval rubrics (not 'rate 1-5')

• Calibrate vs domain experts (target r > 0.75)

• RAGAS: Faithfulness · Context Precision

• Known bias: verbosity, position, self-enhancement

3

Regression Gate

• Every deploy runs full eval suite

• Blocks regressions on primary dimensions

• Prompts = versioned API contracts

• 15 min to catch here vs 3-day war room in prod

4

Online Eval Pipeline

• Sample 1–5% production traffic async

• Weight toward low-confidence + re-asks

• Feed failures back to golden set

• After 12 months: 40–60% quality improvement

Frameworks: RAGAS · G-Eval / MT-Bench · HELM (Stanford) · TruthfulQA · EleutherAI Eval Harness · Vectara Hallucination Leaderboard

18 / 24

ACT 4 · FAILURE PATTERNS — 19 OF 24

FAILURE TAXONOMY — CONVERSATIONAL VS. NON-CONVERSATIONAL AGENTIC

Different failure modes require

different mitigations.

CONVERSATIONAL AI

NON-CONVERSATIONAL / AGENTIC

COMMON ACROSS BOTH

Sycophancy

Agrees with incorrect user assertions

Persona drift

System prompt eroded by accumulated turns

Context overflow

Token limit → model forgets earlier turns

Hallucination QA

Confident wrong answers near training cutoff

Doc misclassification → schema contamination

Invoice classified as PO → wrong extraction → ERP corrupted. Silent.

Extraction hallucination

Missing field → LLM fabricates value. Schema validates. Data wrong.

Router misrouting

Wrong agent dispatched → irreversible action taken.

Tool call parameter hallucination

Valid JSON, wrong params. API executes incorrect action silently.

Model version drift

Provider updates silently. Behavior shifts without deploy.

✏ Your Walmart story here

Prompt injection

Any retrieved text is a potential attack surface.

✏ Your Walmart story here

Conversational: affects user trust. Non-conversational pipeline: propagates silently into ERP/CRM/databases at volume — different detection strategy required.

19 / 24

ACT 4 · ECONOMICS — 20 OF 24

THE MODEL ECONOMICS REALITY

Same workload. 169× cost difference.

Model selection is your biggest lever.

100,000 queries/day · 3,000 input + 800 output tokens/query · mid-2025 pricing

Routing payoff

80% via Gemini Flash ($1,488) + 18% via GPT-4o ($8,370) + 2% via Sonnet ($1,260)

= $11,118/mo

vs $46,500 for all-GPT-4o → 76% cost reduction · 169× gap cheapest to most expensive

20 / 24

ACT 4 · GO DEEP — 21 OF 24

AGENT OPS · SECURITY · ORG REALITY

Three realities nobody addresses

until it's too late.

Agent Ops — new discipline (≠ DevOps, ≠ MLOps)

• Interpret semantic metrics: escalation rates, confidence distributions

• Prompt lifecycle: version control + regression gates + ownership

• Agent incident response: diagnose silent failures from traces

• Autonomy calibration: expand thresholds as reliability proven

Security — unique to agents

• Prompt injection: any retrieved text is attack surface

• Cross-tenant leakage: permission isolation at every context point

• PII exposure: scrub BEFORE LLM sees data

• Immutable audit log: every action traceable

Org structure — only one model scales at enterprise

Centralized CoE

Bottleneck · Nothing ships · Best talent leaves

Fully decentralized

47 vector DBs · 12 policies · Ungovernable debt

Federated ✓

Central: gateway, eval, governance · BU: use cases, adoption

21 / 24

ACT 5 · VISION — 22 OF 24

FUTURE OUTLOOK — GROUNDED IN WHAT IS ACTUALLY BEING BUILT

Back-office automation is NOT future.

It's production today.

NOW — Already in production (2024-2025)

• Autonomous document processing: invoices, claims, contracts at scale (Walmart, insurers, legal tech)

• Agentic procurement: PO processing, vendor matching, approval routing

• Code generation in CI/CD: GitHub Copilot Workspace, Cursor (full PR from issue)

• Customer service: IVR Integration and call routing

NEAR-TERM — Scaling now, mainstream 2026-2027

• Multi-agent cross-department workflows: procurement ↔ finance ↔ compliance

• AI-native BPM replacing Camunda / Pega / ServiceNow

• MCP/A2A protocol standardization: commodity infrastructure

• EU AI Act compliance tooling: audit trail as regulatory requirement

MEDIUM-TERM — R&D to production 2027-2029

• Self-healing agents: detect own quality degradation, auto-recalibrate

• LLM-based codebase maintenance at 10M+ line legacy scale

• Federated multi-enterprise agent networks

• Unified multimodal: text + structured + image + audio in one pipeline

Constraint on all of this: semantic observability maturity — answering 'are our AI decisions correct at scale?' with the same confidence as 'is our API within SLA?'

22 / 24

ACT 5 · CLOSE — 23 OF 24

THE FINAL COMPARISON — LEAVE THIS ON SCREEN DURING Q&A

You now know the difference.

Go build the right one.

"One action this week: write 100 representative queries for your golden eval dataset. That document is the beginning of the right column."

23 / 24

Thank You

The Enterprise GenAI Playbook: From Prototype to Production

AD

Ayush Dwivedi

Senior Manager, Data Science

Walmart Global Tech

Building AI systems at Walmart scale.

"The challenge is not generating text anymore.

It's building reliable systems around probabilistic models."