William McLean (xbill)

CTO Glitnir Ticketing, GDE Cloud + AI, AWS CB (AI)

Inference on AWS

Bedrock, SageMaker, GPUs, Inferentia and Trainium behind one Strands agent.

William McLean (xbill)

AWS Community Builder 2026, AI

GDE (Google Developer Expert), Cloud and AI/ML

CTO and co-founder, Glitnir Ticketing

2026 goal: multi-cloud agents, tools and models

Where to Find Me

LinkedIn linkedin.com/in/xbill

Dev.to dev.to/xbill

Medium medium.com/@xbill999

GitHub github.com/xbill9

Hugging Face huggingface.co/xbill9

AWS Builder Center builder.aws.com/community/@xbill9

One Question, Six Ways to Answer It

The Six Backends

Same weights on g6 and SageMaker; same compiled image on inf2 and trn1.

Precision: What You Trade

Gemma 4 E2B. Memory: MI300X sweep 2026-10-07. Quality: 3,880-record suite on TPU v5e, 2026-09-29. Native math: NVIDIA, AWS Neuron docs (fp8 compiler support pending).

Memory and Bandwidth Across AWS

Neuron: 16 GiB per NeuronCore, not one pool; a model splits across both cores or fits in 16. Bandwidth sets the ceiling; the engine decides how close you get. tok/s: one request, 2026-10-12.

Strategies

What decided whether a run happened today

Quota Does Not Mean Availability

inf2.xlarge in us-east-2a: InsufficientInstanceCapacity, with 80 vCPUs of quota free

trn1.2xlarge is offered in one zone of the regions checked: us-east-2c

Trainium spot quota starts at 0; requests went to support cases

Check zone offerings first; keep a second zone ready

Source: github.com/xbill9/aws-inference-strands, results/2026-10-05

Debug With the Small Model

Every path was brought up with E2B first

E2B loads on Neuron in about 2 minutes; the 26B in close to 4

Flags, port forwards and tool parsing fail the same way at any size

Move to the big model once the small one answers

Source: github.com/xbill9/aws-inference-strands, results/2026-10-05

Check Your Numbers, Verify Results

Code counts, times and ranks; the model quotes the result

Grade the filter the agent sent, as well as the number it said

One agent quoted a correct table, then named the wrong winner

Every figure in the writeup traces to an evidence file

Source: github.com/xbill9/aws-inference-strands, results/2026-10-05

Keep IAM Loose While You Build

One instance profile with SSM access served every EC2 instance

One SSM-only security group per region, kept between runs

No inbound ports: reach servers through SSM port forwards

Tighten roles before anything faces users

Managed: Bedrock and SageMaker

Nothing to start, or one endpoint to start

Bedrock: Nothing to Start

An IAM permission and a model id

Use the inference profile id: us.amazon.nova-micro-v1:0

Pay per token; tool calls work out of the box

Strands: BedrockModel(model_id=...)

SageMaker: You Bring the Container

create-model, create-endpoint-config, create-endpoint. Billed per instance hour until the endpoint is deleted. Source: sagemaker-gemma sm.py

SageMaker: The Same vLLM, Managed

AWS vLLM 0.31.0 container on ml.g6.xlarge

Tool calls through SM_VLLM_ENABLE_AUTO_TOOL_CHOICE and SM_VLLM_TOOL_CALL_PARSER

About 9 minutes from deploy to InService

Strands: SageMakerAIModel, with a fix for vLLM's usage field

Source: github.com/xbill9/aws-inference-strands, results/rehearsal-20261012T1429Z

GPUs on EC2

vLLM on an L4, and Arm with CUDA

g6.xlarge: vLLM 0.30 on an NVIDIA L4

4-bit embedding repack of Gemma 4 E2B

--enable-auto-tool-choice --tool-call-parser gemma4

16 minutes from launch to ready; first zone had no capacity

131.9 tok/s, one request at a time

Source: github.com/xbill9/aws-inference-strands, results/rehearsal-20261012T1429Z

NVIDIA L4: The GPU in g6 and SageMaker

Sources: NVIDIA L4 datasheet (nvidia.com), NVIDIA CUDA GPUs list, AWS EC2 G6 page

g5g.2xlarge: Arm Host, NVIDIA T4G

vLLM's arm64 image has no sm_75 kernels

Gemma 4's 512-wide heads need a Turing shared-memory patch

vLLM 0.27.2rc0 built from source, on a prebuilt AMI

37.2 tok/s; first boot reads weights for 521 s

Source: github.com/xbill9/aws-inference-strands, results/2026-10-05

Arm + CUDA: What Is Old Is New

AWS: G5g are "the first Arm-based instances in a major cloud to feature GPU acceleration." Sources: aws.amazon.com/ec2/instance-types/g5g; NVIDIA newsroom, May 31, 2026

AWS Silicon: Neuron

Inferentia2 and Trainium

Inferentia2: Gemma 4, Ported by Hand

No vLLM release serves Gemma 4 on Neuron

torch_neuronx graph compiled ahead of time, baked into a container

Fixed at 512 tokens total and 128 for the prompt

36.4 tok/s on inf2.xlarge; no tool calls

Source: github.com/xbill9/aws-inference-strands, results/2026-10-05

Trainium: The Same Image, No Rebuild

Both chips: two NeuronCore-v2 cores, 32 GiB each

E2B: ready in 121.2 s, 37.1 tok/s

26B MoE: ready in 220.1 s, 5.5 tok/s

trn1.2xlarge has 32 GiB host RAM: no swapfile needed

Source: github.com/xbill9/aws-inference-strands, results/2026-10-05

vLLM on Neuron Lags the GPU Releases

Source: github.com/vllm-project/vllm-neuron README; no release lists Gemma.

Why Neuron Is Hard to Serve On

No vLLM release serves Gemma 4 on Neuron: a hand-ported engine, no tool calls

Compiled ahead of time for fixed shapes (512 total, 128 prompt); a new shape costs ~3.9 s to compile

2 x 16 GiB, one per NeuronCore; a gather above ~4-5 GB returns zeros, silently

int4 weights become a dense bf16 copy before each matmul: 27x per call

Result: 2.0-2.7x the L4's bandwidth, 0.28x its decode speed. Source: gemma4-dev HARDWARE.md, measured on inf2.xlarge

One Strands Agent

A model object per backend; grading and timing in code

One Agent, Four Model Classes

Demo 1: Every Backend Picked the Right Filter

How many of 11 ids are 10 or more? count_ids(op, value) computes it; graded on id >= 10.

Demo 2: Right Table, Wrong Summary

Agent's summary: "The fastest response came from the Trainium backend." Run 2 of 3.

Same Prompt: vLLM on g6 Decoded Fastest

"Write a short paragraph about the ocean." 100-token limit, 5 runs each, timed from the client.

Per Token, the g6 Is the Cheapest Instance

On-demand list prices, AWS Pricing API, 2026-10-12; $/M = per million output tokens, busy hour; 16 req = 16 in parallel. Bedrock input $0.035/M. A 24x7 g6 ($588/mo) beats Bedrock above ~710 output tok/s.

So, Which One?

Start on Bedrock: nothing to run, pay per token

vLLM on g6 for your own weights: fastest and cheapest per token here

SageMaker: the same container as a managed endpoint

g5g: Arm and CUDA, with a patched vLLM

inf2 / trn1: when the model already exists for Neuron

Clean Up Four Regions

Terminate instances; delete endpoints, configs and models

Check us-east-1, us-east-2, us-west-1 and us-west-2

Failover launches leave resources in other zones and regions

Keep security groups and instance profiles for the next run

Links

Code and results: github.com/xbill9/aws-inference-strands

Rigs, incl. Trainium: github.com/xbill9/gemma4-dev

Write-up: Inference on AWS (dev.to/aws-builders)

Strands Agents: github.com/strands-agents/sdk-python

Questions

github.com/xbill9/aws-inference-strands

Article: Inference on AWS (dev.to/aws-builders)