1 of 17

The Israeli Systems & AI Workshop (ISAI) 2026

The Evolution of AI:

A Compute Story

Adi Fuchs, PhD�NVIDIA

“FROM VACUUM TUBES TO HETEROGENEOUS SILICON SYSTEMS"

2 of 17

SAFE HARBOR

The Evolution of AI · A Compute Story

DISCLAIMER:

THIS TALK DOES NOT REPRESENT NVIDIA

PERSONAL & INDEPENDENT

This presentation reflects my independent research and personal views.

NO AFFILIATION

It was prepared independently and is not sponsored, endorsed, or otherwise affiliated with NVIDIA or any of my current or former employers.

NO REPRESENTATION

Nothing presented here should be understood as representing the views, positions, products, or plans of NVIDIA or any current or former employer.

For informational and educational purposes only.

ISAI 2026 · ADI FUCHS

3 of 17

ACT I · THE QUESTION

ISAI 2026 · 01 / 10

The Evolution of AI · A Compute Story

THE DAWN OF THINKING MACHINES

AI has been with us a LONG time ago …

1950–1958

The Turing Test: “Can Machines Think?”

Dartmouth Workshop: McCarthy, Minsky, Shannon, and Rochester. Coined the term: “Artificial Intelligence”

Rosenblatt's Perceptron: a learning machine built from motors, wires and 400 photocells.

1950

1956

1958

… BUT compute was NOT.

4 of 17

ACT I · THE BOTTLENECK

The Evolution of AI · A Compute Story

The Ideas Were There,

But Scale Was NOT

THE ALGORITHMS EXISTED. COMPUTE CAPABILITIES DID NOT MATCH.

1960s compute ceiling

CDC 6600

10 MFLOPS peak

HYPOTHETICAL SINGLE-MACHINE TRAINING TIME · LOG SCALE

10³

10⁴

10⁵

10⁶

10⁷

10⁸

10⁹

YEARS

COMPUTER VISION

AlexNet · ImageNet 2012

4.7×10¹⁷ FLOPs

1.5K y

1.45 GFLOPs/image × 1.2M × 90 × 3

MACHINE TRANSLATION

Transformer-base · WMT14

3.3×10¹⁸ FLOPs

10.5K y

reported training cost · Vaswani et al.

IMAGE GENERATION

DDPM-CIFAR · 2020

3.8×10¹⁸ FLOPs

12.1K y

reconstructed U-Net count · 12.42G × 128 × 800K × 3

IMAGE GENERATION

DiT-XL/2 · ImageNet 256²

6.4×10²⁰ FLOPs

2.0M y

118.6 GFLOPs × 256 × 7M × 3

Sources: CHM CDC 6600 (1964) · Krizhevsky et al. (2012) · Vaswani et al. (2017) · Ho et al. (2020) · Peebles & Xie (2023)

IPhone 17

2.5 TFLOPs peak

250,000x faster!!

This is the mismatch that shaped the first AI era, and every era to follow: An idea can exist a long time before we have system that can carry it. �Back in 1960 the fastest computers were painfully slow compared to what we have today. A CDC 6600 mainframe peaked around ten megaflops which is almost a million times slower than a modern iPhone we have in our pocket.

Now compare that with the compute needed for modern AI tasks: computer vision, machine translation, and image-generation models. Even for the basic models training on a single CDC machine can stretch from thousands of years to millions. That is not an optimization gap. That is a civilization-scale mismatch between ambition and available FLOPs. The key point is not that early researchers lacked imagination. They lacked the ability to make the loop run fast enough to learn. When the learning loop takes longer than a career, even good ideas look like failures.

5 of 17

ACT X · THE OUTCOME

The Evolution of AI · A Compute Story

The Outcome: AI Winters

PROMISES OUTRAN CAPABILITY

The field did not disappear. The funding did.

1969

1974

1987

FIRST WINTER · 1974–1980

× Perceptrons as general intelligence

The XOR limitation (Minsky & Papert's 1969) read as a verdict on neural nets.

× Rule-based machine translation

ALPAC (1966) found progress below the promised payoff; funding receded.

SECOND WINTER · 1987–1993

× Expert systems

Useful in narrow domains; brittle and costly to update outside encoded knowledge.

× Lisp machines · Fifth Generation

Cheaper workstations won; Japan's 11-year project closed in 1992 without the expected commercial impact.

Sources: Lighthill (1973) · ALPAC (1966) · Minsky & Papert (1969) · House of Lords AI Committee · ICOT / Fifth Generation Computer Systems

Lesson: Hype Creates Investment. Capabilities Sustain it.

And when the promise outruns capability, the market eventually notices and funding runs dry. and so, we got to a period called “AI winters”

The first winter followed a series of claims that neural networks and machine translation were closer to general intelligence than they really were.

The second winter came when expert systems proved useful in narrow domains but turn out too brittle, expensive, and painful to maintain at scale. Notice what actually froze: not the ideas, but the funding. Researchers kept working; the outside world stopped believing the timelines. That distinction matters. AI winters are not proof that the field was nonsense. They are what happens when a compelling narrative meets a hard capability ceiling. The lesson for today is uncomfortable: hype can accelerate investment, but only sustained capability keeps it.

6 of 17

ACT II · LEARNING AT SCALE

The Evolution of AI · A Compute Story

WHEN DATA CENTERS LEARNED TO SEE

Computer Vision: NNs First Trained On Distributed CPUs

2000s–2012

01

2000s — statistical ML wins with hand-crafted features.

02

2012 — Google DistBelief scales a neural net across a CPU cluster.

03

10M unlabeled YouTube frames → one neuron learns “cat.”

04

The result is still CPU-bound: 16K cores, 1B parameters, 3 days.

Deep learning's brief CPU-cluster era · 2012

16,000

CPU CORES ACROSS 1,000 MACHINES

1 billion

PARAMETERS — LARGEST NET OF ITS DAY

3 days

TO TRAIN — AND STILL CPU-BOUND

So what do we need? We need scale and to reach scale, neural networks were first using the traditional cloud datacenters that were based on general-purpose CPUs.

Google’s DistBelief paper distributed a neural network across roughly a thousand servers with roughly 16,000 cores. The system could ingest millions of unlabeled YouTube frames and discover a cat without being explicitly taught the concept of what is a cat.

That is an extraordinary result—but also a warning sign. Using thousands of computers to simply identify a cat is very inefficient.�The model had about a billion parameters, yet training still took days and remained fundamentally CPU-bound. The important transition is from one machine to a cluster of machines.

Once learning became a distributed systems problem, we asked: what if the individual processors were massively parallel and were optimized for the math neural networks actually needed?

7 of 17

ACT III · THE GPU BIG BANG

The Evolution of AI · A Compute Story

AlexNet: When GPUs Started The Deep Learning Era

2012

01

AlexNet wins ImageNet 2012 with a 15.3% top-5 error.

02

THE MAGIC: First Implementation of CNNs on

massively-parallel GPUs (Two NVIDIA GTX 580s)

03

A 60M-parameter CNN trainable in 5–6 days, not years.

04

Slashed the error of state-of-the-art solutions by almost half

Source: Krizhevsky et al. (2012), Abstract + Tables 1–2. 2010 and 2012 values are shown as separate competition groups.

That moment came with AlexNet which started the era of deep learning in 2012 and this is where the accelerated hardware story becomes visible.

The model did not merely improve image recognition; it made a compelling case for using GPUs to do the neural network number crunching.

The authors used two NVIDIA GTX 580s GPUTs to train a 60-million-parameter CNN in roughly five to six days—an enormous improvement over the years implied by earlier systems. The result was a dramatic drop in ImageNet error, and the message was unmistakable: massively parallel hardware can change what counts as a practical model. This is the breakthrough to remember. The algorithm was important, but the accelerators changed the time horizon. Once training moved from “maybe someday” to “this week,” deep learning stopped looking like a research curiosity and started looking like a real industrial problem.

8 of 17

ACT IV · BREAKING THE SERIAL CHAIN

ISAI 2026 · 05 / 10

The Evolution of AI · A Compute Story

Language: Transformers Broke the Serial Bottleneck

1997 → 2017 · state encoding meets parallel context

01

RNN / LSTM encode the past in a rolling state hₜ.

02

The bottleneck: hₜ cannot exist before hₜ₋₁.

03

Longer sequences mean longer chains — less parallelism.

04

Transformers encode context in parallel with all-to-all attention.

RNN / LSTM · STATE ENCODING

Hochreiter & Schmidhuber, 1997

TRANSFORMER · CONTEXT IN PARALLEL

Vaswani et al., 2017

ATTENTION · PARALLEL “ALL-TO-ALL” PROCESSING

THE PAYOFF

12 hours · 8 P100 GPUs

significantly more parallelization

Sources: Hochreiter & Schmidhuber(1997) · Vaswani et al. (2017), §1, Table 1, Figures 1 & 3

GPUS · AGAIN – THE ENABLERS

Vaswani et al., 2017

Language had a different problem. we had some neural architectures running on GPUs, specifically RNNs and LSTMs but training them was slow since they encode the state serially, one word gets encoded, then the second, and so on so each word depended on all predeceasing state.

That serial chain is a natural way to encode history—but a terrible way to use parallel hardware.

Transformers changed the geometry of the problem. Attention lets every token look across the context in parallel, turning sequence processing into a matrix operation that accelerators can attack all at once. This is why the Transformer was more than a better architecture. It was a better fit for the machine. Once context could be processed in parallel, the path from data to scale opened up—and language became the engine of the next AI wave.

�- OpenAI ImageGen, generated conceptual illustration for this presentation, 2026-08-30.

9 of 17

ACT V · THE AGE OF SCALING LAWS

ISAI 2026 · 06 / 10

The Evolution of AI · A Compute Story

01

Scaling laws: loss falls as model size, tokens. and FLOPs grow.

02

Frontier training compute grows ~4.4× per year (Epoch AI).

03

GPT-4, Gemini Ultra, Grok 4 now reach 10²⁵–10²⁶ FLOPs.

04

Bitter Lesson: scale wins over human eng; but at what costs?

THE DOWNSTREAM PROBLEM

“That’s going to take new economic models or much cheaper inference to get wider spread.”

— Sam Altman, OpenAI Forum (2026)

SCALING LAWS · KAPLAN ET AL. (2020)

EPOCH AI · TRAINING COMPUTE (FLOP)

4.4× / year

Sources: Kaplan et al. (2020), Figure 1 · Epoch AI, “Training compute of notable models” · Touvron et al. (2023), LLaMA Figure 1 · arXiv:2302.13971 · Sam Altman, OpenAI Forum (2026)

Scaling Laws: More Parameters, More Data, More Compute

THE TRILLION

DOLLAR PAPER

And in language models, we saw this trend called the “scaling laws.” with OpenAI’s 2020 on scaling law paper which I like to call “the trillion dollar paper”.

The takeaway was simple: you increase model size and data, and at the cost of more compute—you get a better model. These curves does no tell you everything, but they turn model quality into an compute budget problem. This triggered the race for building bigger bigger GPU training clusters, the amount of compute for the largest models increased by more than 4x year after year. And this made chip manufacturers, especially NVIDIA, hundreds of billions of dollars.

However, this created a downstream problem. The increased cost of a bigger model is not just a one-time thing during training, it is also recurring again and again during serving time. We cannot possibly use the entire model for all tokens.

10 of 17

ACT VI · THE SPECIALIZATION TURN

ISAI 2026 · 07 / 11

The Evolution of AI · A Compute Story

THE HARDWARE PRECEDENT FOR SPARSE EXPERTS

01

Dennard Scaling that powered >50y of CMOS scaling stopped working.

02

Transistors multiplied; full-chip power rose as S².

03

At 80 W budget, usable silicon fell 17.6% → 6.5% → 3.3%.

04

The response: specialized circuitry, each built for a hot path.

THE UTILIZATION WALL A.K.A DARK SILICON

You can pack more transistors, but can’t power them.

The solution: specialize the data path, use the transistors you have more efficiently!

CONSERVATION CORES · FULL-DIE POWER

300 mm² die · full frequency

full-chip power (watts)

DARK SILICON · % OF CHIP OFF

22 nm: 21% off → 8 nm: >50% off

Power rose 5.3×; usable silicon fell 5.3×.

SPECIALIZED SILICON · C-CORES

general CPU vs. CPU with specialized cores

More capacity, less active compute — the same trick is coming to the model.

Sources: Venkatesh et al. (2010), Table 2 & Fig. 1 · Esmaeilzadeh et al. (2011), Fig. 4 · Fedus et al. (2022)

Dark Silicon: Silicon Lessons From Unsustainable Scaling

now that we scaled and scaled into a point where budget increased drastically, it is worth noting that the silicon industry has already experienced this with the end of Moore’s law or more accurately Dennard scaling. At some point transistor power stopped dropping transistors kept multiplying, but the chip could not power all of them. That is dark silicon: capacity exists, but power constraints make it unusable. The response was not to stop getting more transistors, and stop scaling all at once. It was to specialize them. By building task-specific sub parts of the chip and activate only the specialized part for the a given workload. More capacity, less simultaneous activity. The equivalent in the world of AI models is that when dense computation becomes too expensive, we do not throw away scale. We route tokens selectively to sub-parts of the model.

11 of 17

ACT VI · THE SPECIALIZATION TURN

ISAI 2026 · 08 / 11

The Evolution of AI · A Compute Story

MoE: Dark Silicon Lesson Applied to AI Models

DENSE MODELS CANNOT KEEP PAYING FOR EVERY TOKEN.

WE NEED TO SPECIALIZE THE DATAPATH FOR AI MODELS LIKE WE DID FOR CHIPS!

MODEL-SCALE SPARSITY

DENSE MODEL

MIXTURE OF EXPERTS

t1

t2

t3

FULL BLOCK 1

FULL BLOCK 2

FULL BLOCK 3

FULL BLOCK 4

100% of the stack is activated for every token.

t1

t2

t3

R

EXPERT 1

EXPERT 2

EXPERT 3

EXPERT 4

1 of N experts is active per token.

A modern LLM would consist of many experts, each specializing at a subset of problems,

much like an SoC consists of HW accelerators, each specializing at a subset of problems

AI-SILICON PARALLEL FUTURE:

12 of 17

ACT VII · THE INFERENCE ERA

The Evolution of AI · A Compute Story

2025 DISAGGREGATED SERVING

Serving LLMs: the Prefill / Decode split

PREFILL · BUILD THE CONTEXT

Computes all to all attention from the input.

Builds the KV Cache (Context)

Writes to memory only once

Highly parallel. Compute Heavy. Bandwidth Light.

REQUEST

prompt

prefill

KV cache

decode

DECODE · AUTOREGRESSIVE PREDICTION

Generates one token at a time.

Append KV pair to context. Then, predict next token.

Weights + KV cache move each step.

Inherently serial. Compute Light.

Bandwidth Heavy.

.

KV CACHE HANDOFF

PREFILL POOL

DECODE POOL

TWO PHASES. TWO SYSTEMS. TWO PROPERTIES

So terrific – now we have reasoning model mixture of expert models that power chatbots and coding agents. Pretty much the most popular workload in the world right now is LLM serving, but this raises a new system challenge – how do we efficiently serve these models for billions of users? and for that we need to look at the anatomy of LLM inference. Each request is actually two workloads.

1. The Prefill which reads the full prompt in parallel and builds the KV cache.

That means large matrix multiplications, and a bottleneck dominated by compute. User feel this as time to first token.

2. The Decode which flips the profile. The model generates one token at a time, repeatedly streaming model weights and an expanding KV cache from memory. The Chip sits mostly idle while data is brought from memory. So memory bandwidth, is the performance bottleneck here. Users feels this as time per output token.

We have two phases, with two distinct system properties.

��[Sources]�- OpenAI ImageGen, generated conceptual illustration for this presentation, 2026-08-30.

13 of 17

ACT VI · A CAMBRIAN EXPLOSION OF SILICON

The Evolution of AI · A Compute Story

ONE WORKLOAD, MANY MACHINES

2016–NOW

The GPU won the war — so specialization began. No one size fits all

LLM serving is also enabled by hardware – a slew of heterogenous chips

Google TPU

Systolic matrix array — dense matmul at datacenter scale.

Cerebras WSE

Wafer-scale: ~4T transistors, 900k cores, no off-chip wall. High bandwidth and compute. But limited capacity, GOOD FOR DECODE

Groq LPU

Deterministic dataflow, all-SRAM — record decode latency. High bandwidth.�Limited capacity. GOOD FOR DECODE

SambaNova RDU

Reconfigurable dataflow — the chip reshapes to the model. Three-tiered memory architecture.

DMatrix

Massive MIMD with memory next to every core

AWS Trainium

Networking-first silicon built to scale training out. High parallelism, lower bandwidth – GOOD FOR PREFILL

The GPU won the war — so specialization began: dataflow, wafer-scale, and SRAM machines each attack a different bottleneck.

The right phase should run on the right silicon

And this split has hardware ramifications. GPUs remain flexible generalists: strong tensor compute, large HBM, mature software, and support for both phases. �But the prefill and decode disaggregation opens the door for a heterogenous system consisting of many other specialized chips designed under different architectural assumptions. Prefill rewards wide matrix engines, high FLOP throughput, and fast scale-out links. That is the design center of systolic and tensor machines such as Google TPU and AWS Trainium.

Decode changes the priority. Every token rereads weights and touches the growing KV cache, so performance depends on bandwidth, capacity, locality, and predictable movement. That is why SRAM-rich and dataflow approaches from Groq, Cerebra sand SambaNova become compelling.

And this is not just one design decision - batch size, context length, model size, and KV-cache transfer can change the optimal system, also different KPIs like time to first token, energy efficiency, interactivity of concurrent users and so on. This is again, enabled by hardware, and many of the chips that were designed even years before LLMs were a thing and just happen to be in the right time at the right place.

��[Sources]�- OpenAI ImageGen, generated conceptual illustration for this presentation, 2026-08-30.

14 of 17

ACT VIII · AGENTIC AI

The Evolution of AI · A Compute Story

THE AGENTIC LOOP: STARTED AROUND LATE 2025

Agentic AI: CPUs are back!

A SINGLE REQUEST

GPU / Accelerator

dense tensor math

CPU

control, tools, glue logic

Memory & Storage

KV-cache, retrieval, state

Network

the fabric binding it all

GPU · CPU · MEMORY · NETWORK

Agentic AI: from simply inferring to tool calling

Sending tokens as inputs, parsing program outputs as tokens

Most programs still run on CPUs, and we also need CPUs for orchestration

From 4-8:1 GPU to CPU ratio, we are back at 1:1 or even 1:2 (more CPUs!)

Now we have come to the age of AI agents which arguably started around December of 2025. Agents bring the CPU back into the picture. An agent does not simply run a forward pass; it plans, calls tools, retrieves information, executes code, and loops. That means one request now crosses an accelerator, a host CPU, memory, storage, databases, and a network fabric. The GPU is still essential, but it is no longer alone. The system is heterogeneous because the work is heterogeneous. We see an uptick in CPU demand because unlike the last AI decade days where GPU to CPU ratio was about 4 or 8 to 1, now we are back at 1:1 ratio or even 1:2 ratio.

This is the deeper pattern running through the entire talk: when the workload changes, the machine changes with it. The next generation of AI performance will come from orchestrating the whole stack, not from asking one chip to do everything.

15 of 17

2012-2018

2021-2025

2023-

2024-

2025-

16 of 17

ACT X · THE NEXT FRONTIER

What’s next for AI?

History has taught us: new AI frontiers will always require new means of compute

The Evolution of AI · A Compute Story

So what’s next in the world of AI? The next big trends in AI may take us in several very different and exciting directions.

Physical AI brings intelligence into the real world: world models, sensing an interacting with a physical non-digitized environment.

Continual learning means models can keep adapting from experience instead of being frozen after training. Models at the edge move intelligence closer to where data is generated, under tight power and latency constraints. And Neuromorphic computing challenges the conventional FLOP-centric approach altogether.

We do not know where these trends will lead, but history suggests that whether it is agents, physical AI, edge intelligence, continual learning, or entirely new substrates, the question will again be the same one we started with: what can we compute?

Because every time AI hits a wall, someone builds a new system that will lead the way through it.

17 of 17

LET’S CONTINUE THE CONVERSATION

The Evolution of AI · A Compute Story

Adi Fuchs

So what’s next in the world of AI? The next big trends in AI may take us in several very different and exciting directions.

Physical AI brings intelligence into the real world: world models, sensing an interacting with a physical non-digitized environment.

Continual learning means models can keep adapting from experience instead of being frozen after training. Models at the edge move intelligence closer to where data is generated, under tight power and latency constraints. And Neuromorphic computing challenges the conventional FLOP-centric approach altogether.

We do not know where these trends will lead, but history suggests that whether it is agents, physical AI, edge intelligence, continual learning, or entirely new substrates, the question will again be the same one we started with: what can we compute?

Because every time AI hits a wall, someone builds a new system that will lead the way through it.