1 of 43

The Death of Hardware Knowledge

SCALE 20x

2 of 43

Who are we?

3 of 43

Julia Kreger

Sr. Principal Software Engineer @ Red Hat

Chair @ OpenInfra Foundation

Ex-Ironic Project Leader

4 of 43

Jay Faulkner

Ironic Project Team Lead

OpenStack Technical Committee Member

5 of 43

Why are we here?

6 of 43

What we are not here for?

7 of 43

“We are bare metal cloud developers, who automate things on bare metal”

8 of 43

We do this as part of the Ironic Project which provides Bare Metal as a Service.��Learn more: https://ironicbaremetal.org

9 of 43

But this is not an Ironic talk.

10 of 43

Or is it an Ironic talk?

11 of 43

We’re starting to see a trend…

12 of 43

Where the lower layers we have built newer technology upon, are less understood.

13 of 43

Creating a technology “fog-of-war”

14 of 43

15 of 43

…but it’s not all our fault.

16 of 43

Hardware is an afterthought to many modern developers.

17 of 43

But what are we leaving behind?

What are we giving up?

18 of 43

To answer that, we must recognize the trade-offs we make at each abstraction.

19 of 43

These trade-offs impact your business…

Vendors are working to monetize a lack of knowledge through services offerings

Chaotic Neutral: What we could pull off in 4 minutes, may take 4 hours, or days now. But in most businesses, that is choice driven by CapEx and OpEx blended with how the business wants to operate.
Good: Less to keep in our brains!
Better: We have more freedom to abstract problems away! This is great for industry and the evolution of technology!

20 of 43

…and your technology, too!

Loss of flexibility
Hidden risks in your system
Lock-in to a specific platform
Difficult to reason about performance

xkcd.org/2347

21 of 43

Let’s walk through some abstractions from the bottom up!

22 of 43

Your own Hardware vs. Managed Hosting Provider

Option A:

Your data center and it’s universe is local.

Option B:

Your data is hosted remotely, data is between two or more locations.

Trade offs:

Less specific control over the hardware. What is in the chassis?

The amount of data and your access pattern becomes a HUGE consideration.

23 of 43

Latency and Security management drives many operators to remain in direct control of workloads.

24 of 43

Physical Servers to Virtual Machines

Option A:

Local hardware or hosted hardware

Option B:

Virtual Machines (cloud or self-hosted)

Trade-Off:

Precise understanding of capacity/performance availability and locality plus the variability of noisy neighbors.

Utility-based billing plus auto-scaling compute can lead to surprise cloud bills. Overcoming data gravity can actually cost you money!

25 of 43

26 of 43

Virtual Machines to Managed containers

Option A:

Virtual Machines in a cloud provider.

Option B:

Managed container services (like kubernetes)

Trade-Off:

Lock-in to a single approach. Abstraction layer is the container, which can lead to surprise complexity.�Many traditional services, such as relational databases, can be difficult to manage inside a container.

27 of 43

28 of 43

VMs/Containers to Serverless

Option A:

An architecture built on building blocks of containers or VMs

Option B:

An architecture built on a curated platform (e.g. Heroku, AWS Beanstalk)

Trade-offs:

Control of your technology stack. Lock-in to a specific vendor. �Lack of knowledge or personnel to recover should your vendor shut down.

29 of 43

30 of 43

Mix and match your own trade-offs!

Example 1

Specialized Hardware (Radios) on physical cards in cell sites.
Telephone Call/Packet processing on regional data centers for the fastest data path to enable the end-to-end.
Call Accounting on centralized databases for billing purposes.
Customer-facing websites on a different technology stack altogether.

Example 2

Static marketing website and application assets hosted on a cloud object store
Web application and required dependencies managed by hosted kubernetes
Simple status page hosted on a serverless platform.
All web assets fronted by a CDN.

31 of 43

Technology is evolving.

Popular abstractions have changed over time.

Single host virtualization
API-based cloud computing and provisioning
Single manually managed containers on a host
Container orchestration engines�

Some problems are being approached in a more holistic manner.

Edge Computing
Hybrid Cloud�

32 of 43

Understanding the way you got there helps give you future direction.

33 of 43

Key Factors to Consider - Business

Financial Limitations

Preference for OpEx or CapEx?
Can you afford a long term Capital Expense?
Will doing the most efficient thing be too costly?

34 of 43

Key Factors to Consider - Risk

Risk tolerance or Intolerance?

Do you trust the vendor or technology to exist in the medium-term?
What would the cost be to pivot? To respond to a disaster?
What happens when the thing you thought would never happen, happens.

35 of 43

Key Factors to Consider - Technical

Block IO Latency
Type of IO (Buffered, or Direct)
Remote request round-trip timing
Bandwidth/Packet Loss/Protocols
Distance
Technical Impact of Policy
When will scaling become too prohibitive?
Are there aspects which cannot scale in a cloud?

36 of 43

Conclusion

We’re all more efficient now.
There are always multiple ways, and if your willing, you can engineer your way out of trade-offs, but doing so generally requires you to make deep and rigid decisions on your infrastructure and technology.
Just don’t forget what you’ve given up – Make explicit choices about where to draw your line of abstraction.

37 of 43

Most importantly!

38 of 43

Learn to negotiate your technical and business requirements.

39 of 43

Learning about the lower layers can be fun!

40 of 43

And if you have knowledge of the lower layers, mentor & teach!

Jay

Obviously you can’t learn everything from playing around in a homelab. In fact, many of the most important trade-offs we’ve mentioned you might not encounter until you’ve had the opportunity to work at scale (no the conference). This means that showing interest and a playful approach to learning isn’t enough. Sharing is required.

There are large communities of people out there willing to share knowledge, lessons learned, and try to prevent the mistakes of the past from being repeated.

This means:

If you’re knowledgeable about hardware; about problems at hyperscale, about pitfalls for migrating to cloud and containers; share them!

If you’re starting out, and may want to gain some of this knowledge, reach out! Some of you listening to this may be the ones to build the next abstraction. I don’t know what it is, but I want you to know the foundations you’re building on so we can make a future that lasts.

[plug office hours as an example of (my attempt to) to share knowledge]

41 of 43

Thanks for listening! Questions?

Jay can be found at:

OSS Office Hours

Tues 8a-9a, Thurs 3p-4p PST
https://jay.jvf.cc/officehours

JayF @ OFTC, Libera.chat
https://fosstodon.org/@JayF

Want a job? Talk to me!�G-Research is hiring!

Julia can be found at:

TheJulia @ OFTC IRC
@ashinclouds on matrix.org
juliaashleykreger@gmail.com
jkreger@redhat.com

42 of 43

For context:

The speed of light is:

299,792,458 m/s.

… In a medium (fiber):

~200,000,000 m/s.

43 of 43

One way:� ~622000 Meters

~ 3.11 milliseconds

Except: Protocols have error detection, you are not getting the entire internet archive back in ~7 milliseconds.