1 of 56

Talk blurb from the LocoMocoSec 2022 Agenda:

Hopefully by this point we’re all pretty sold on the concept of “paved roads” for security. What feels less obvious is exactly how we *make* a paved road, or why some attempts to pave a road end up with dead ends, or maybe a few vaguely-gravel-like paths.

We’ll look at some examples of successful security paved road investments, and a few less-so, and tease out some of the principles and practices that make them more effective for both engineering joy and risk reduction. I’ll urge more “productizing” of security work: from getting product management skills (and roles!) more involved, through building capabilities that are easier to deploy, reason about, and argue for.

2 of 56

Productizing Security For Leverage and Scale

Patrick Thomas (Netflix)�@coffeetocode

LocoMocoSec

Honolulu, HI, June 30, 2022

Q&A:

bit.ly/LMSQA

3 of 56

Image Credit: Florian Wehde

4 of 56

Image Credit: Photoholgic

At the same time, I know that we try a lot of stuff, and a lot of it doesn’t work.
Sometimes an approach seems really exciting to us, but it never quite gets the adoption needed to be worth it,

or needs too much fiddling by owners/developers to reduce real risk,
or maybe sometimes it does get deployed and adopted and people do all the fiddling the operationalize it… and we still don’t feel like we’ve gotten real “take it to the bank” assurances that we celebrate as risk wins.

These things tend to atrophy and get abandoned. It really sucks when this happens.
The common thread in many of these cases is that we built the wrong thing.
And here’s the painful part: I think in a lot of cases, we come *really close*.
Like we built *most* of the right thing, but didn’t realize how important that next 10 or 20% was.
Or built something that solves the part of the problem *we* really care about, but missed a key perspective or a need.
And further, I think that missed little bit to the finish line of real risk reduction is work that looks a lot like things that product managers know is important

(2 min - :02) Intro:

(Other image options for “missing”: leaf 1, spiderweb 1)

(Other image options for “so close”: one, two, three)

5 of 56

Productizing Security

Dictionary def:

Making something “marketable for sale”

For us today:

Treating our internal engineering orgs as customers, and developing a security capability to a level where it provides unambiguous value (to both product teams and security teams) and an experience they would choose to use.

So, I’m going to call that “productized”, here’s a working definition
Treating our internal engineering orgs as customers
Developing a capability unambiguous value
experience they would choose to use.

unpacking what I mean by that is going to be the meat of this talk.

And also, I’m casting a very wide net here with the definition of a thing that we can and probably want to think about as productizable.
Really anything that we want people to use, and, at least implicitly, promise that it will have some value.
Everything from an entire framework down to a single checkbox or feature flag. We should at least be willing to think about it is something that has these properties.
All sorts of browser features and header flags
CSP
Compiler flags
Lockfiles
Secure Libraries
Vuln management tools
Alerts
Slackbots
Security pipeline steps or scanners
Frameworks or framework features

6 of 56

Focus Today

Teams building to impact risk posture

Internal building for internal

Failure modes especially relevant to highly technical security teams

(eg: you are your own SME)

“More Productized” Not Always Better

Now, Product Management is a huge discipline. I’m going to talk about the small corner that I think is most relevant to this audience.”

First, we’re focusing on teams building things designed to have an impact on risk posture.
Mostly internal, meaning that we’re building where the consumer is other engineering teams in our some company or organization

(This probably works for building security products for sale too, but not the focus)

Failure modes especially relevant to highly technical security teams where they are the SMEs
I’m gonna say a lot of nice things about doing a little more productizing than we might otherwise.

However, I want to make sure that I’m not giving the impression that turning that knob all the way is always the right choice.
Sometimes we don’t even necessarily want to turn the knob at all, we might just benefit from communicating more crisply about what a thing is and is not.

My Bias / My perspective: hundreds of engineers building and operating hundreds of (mostly micro-) services. Thinking at ecosystem level

(2 min - :04) Definition and Focus of this talk

7 of 56

Productizing Security

Easy to determine applicability

Easy to consume/adopt

Clear value proposition

Usage produces invariants

Specific branding/messaging

Product/Business value metrics

8 of 56

Don’t need an “it depends” conversation with a security SME

The problem is clearly defined

The customer is clearly defined

Can understand what “solved” looks like

▸ Easy to determine applicability

Easy to consume/adopt

Clear value proposition

Usage produces invariants

Specific branding/messaging

Product/Business value metrics

Honestly, this is product management 101.
I’ve phrased it as “easy to determine applicability” because that’s the property that I think a solution should have when viewed by a customer

Eg, someone doesn’t need a security expert to help them choose it; that can feel like a high bar, but you’ve seen how few of us there are. If people have to gate

…But what’s a necessary precondition for that is having a really crisp, clear, articulation of the *actual problem*, as distinct from any concept of a possible solution.

“Solutioning first” is gonna been a theme of how we get in trouble.

And also the actual user, and why they want this (or need it and believe they need it)
You’ll hear these together as “product-market fit”, which I think too easily causes those of us building for internal teams to think “I’m not worried about a market, I’m just thinking of engineers in the building next door”

This is that assuming going on; that we’re the security SME, we know what people need. That’s an assumption to be wary of.

So think about this from a customer’s perspective where, again, our customers are the engineering teams were here to support
Can someone very quickly understand:

This thing is for a problem I have
I’m able to use it
It will solve that problem

Yes. THOSE PEOPLE. Go get specific. Not “engineers in our product org”, but which person. Exactly why.
Getting specific is how we really learn what to build.
One neat example of that a couple PMs drilled into me was don’t ask “do you ever” or “what if you could”. Ask, “In the last three months have you done this specific thing, or had this specific problem? How did you solve it?”
Can also think about this as a central security team; if we have to field a lot of questions about something that we think shold be helping people, we
… and thinking about each of those, squeezing those pieces down until they don’t exist is how we turn this productization knob.

9 of 56

Self-serve

As few knobs/dials/changes as possible; Zero is a good number.

Supports “land, then grow”

Easy to determine applicability

▸ Easy to consume/adopt

Clear value proposition

Usage produces invariants

Specific branding/messaging

Product/Business value metrics

10 of 56

Specific class of issue killed

Compensating controls or processes not required

Responsibilities entirely offloaded from the adopter

Easy to determine applicability

Easy to consume/adopt

▸ Clear value proposition

Usage produces invariants

Specific branding/messaging

Product/Business value metrics

11 of 56

When used, always produces the value proposition

Creates evidence of that

Makes the environment easier to reason about

Easy to determine applicability

Easy to consume/adopt

Clear value proposition

▸ Usage produces “invariants”

Specific branding/messaging

Product/Business value metrics

This one is also more about things that make products valuable to central security teams
Top one: the bare fact of a thing being used, ideally, tells us that we can know that it’s giving us that value proposition.
The thing produces evidence that it’s in use (or makes it easy to gather that)
I’ve used this word here “invariants”: that sometimes crops up in formal verification of software, and I think it’s a concept we could port the idea of a little bit.

Essentially, we’re talking about properties of a system that, because of some other choices, WILL ALWAYS be true, or WILL NEVER be true.

Requests to this handler are ALWAYS authenticated.
Invocations here ALWAYS pass a policy which ALWAYS has an audit event.
This secret WAS NEVER available on the service at the same time it was processing malicious requests.
Code on this box CAN NEVER reach out to fetch second stage payload.

If a security product always produces a specific value proposition when used, AND we have reliable evidence of where that’s used, in the environment
Then that can make the entire system easier to reason about.
During an incident, are certain things out of incident scope because evidence generated by a thing?
Can certain attack trees end
Calling back to Clear value proposition -- it’s the difference between taking responsibility for a problem, and definitely solving it.

12 of 56

Be a specific thing

Others can articulate your value prop

Concise enough for leadership messaging

Think: applicability, value prop, invariants

Easy to determine applicability

Easy to consume/adopt

Clear value proposition

Usage produces invariants

▸ Specific branding/messaging

Product/Business value metrics

When we talk about it: it’s a specific thing, means same to everyone, others can articulate the value prop
We hate vanity bug naming, but c’mon, it works right?

No one knows CVE-2014-0160, but if I say heartbleed now we’re all on the same page

The difference between talking to an eng leader

“we see that your teams apps do user callbacks, we need you to make sure someone from each team looks at this coding standard on the wiki and follows the checklist for that kind of functionality”
“We’re slowly cutting your apps over to the safecallback module, but 6 of your app teams have said they can’t or won’t and will need to plan time to build equivalent controls. “

This isnt’ just about getting people to use something; it’s about someone telling you they used a thing, and you know something specific (and think back to the invariants)
Usage produces numbers that can be success metrics.

13 of 56

If this helps, how can you prove it?

If you were charging for this, how would you justify your price?

Easy to determine applicability

Easy to consume/adopt

Clear value proposition

Usage produces invariants

Specific branding/messaging

▸ Product/Business value metrics

14 of 56

Even a little bit of productization helps!

Highly productized approaches to security (with commensurate impact)

This didn’t go so well. Missed opportunities, lessons learned

15 of 56

Even a little bit of productization helps!

Highly productized approaches to security (with commensurate impact)

This didn’t go so well. Missed opportunities, lessons learned

2 first attempts at least privilege

Secure Workload Identities (Netflix’s equiv of SPIFFE/SPIRE) + ServiceMesh w/ mTLS

NaCl, libSodium, & Tink cryptographic libraries

#2 & 3

#7

Resource Templates & Modules

#6

#8

High leverage API Gateways (a Netflix example)

#1

2 access control stories

#4 & 5

16 of 56

Blog: The Show Must Go On:

Securing Netflix Studios At Scale

streamline any security processes

raise the overall security bar

Need 1:

Need 2:

A Story:

Supporting Netflix’s “Studio in the Cloud”

Here’s that first one.
I’ll also preface this: I don’t wanna give “that” kind of “hey, look at us, we did something neat” talk.

I’m Gonna share just enough of this for you to hopefully see why I’m excited,
then go talk about failing to do this, and learning.
There’s some bumps and wrinkles I’m skipping right over; see the blogpost if you want more.

okay?
So, back in 2017, Netflix was deciding to really go big on becoming a studio,

and as part of that it was “if we’re going to be a studio we want to have all the tools that we need to do that,
and available to the absolute army of people around the world who take things from pitch to play.”

So the need was:

we have an entire new line of business,
with many, many unique apps,
exposed to a new-to-us user population that’s gonna be roughly the size of “everyone on the planet who works in movies and TV”,

and it’s all gotta be on the internet becasue…
<click>

17 of 56

18 of 56

too many requirements; checklists make it worse

Observation:

😰😱

Our strategy for this focused on two observations.
The first was that while we *had* checklists for developers, there was waaaay too much stuff on them, and there were too many different options with like, the worst kind of choose your own adventure.

(“new apps do this, but legacy apps do that”;
“Java apps should use this approach,
but Ruby apps should try one of these four things”

… yes, there were flowcharts inside checklists. Ouch.).

For development teams, just working through the flowcharts of requirements and options was a monumental task,
but then because there was so much inclarity in the alternatives and combinations of options, that ultimately security engineers needed to be significantly involved in guiding and reviewing choices.
The first was that there were too many security things that each software team needed to think about — things like TLS certificates, authentication, security headers, request logging, rate limiting, among many others.
There were security checklists for developers, but they were lengthy and mostly manual, neither of which contributed to the goal of accelerating development.
Adding to the complexity, many of the checklist items themselves had a variety of different options to fulfill them
Supporting developers through those checklists for edge cases, and then validating that each team’s choices resulted in an architecture with all the desired security properties, was similarly not scalable for our security engineers.
Checklists here were also a *symptom*: we hadn’t really defined the problems or the solutions well enough that anything other than a human could interpret them.

19 of 56

bulletproof authN reduces the entire risk of a an app

Observation:

🤩

😊

(Using FAIR, but

these are fake numbers)

Our second observation centered on strong authentication as our highest-leverage control.
Missed or incomplete authn was the highest impact issue that we regularly faced, but at the same time anything that had bulletproof auth could just be shifted way down the risk curve.

And quick aside: if you haven’t seen risk curves like this, they’re a way to model risk not as red/yellow/green or a list of controls, but as probability of loss exceeding certain value, based on estimated frequency and impact.
So we get really excited about controls that let us pull that forecast loss curve right up there gainst the Y-axis where it belongs.

Missing or incomplete authentication in an application was the most critical type of issue we regularly faced, while at the same time, an application that had a bulletproof authentication story was an application we considered to be lower risk.
Concepts like Zero Trust, Beyond Corp, and Identity Aware Proxies all seemed to point the same way:
there is powerful assurance in making 100% authentication a property of the architecture of the application rather than an implementation detail within an application.

This is hinting an a thing I talk about as an invariant. More on that later.

Image source: https://sites.google.com/a/netflix.com/risk/fair-overview

20 of 56

21 of 56

😰😱

We ended up partnering with our Cloud Gateway team,
Their answer for our second observation was a custom build of our “front door” gateway service called Zuul, with a new always-on filter for SSO, and boom: an app deployment strategy that guarantees authentication for all services behind it. We called it API Wall, and then eventually it got the more affectionate nicknamed Wall-E

(...disney please don’t sue us it’s just a nickname).

This gave us pretty exciting pattern of adding security requirements as filters.
We thought back to our checklist through the lens of: which of these things are consistent enough across applications to add as a required filter?

A web application firewall (WAF), DDoS prevention, security heades, and durable logging all fit the bill, oauth revocation.
Each of these could transition from ‘individual app developer-owned’ to ‘Wall-E owned’ (and consistently implemented!).

Eventually, the bulk of the “going internet-facing” checklist for Studio applications boiled down to one item: Will you use Wall-E?

(5 min - :18) A brief example story about why I’m excited here:

who has historically owned the Netflix “front door” service called Zuul that ingresses all of our customer-facing traffic to our microservices.

22 of 56

Now, they could have stopped there and we’d have been, ya know, better off. Risk reduced, call it a day.
But they did another thing that seemed simple at the time but is deceptively meaningful: they asked app teams to integrate with them by creating a version-controlled YAML file.
Originally this was intended as a simplified and developer-friendly way to help collect domain names and some routing rules into a versionable package, but they quickly realized they’d stumbled into a powerful model: harvesting developer intent.
This was flipped the relationship with development teams: since cloud gateway got a concise, standardized definition of the app which developers intend to expose, they could proactively automate a lot of the setup.

Specify a domain name? We can ensure that it automagically exists, with DNS and TLS configured correctly.
Need static files? S3 buckets automatically safely configured, with publishing pipelines.
Tell us the user populations, you’ll get a reasonable OAuth policy set up.

So we could tell developers that setting up Wall-E would only take a few minutes and the our tooling would automate most of it.

(5 min - :18) A brief example story about why I’m excited here:

23 of 56

< 10 min

from “git init” to production-ready, fully authenticated, internet accessible app… that doesn’t (usually) need manual security review

Note: Developer Productivity org also needs a lot of the credit here

As all of these pieces came together, app teams outside Studio took notice.
For a typical paved road application with no unusual security complications, a team can go from “git init” to a production-ready, fully authenticated, internet accessible application in a little less than 10 minutes.

To be clear, we’re talking hello world here; but it’s a hello world that doesn’t scare the security team, and where it doesn’t become scary as soon as you start adding real code

The automation of the infrastructure setup, combined with reducing risk enough to streamline security review saves developers days, if not weeks, on each application.
Developers didn’t necessarily care that the original motivating factor was about security: what they saw in practice was that apps using Wall-E could get in front of users sooner, and iterate faster.

24 of 56

Also, with fewer exceptions and clearer criteria for apps that should adopt this paved road, our we could give more automated nudges for adoption, and more explicitly track who was making choices that put them in a lower-risk bucket.

With Wall-E as a clear, durable, paved road choice, we finally had enough of a carrot to move these teams away from supporting unique, potentially risky features.
The value proposition wasn’t just “let us help you migrate and you’ll only ever have to deal with incoming traffic that is already properly authenticated”,

it was also “you can throw away the services and manual processes that handled your custom mechanisms and offload any responsibility for authentication, WAF integration and monitoring, and DDoS protection to the platform”.

Overall, we cannot overstate the value of organizationally committing to a single paved road product to handle these kinds of concerns.
It creates an amazing clarity and strategic pressure that helps align actual services that teams operate to the charters and expertise that define them. The difference between 2–4 “right-ish” ways and a single paved road one is powerful.

25 of 56

From Product to Platform�Manual Changes vs Automated by Higher Level Abstraction

2019

2021

Finally, as other teams like UI Engineering and Developer Productivity came to see Wall-E as the clear answer for many of their customers, they started integrating their tools to configure Wall-E from the even higher level developer intents they were harvesting, stuff like new project generators or platform migration tools.
In effect, this moves authentication and traffic routing (and everything else that Wall-E handles) from being a specific product that developers need to think about and make a choice about, to just a fact that developers can trust and generally ignore.
In 2019, essentially all of the Wall-E app configuration was done manually by developers.
As fo late 2021, more than 50% of app configuration in WallE is done by automated tools (which are acting on higher-level abstractions on behalf of developers).

�(5 min - :18) A brief example story about

zuulconfig_committers for WallE blogpost

26 of 56

Image Credit: Katelyn Greer

27 of 56

Image Credit: Cottonbro

28 of 56

Repoman

▸ What did we try?

What went wrong?

What did we learn?

Two first attempts

at least privilege

My next example is a 2-for-1 deal, and pleasing to me because it feels kindof as close to laboratory conditions as we can get. Do an experiment, change one thing, repeat the experiment, see what happens.
Let me start by describing Repoman.

You may have heard of Netflix’s successful open source product Repokid; this is the first swing at that. (the InfraSec team said I could use the story)
Conceptually, it’s both simple and important. Overprovisioned AWS roles are dangerous, but if we crank down access too much, developers will hate us.

The idea was that if you automatically granted a reasonable starting set of permissions, you could observe over time, and automatically prune away anything unused to continually right-size permissions. Elegant, powerful, great.
So with this idea in mind, the InfraSec team set out to build that. They build all the infrastructure to get the data, diff permissions against actual usage, and throw it all into a dashboard that would notify developers when it had changes, and give them buttons to click to afix.

29 of 56

Repoman

What did we try?

▸ What went wrong?

What did we learn?

Two first attempts

at least privilege

30 of 56

Repoman�Repokid!

What did we try?

What went wrong?

▸ What did we learn?

Two first attempts

at least privilege

31 of 56

Repoing, Revisted:

Drivekid

What did we try?

What went wrong?

▸ What did we learn?

Two first attempts

at least privilege

Now, I said this was a 2 for 1.
Fast forward to 2020, and our Corporate Security team is thinking about the risks of Google Docs in our ecosystem, especially those “public with link” ones that always seem to crop up. This “repo” model is really powerful, so they leaned toward that.
First pass was a way to give people context on the kind of exposure they have for their shared files (I even asked for this because I didn’t know the full repoman story at that point and hadn’t learned that lesson. )
But… we have a PM on that team! They just made mockups for people, and asked them to click through it.
And realized that people saw this as an “audit” and worried primarily about breaking stuff they didn’t want to be thinking about, so would just bulk ignore rather than need to think about it.
So, their next iteration came to the same conclusion as Repokid: the users perception and experience is wrong here.
So they focused on files they could show were unused for a long time, and marketed it as a “helpful file cleanup”; and just *did it* rather than asking.

“Affected files” to “files being updated”
“Google Drive File Sharing Changes Scheduled”... that doesn’t help understand. “Netflix SecurityGuide Notice: Google File Sharing Cleanup”
“This notification is to inform you about …” to “Sometimes old documents are shared more broadly than they need to be. This notification is to inform you that public access is going to be removed for some of your old files.”

So, what changed for both of these in productizing?

Really, they both seem to have hit each of the dimensions (having clear applicability, easy adoption, a value prop, invariants for resources, doing some light branding, metrics on value)
…with one small extra effort paid on the second iteration to “how easy is it to use”, where “really easy, just 2 clicks!” was still not easy enough.
Notice need to change a branding there too

(4 min - :22) Example 2 (and 3): Netflix Repoman/Repokid & Drivekid

32 of 56

Access Control Stories: SCM & GraphQL

▸ What did we try?

What went wrong?

What did we learn?

“Risk” is a customer, (but…)�
“Least Privilege” is a strategy, (but…)

Another pair of examples here. Both of them involve us wading into an area where there was significant historical investment in locally-developed access control systems.
The first is a source code management:

We use an off the shelf vendor system, and it has lots of security knobs and dials that can be turned on at various levels of projects and repositories.
And for a lot of our history our approach was “here is something that can express any security needs you have, we’ve done some basic setup for you, go ahead and configure it further to meet any more needs. You’re an admin on your projects, we believe in you, go forth.”

The second is a migration of a large ecosystem to graphql, where we were taking a few hundred frontends and backends and getting them to all standardize on a single graphql graph, and we saw an opportunity to standardize on authorization approach as well

Of course, what that meant in practice was that we had to deal with all of these basically localized authorization conventions that had emerged over the years as this kindof coevolution between various frontends and backends.
We approached it with this perception that the most pressing need was to understand all of those, and get a sense of what kind of system could honor each of those local authorization norms

33 of 56

Access Control Stories: SCM & GraphQL

What did we try?

▸ What went wrong?

What did we learn?

“Risk” is a customer, (but…)�
“Least Privilege” is a strategy, (but…)

For the SCM, when we recently revisited it to get a better handle on our potential risk there, we found that people had indeed been using all those knobs and dials that it offered, in all sorts of ways.

Overall the result was that for both individual users and project owners, folks really weren’t certain exactly who had access to what (or when they did have access, why or exactly what mechanisms were making that possible). This also led to a lot of “cargo culting” in both adding and removing permissions, because that’s what other people seemed to do and you assumed they knew something you didn’t. No one really owned the properties of the whole system, so no one could meaningfully talk about making any kind of systemic improvements.

With the Graphql migration, that idea of understanding and honoring the local authorization norms became too fixated. As conversations and modelling went on, it slipped from a potential approach, to the definition of the goal. The effect of that was that we stopped seeing all of these local authz complexities and wrinkles as evidence or examples of types of risk that someone percieves that we may want to take into account, we saw each and every one of them as hard requirements.

Effectively, this ended up with a design, and even some pilot implementations, sortof importing all that distributed complexity to centralized complexity. In the old world teams could partake of just the local complexity that affected their caller/callee relationships, and we were threatening to build something where they had to understand the universe to configure their service.

34 of 56

Access Control Stories: SCM & GraphQL

What did we try?

What went wrong?

▸ What did we learn?

“Risk” is a customer, but don’t overindulge it.�
“Least Privilege” is a strategy, not a goal.

In both of these cases we ended up walking back a lot of that thinking and willingness to support complex controls, and really focusing on key, big risks we wanted to manage, balanced against the other key goal having a system that was simple enough to reason, actually understand risk, and make systemic changes.

, and the idea of. In both of them, we ended up finding a lot of types of controls that were clearly put in previously to address a type of risk where that risk/compelxity tradeoff made sense for teams that only need to deal with a very narrow slice of the world.

It’s really easy to assume that we all agree that the risk is that we’re trying to address, and the real hard questions are how to go about that.

But that’s a subtle challenge about risk in general and access control system design in particular: we can always chase risk with more complexity. We’ve been talking about our peer engineering teams as a customer, but it’s also useful to to think of risk as a customer here too. A really, needy customer, who will never tell you “yeah, that’s enough.” So we need to know that it’ll never say that, and really intentionally make that decision.
I think as part of that, I’m now more cautious of this idea of “least privilege” as an inherently right answer or approach. At the very least we should be weighing it against a minimal access control experience that produces sufficient risk reduction, and really owning what “sufficient” there looks like.
A colleague of mine phrases it as: “””When adding security to a system, the absolutely best we can possibly do is “it’s slower and it doesn’t work sometimes” …”””. So if that’s our best case scenario, we should be that much more thoughtful on how much we do both of those.

(4 min - :26) Example 4&5: Authz stories

Quick aside on the SCM example, as a difference between internal and external products:

Off the shelf products have to cater to a wide variety of needs; internal products can and should be far more opinionated about what kinds of features/levers/complexity is exposed, and what risk is worth addressing. Use that same thought process when configuring vendor product systems.
We instead should point clearly at what the minimum complexity that reduces

35 of 56

Incremental productization helps:

NaCl, Sodium, & Tink

OpenSSL is the space shuttle of crypto libraries. It will get you to space, provided you have a team of people to push the ten thousand buttons required to do so. NaCl is more like an elevator – you just press a button and it takes you there. No frills or options.

I like elevators.

Matthew Green

”

“

I am not a cryptographer. YMMV.

36 of 56

Image Credit: Carl Tashian

If OpenSSL

were a GUI:

37 of 56

Incremental productization helps:

NaCl, Sodium, & Tink

3 projects that each move us forward

NaCl: OpenSSL is a box full of DIY footguns. Let’s make useful primitives.

Sodium: NaCl, but with intent-oriented APIs and simpler, safer, defaults.

Tink: We like libsodium, but know that key management is half the battle.

Thanks @SchmiegSophie & Tink folks

3 projects that each move us forward, and I think the folks building them really understood the idea of making a security capability into a product and, especially as we look at sodium and tink, the value of incremental productization.
I’ve got them very rudely summed up there, and I think the productizing angle, if you’re a developer who just wants some confidentiality or integrity, is clear.
But looking at this as a central product security team, if you get opinionated about how your engineers should do crypto, even if you’re not delving into the internals you can then do things like divide the world into:

Teams that use our blessed crypto thing (and therefore don’t have a whole class of problems)
Teams that use literally anything else
Like… all the sudden grep becomes powerful enough to find crypto issues in your ecosystem.

Actually: let’s be more specific. We’re not interested in finding crypto issues, we’re interested in having assurance that they’re just not there at all. That’s… actually way more interest.

(Sophie, see here) As I was putting this together, a caution that I got from Sophie, a member of the Tink team, was that there can be diminishing value in turning this productization knob too far, especially when you’re pushing polish at the expense of applicability.

If you take away too much configurability because you’re assuming too much about your user’s need, you can create a “paved roads next to steep cliffs” situation.
We actually saw the same thing with the Wall-E example before; we took a pretty opinionated stand early on that we simply wouldn’t build a mode that didn’t enforce authentication for access to services… but we’re regularly getting service owners who still need the *rest* of that checklist, and all of our engineering effort and support has gone into paving the heck out of that road.

(4 min - :30) Example 4, 5: Tink, & (some other “a little bit helps”?)

https://cryptojedi.org/peter/data/tenerife-20130121.pdf
Developers are users too: https://www.usenix.org/sites/default/files/conference/protected-files/hotsec15_slides_green.pdf

38 of 56

Even a little bit of productization helps:

Resource Templates & Modules

( Terraform,

Cloudformation,

vhosts templates,

new app init,

etc, etc… )

From “templates exist” to “this is the way these resources are created”

Get it into a CLI or a UI so that people can “pick off the menu”

Standardization leads to metrics

Gonna keep saying it: “Hitch your security wagon to dev productivity”

I’m putting this here as another example of something that might not feel like a “product” at first glance, but if we’re willing to think a little more of productizing as a continuum, where an approach or a thing can be less or more “productized”, we start to see how to invest in a way that’s more likely to be successful.

Aside: I thought this was a pretty well known pattern for security, but I didn’t find as much online about it as I was hoping. So if you’re someone doing, this, write more about it please.

Think about an extreme here: Security org just goes and sets up some examples; makes some links from a wiki somewhere, calls it a day. People who find them can copy/paste if they want.
And a sliiiightly more productized thing is: Security org partners with engineering tools org, figures out what the typical process is to spin up a new thing (resource, project, environment, whatever) and finds some way to get a menu in front of people that they can pick off of, and like sorts it by most common or most recommended by user type.
And you can probably start imaging *very* productized versions of that, but I think Travis is trying to make a business out of that, so ask him if you wanna see what it means to turn that knob even further.
Security org partners with engineering tools org, figures out what the typical process is to spin up a new thing (resource, project, environment, whatever) and builds some kind of experience for “what are you trying to set up, here’s a library of standard things you can ask for, and a few typical configurations you can opt into. If you choose from this list, here’s a set of things central teams are owning for you… and anything created that way is managed; meaning we’ll do automated upgrades//whatever.
Again then central produce security teams can split the world of resources into 2:

Stuff that’s set up in a way that we understand the security properties of, and we know the format of in a way that we can bulk-update.
Stuff that’s not.

(4 min - :30) Example 4, 5: Tink, & (some other “a little bit helps”?)

So as we think about these things, we can probably turn each knob maybe a little bit further if we think about it and decide the value is there:

applicability
adoption
value prop
invariants
branding
Business value metrics

39 of 56

Workload Identity

(eg SPIFFE/SPIRE + mTLS),

Service Mesh, &

API Gateways

(a prediction)

Image Credit: Pavel Danilyuk

Okay, last one. This one is still an evolving picture, but I’m going to use it and kinda make a prediction.
Based on everything we’ve talked about above, I think these approaches are going to become increasingly the norm for orgs that are technically capable of them.
They represent to me an incredibly productizable approach.
Full disclosure: we have an internal version of workload identity (which I really like) that is conceptually similar to SPIFFE/SPIRE, but I don’t have direct experience with those so I’m doing a little bit of translating for an external audience. If you think I’m overstating the case here, please teach me.
But for workload identities: Roughly what we’re talking about here is that some part of your deployment process stuffs unique, provable keying material into every single workload, which it can then use when it talks to any other workload.

So we can have one reliable kind of identity that’s used everywhere, and we know that everything in our environment is capable, out of the box, of mutual authentication with everything in our environment,
then we’ve kinda got the mother of all paved roads to build other stuff on top of, and to me, it hits all of our “productized experience” boxes if you’re a developer just trying to get things done.

40 of 56

Workload Identity

(eg SPIFFE/SPIRE + mTLS),

Service Mesh, &

API Gateways

(a prediction)

Image Credit: nginx

Now let me add to this concept of service mesh. (I won’t try to full describe what it is here, but go look if you’re not familiar)
You can do service mesh in a effectively “shrink wrap” the instance or container so that all traffic going in and out *must* pass through a standard sidecar proxy. That proxy reliably gives you all kinds of metrics, tracing, observability in a standard way, and can also be used as a really bulletproof control point.

You want to create an invariant that all traffic to service is encrypted, authenticated, and gets checked against a policy? One place to do that, everywhere.
You want to make sure you have a complete list of all the random administrative consoles opened on random ports alongside the main service? Same answer.

And for developers, the questions of applicability and useability just vanish because it becomes the water they swim in without noticing, and we get strong guarantees about the things we assert using datamesh and universal workload identities.
And of course we already talked about the power of API gateways in our story
So I’m bullish on these kinds of approaches.

(4 min - :34) Example 6: (nflx’s internal variant of) SPIFFE/SPIRE and 🤞 ServiceMesh

Image credit: https://www.nginx.com/blog/what-is-a-service-mesh/

41 of 56

Takeaways

“Productizing” explicitly sees developers as customers

… which forces us to clarify value prop, reduce pain, & integrate better
… which gives us even more leverage to be effective at risk reduction

Scope the problem before you scope the solution (helps w/ NIH too!)

Productizing is a continuum; the sweet spot might be “a little” or “a lot” in any area

Plan (lots of) time to get feedback and iterate

The last X% may be a different skillset, & often means working cross-functionally

Risk is also a customer, but our job is to tell it “that’s enough”

42 of 56

Thanks

Jose Fernandez

Julia Knecht

Arthur Gonigberg

Sunil Agrawal

Lakshmi Sudheer

Romulo Salazar

Matthew Lapworth

Sarai Rosenberg

Astha Singhal

Ian Haken

John May

Christine Rhinehart

Anna Westelius

Jesse Kriss

Clint Gibler

Ronnie Flathers

Maya Kaczorowski

Leif Dreizler (did it first)

Travis McPeak

Alaeddin Almubayed

43 of 56

Thank You!

Patrick Thomas (Netflix)�

@coffeetocode

Slides:

bit.ly/lms22productizing

Q&A:

bit.ly/LMSQA

CREDITS: This presentation template was created by Slidesgo, including icons by Flaticon, infographics & images by Freepik.

44 of 56

Bonus Slides

45 of 56

Risks & Failure Modes

Scaling Up Badly: a bad process, automated, is actually worse. (Jesse)

The productizing disease: “Stuff can feel like migrations when they should feel like upgrades.” (Jesse)

Different skillset: Not safe to assume an existing, successful team can turn the knob arbitrarily far without help (Christine)

Paved roads next to steep cliffs: Teams that are different by necessity can be alienated quickly (Sarai)

Who owns failure?: If security engineering teams still own failure of people who chose not to use your stuff, this might not work well.

46 of 56

For the sake of argument…

can we productize something like “a Pentest”?

Yes, service ≠ product. 🍎& 🍊! 🐱& 🐶!

…but we all still do them and inflict them on our engineering partners

…and we want more leverage and scale out of them.

What does a “more productized” version of that look like?

Come argue with me later.

47 of 56

Productizing Security

“...treating developers as their customers, and being customer-obsessed in the ways that our companies' core engineering and Product teams are.”

“...the answer can never be "devs aren't doing what we want because they're stupid," just as a product person can never say, "Oh users aren't really adopting our product because they're dumb."

Hi Clint 👋, I stole your words:

Clint Gibler

@clintgibler

48 of 56

Even a little bit of productization helps!

Templatized & managed configs (apache vhosts, cloudformation/terraform, etc)
Browser Content Security Policy (CSP)
NaCl, libSodium, & Tink cryptographic libraries
Resource Templates & Modules
Security.md on open source projects

… etc

Highly productized approaches to security (with commensurate impact)

AWS Access Brokers (ConsoleMe, aws-okta)
HIgh leverage API Gateways (a Netflix example)
Data-Driven Least Privilege (Repokid)
Managed Cloud PKI (Lemur)
Secure Workload Identities (SPIFFE/SPIRE, & Netflix’s impl) + ServiceMesh w/ mTLS
Memory-Safe languages

… etc

This didn’t go so well. Missed opportunities, lessons learned

2 first attempts at least privilege
An anomaly detection system that we built because we could Access Control complexity: SCM & GraphQL
A secure data storage service sunk by not solving the actual problem developers had

… etc

2 first attempts at tuning access

Secure Workload Identities (Netflix’s equiv of SPIFFE/SPIRE) + ServiceMesh w/ mTLS

NaCl, libSodium, & Tink cryptographic libraries

#2, #3

#5

Resource Templates & Modules

#4

#6

HIgh leverage API Gateways (a Netflix example)

#1

49 of 56

Even a little bit of productization helps!

Browser Content Security Policy (CSP)
Security.md on open source projects

… etc

Highly productized approaches to security (with commensurate impact)

AWS Access Brokers (ConsoleMe, aws-okta)
Managed Cloud PKI (Lemur)
Memory-Safe languages

… etc

This didn’t go so well. Missed opportunities, lessons learned

A secure data storage service sunk by not solving the actual problem developers had

… etc

50 of 56

Plugs & References

(TODO)

51 of 56

An Unlikely Friendship: Why Security Engineers and Product Managers should be working together

BSidesSF 2022

52 of 56

Matthew Green & Matthew Smith

“Developers Are Users Too”

USENIX HotSec15

53 of 56

Matthew Green & Matthew Smith

“Developers Are Users Too”

USENIX HotSec15

54 of 56

Backup Slides

55 of 56

(Much) static analysis & security scanner pipeline work

TODO

OPTION: Maybe add this as an example, could lump in appspider

This is such a common pattern in product security circles that I think it deserves a mention
Integrate some misc scanning tool (static analysis, some dynamic scanner, whatever) into a pipeline, so that all developers have it as part of their build process. This is like held up as the definition of DevSecOps in some places.
I get it. It’s so attractive, it’s so promising, it seems like it’s going in the right direction. But at the same time, this is also the kind of system that when we follow up later, eventually seems to get torn down and dropped. We’ve taken 2 swings at that in the last 6ish years, and ended up removing both of them.
I feel like the way we *usually do it* just doesn’t hit those “productization”

Specific examples from us: AppSpider, Monterey, … finding bugs that are easy to find vs creating a system that we think is more robust.

56 of 56

Secure by Default

Springboot

▸ What did we try?

What went wrong?

What did we learn?

TODO