2 of 17

Threat Detection for Containers, Kubernetes and Cloud

Name

Title, Sysdig

Credits: this deck was created by Loris. The script was added by ewj@, after listening to Loris deliver it!

Hi everyone, I'm $NAME. I work for Sysdig, where our mission is to make every cloud deployment secure and reliable.

e.g. for Edd, "I've spent my career working with open source, from web development to big data and analytics, and from startups to Google. I'm really excited to be focusing my attention on modern security and to be able to talk to you today."

Today I will be giving an introduction to run-time threat detection in the cloud, why we need it, and how Falco provides an open source solution to the problem.

who here has heard of Falco? is anyone running it?
who's running Kubernetes? VMs? in a Cloud? On-prem?
what are you hoping to get out of today's session?

3 of 17

Once, there was a perimeter

You had a perimeter guarded by a firewall

Detecting intrusions was your breach indicator

4 of 17

Now, there is no perimeter in the cloud

Cloud providers own external connections

Cloud is exposed to the outside world

You need to control access to services your team uses

You need to detect �unusual activity

Today, an organization's infrastructure footprint looks a lot different from a castle. It's orders of magnitude more complex!

In fact, you can think of it much more like an amusement park than a castle.

You no longer have physical control over the ways in and the ways out, and there are a lot more of them. Cloud services are exposed to the outside world, and are run by people other than yourself. You don’t just depend on software you install or write, but on network services, and open source and 3rd party software. There are plenty of vectors for bad actors to break in – or be smuggled in!

Controlling access is thus much more complicated than a castle and gate: it’s not a binary condition of "are you in or out", but one made up of teams and roles and permissions, both inside and outside your organization. To make it even more complicated, those roles apply to both humans and computers.

The consequence is that the attack surface has multiplied, and the scope for both accidental or malicious compromise is huge. But the core problem remains, how do you know if you have an intruder?

Without an obvious front door or impregnable walls, how do you detect if bad things are happening?

5 of 17

Without a perimeter, a security camera is more important than a good lock

Watch for changes that create security gaps

Identify intruders and suspicious insider behavior

Send an alert and take immediate action

The solution, of course, is very similar to the real world. We don't live in castles anymore either, so we must do what modern urban planners do. Instead of creating a perimeter that can be locked up, we use sensors and cameras to watch what is going on and let us now when bad things happen. When there’s no simple perimeter to guard, you must monitor your infrastructure and applications to understand if bad things are happening.

You need to know if your systems are changed — even accidentally — in a way that compromises your security. Maybe a permission is flipped in error, and it enable write access by unprivileged users. Or via some platform vulnerability, an intruder gains access and can start a root shell. You want immediate notification in these circumstances, so you can take action.

So, security cameras turn out to be much better assurance than a good lock. Of course, we don’t leave doors open deliberately, but we accept the reality that we cannot protect a perimeter any longer in a way that guarantees security.

We know that security is a multi-layered activity. We have to pay attention all the way from our software supply chain right to the execution environment. For example, we hear a lot today about shifting left, and moving security responsibilities earlier in the design process. However, as the news often shows us, even the best thought-out defense is ultimately fallible. It's simply not economic to rule out vulnerabilities through system design and implementation: our systems are too complex to rely on preventative measures alone.

Instead, the security camera lets us take a behavioral approach. We may not be able to catalog every single bad actor or threat – there are always the famous unknown unknowns – but when we see a bad thing happening, we can do something about it.

This is why threat detection and runtime security is vital for your infrastructure: it's the final and essential line of defense.

6 of 17

The Security Camera for Modern Apps

CNCF INCUBATED PROJECT

created by Sysdig

The exact problem we just talked about was the inspiration for creating the Falco project. "It's 10pm, do you know what your containers are doing?"

Falco was created by Loris Degioanni, borne from his experience in network security gained as a lead developer of Wireshark. Network operations and security are impossible without deep visibility into hosts and traffic, so why not bring this same mentality to monitoring computing endpoints?

Falco aims to be the security camera over hosts, containers and Kubernetes, and cloud services. It is an open source project, hosted under the aegis of the Cloud Native Computing Foundation (CNCF). Loris and Sysdig created and contributed the project to the CNCF, and it now has contributors from many other vendors.

If you’re not familiar with the CNCF, it is an industry foundation created as part of the open-sourcing of Kubernetes by Google, and it is a home to many projects that are part of the modern application infrastructure stack, such as Kubernetes itself, Prometheus and gRPC.

7 of 17

What is Falco?

Runtime security engine
Observability for endpoints and cloud infrastructure
Built on eBPF
Integrated with Kubernetes

CNCF INCUBATED PROJECT

So what is Falco?

It’s a runtime security engine: that means it provides real-time threat detection from within your infrastructure. It supports observability into the endpoints you have: whether Linux hosts, virtual machines, or a container and Kubernetes based infrastructure. Additionally, as we’ll see later, it can monitor cloud infrastructure more broadly.

Falco is built on eBPF, which means it provides a safe and standard way to add kernel-level observability capabilities. This gives Falco the highest degree of visibility possible inside the kernel — down to the syscall. The syscall, or system call, is a very important concept in an operating system kernel, mediating facilities such as file access, network operations, managing processes, and changing permissions. If something is happening, there’s going to be a syscall involved.

(Of course, eBPF isn't everywhere yet, so Falco provides other methods such as kernel modules to provide the same functionality.)

And as is appropriate for a modern security solution, Falco is deeply integrated with Kubernetes. You get an efficient deployment pattern, and added the context you need for understanding exactly where security events are happening inside your clusters. Think about it – your container may only live for a few minutes – it's vital that your security solution can still be effective even though the compute instance could be remarkably short-lived.

8 of 17

About Falco

To give you a nuts and bolts idea of what Falco can do, take a look at a couple of these sample alerts, with different priority levels.

You can see that you can track things such as whether an interactive shell session was created inside a container, or whether a program was spawned in such a way as to enable potentially harmful behavior, in this case remote code execution with netcat.

You can also see that the alert has been augmented with other pertinent information such as the user, full command line, and container identity.

You configure Falco with a series of rules to detect events like these: it receives the raw events from the kernel, and then matches to see if harmful situations are occuring. And of course, Falco ships with a collection of rules that address well-known threats.

9 of 17

The Falco sensor

Let’s have a look at the fundamental concept that Falco uses to drive these alerts, a Sensor. In much the same way that a contemporary physical security environment deploys a range of sensors in important places, so does Falco. The input to each sensor is twofold: the raw events that are occurring in the environment — such as Linux syscalls for example — and a set of rules that tell the sensor when it ought to generate an alert.

The rule language is the engine room of Falco. Under the hood it's a YAML file that contains many instructions to counter common security threats. We already looked at shell spawning, or malicious processes. Other conditions that the default rule set can pick up on include:

adding or changing users
installation of software, or writes to a /bin/ directory
SSH connections to unapproved hosts
addition of cron jobs
attempts to rewrite configuration for well known software such as nginx

One important thing to note here is that the intelligence is pushed to the edge. There is a staggering amount of information generated by any particular computing endpoint, making real time threat detection extremely costly and difficult if you wanted to first collect and pool all the raw data. Instead, Falco is designed to recognize what is interesting as a threat, and forward only those alerts.

These sensors execute on the edge, one per kernel: so in Kubernetes, that’s per-node (not per container or pod), or once in every host or VM.

10 of 17

High level architecture

Sensor

System Calls

Audit Logs

CloudTrail

Alerts

Collector

So if we’re putting together a Falco deployment in practice, what does it look like? Here we can see a variety of sensors that may emit alerts: some from computing nodes, and others from cloud services or Kubernetes. Where those alerts go to next is up to you as the user. That's the role of the "collector" here. By default, Falco can send alerts to files, a syslog, a web service or gRPC endpoint, or another program.

One common collector that is part of the project is called Falco Sidekick. Sidekick offers you a wide variety of integrations enabling you to forward events. So, you can direct them to realtime services such as Slack, or PagerDuty, forward them into a message queue on AWS, or collect them in an S3 bucket, for example.

Falco Sidekick's outputs include chat services, metrics or observability destinations such as Datadog or Prometheus, alerting services, log aggregators, message queues, or even execute serverless functions such as AWS Lambda, Knative or Google Cloud Run.

Being open source and flexible, you can integrate Falco into your broader security and monitoring infrastructure pretty easily. And if you want to share with others, it's not hard to join and contribute to a project such as Falco Sidekick: many others have.

Obviously there's a lot beyond whichever "collector" you implement. So, why and where do we see people installing Falco? Here are some examples:

Falco can provide the intrusion detection system that many compliance standards require, such as SOC2
To provides a base level of visibility into activity in Kubernetes clusters
For fulfilling auditing use cases over network, process, file, activity
Allows easy customization of detections to send events into SIEM/centralized security logging

11 of 17

High level architecture

12 of 17

Beyond system calls and containers

Plugins are dynamic shared libraries

which allow Falco to collect and extract fields

from streams of events

On the previous slide, you saw events coming not just from operating system calls, but cloud infrastructure too. You’re probably wondering how they got there?

Early in 2022, Falco introduced the concept of plugins. A plugin enables you to write an adapter for any event source, which will direct the input to Falco and enable you to write rules to take appropriate responses.

Why are these useful? Let’s go back to our original analogy of the cloud being an amusement park – a collection of multiple diverse services and platforms. The attack surface is complex and spans multiple layers of concern. You can’t pick up a change in a cloud-configured role access via Linux syscall, for example. It's not just about the data in the host kernels anymore, even when you're running Kubernetes.

Falco plug-ins enable us to address this through being able to integrate many external event sources. All you need is a plugin to deliver the events. One particularly powerful thing to note is that you can use your cloud’s logging service as an input – so that can route many data sources that you're interested in, as long as it can write to the logging service.

As an example of this cloud capability, let’s think about GitHub. Many many modern enterprises are now using GitHub for source control, and unfortunately not every developer is as careful as we‘d like. One common anti pattern is a developer accidentally checking in a secret credential to the code base. With Falco’s GitHub plugin you can monitor every commit and immediately alert if something like this happens, or for instance, if a repository is changed from being private to public. It can also monitor GitHub actions, providing protection against increasingly common attacks such as cryptojacking, where a GitHub action can be compromised into (expensively) mining cryptocurrency.

So, with plugins, Falco is a security monitoring approach that lets you secure the runtime defense of not just your application workloads but your cloud SaaS usage as well.

13 of 17

Open source drives effective cloud security

Closes talent gap
Provides rule transparency
Innovation is faster

I noted at the beginning that Falco is open source, and it’s worth taking a look at why that is important for cloud security, and for you as a user.

Modern platforms are built on open source: we live in an era where large parts of computing infrastructure are based on open source platforms, whether it’s Linux or Kubernetes. Having security co-developed with the platforms you use enables not just great integration, but the attention of a broad number of world-class developers working together. How else do you get contribution and review from engineers from places such as IBM, AWS, Shopify or Apple?

With a common standard for threat detection based on open source, you can also close the talent gap for practitioners. In the same way that Kubernetes has defragmented the environment for container orchestration, Falco is doing for runtime security. The knowledge you get from learning the rules and architecture in Falco also applies equally for any commercial solution that is built on the open source standard.

Secondly, open source is transparent. You can see what’s going into Falco, who’s using it, where the project is going, and you can inspect every line you are running in your infrastructure. As a user you can verify for yourself if the software is actually providing you with the protections it is claiming, simply by looking at the rules.

Thirdly, innovation can happen much faster in open source. Falco’s deep integration with Kubernetes was available years before most commercial providers were able to catch up. The bad guys don’t stop thinking up creative ways to attack your infrastructure, and it’s a huge advantage to have a security solution that can move quickly. As they say about open source, many eyes make bugs shallow, there's a huge value to have the collective attention of many of the world's largest engineering organizations using Falco.

14 of 17

Users and builders

CNCF INCUBATED PROJECT

So, you like the idea of Falco, and want to try it out. But you're worried about supportability and sustainability – who else is using Falco, and who is contributing to its maintenance?

Here are just a few of the organizations that are using Falco. In addition to companies who have Falco deployed in their own infrastructure, there are several vendors with security solutions in the market that either ship Falco as part of their product or embed it somehow.

Being open source, Falco encourages this, and works to integrate with many popular platforms and technologies. For example

On AWS, Falco can be used for Fargate runtime security
We worked to integrate Falco into Google's gVisor application sandbox for providing serverless security
IBM's Sysflow telemetry and security solution uses Falco, and
Red Hat's Stackroxx uses Falco libraries for data collection
Sumologic uses Falco for Kubernetes anomaly detection

At Sysdig it won't surprise you to know we use Falco too, having created it! We leverage Falco for the runtime component of our Secure product, and augment its capabilities with enterprise management and the intelligence from our Threat Research team.

15 of 17

Resources

Get started at Falco.org

Check out the Falco project in Github

Get involved in the Falco community

Meet the maintainers on the Falco Slack

Follow @falco_org on

Join a Falco workshop

So if you’d like to know more, what next?

Short answer: try it! That's how open source works.

Documentation and downloads for Falco can be found on Falco.org, and you can take a look at the code itself. The Falco user and contributor community hangs out on Slack, where you can get help if you’re stuck, or get to know how to join the community and help be a builder of Falco.

At Sysdig, we've also created workshops to help you learn Falco. Look out for us at an industry conference near you: we'll keep you posted about new opportunities, and for large teams could schedule a custom workshop.

And of course there’s the O’Reilly book, Practical Cloud Native Security with Falco, which we’ll be sending to you all as a follow-up from this session, as well as all the links on this slide. Thank you so much for joining us, and good luck in your journey in runtime threat detection!

1 of 17

2 of 17

3 of 17

4 of 17

5 of 17

6 of 17

7 of 17

8 of 17

9 of 17

10 of 17

11 of 17

12 of 17

13 of 17

14 of 17

15 of 17

16 of 17

17 of 17