D-Squared

Subscribe

What is it? 

“eBPF does to Linux Kernels what Javascript does to HTML” – Brendan Gregg, Netflix. 

👆I.e. Makes the kernel programmable, without needing to create completely new kernel modules. 

We’re now able to wedge a program (eBPF) in between the user world (user space) and kernel world (kernel space) to observe and act on all kinds of events happening on your PC. 

eBPF manifested around 2014, but the old school version “BPF” without the “e” was created back in 1992. 

The acronym “eBPF” stands for extended Berkley Packet Filter. Sadly, this is a horrible name for this tool, since it does much more than filter packets. 

The extension part of this has enabled developers to hook onto specific locations within the kernel extending our ability to observe and act. 

Most people (likely you) will not program your own eBPF programs, instead, you’ll pull a tool from an ecosystem of tools being built out or you’ll use it unknowingly (Google-managed Kubernetes – GKE). 

There are tons of interesting use cases, which mainly fall under three big umbrellas (observation, networking, and security). We’ll mainly focus on security use cases, here’s a taster. It’s possible to observe any bash command entered anywhere in the system.

Below is a good little comic highlighting the reason why we prefer to “eBPF” approach over altering the kernel directly (Pg. 8). 😀

How Does it Function? 

Compile C: Compile a restricted version of C into byte code. “Restricted” is referring to limited program size, looping ability, available functions, etc.   

Kernel Load: Not always the case, but most of the time the byte code is loaded into the kernel via a library such as Cilium, bcc, oxideBPF, libbpfgo, Aya, etc.

Verify: The verifier’s job is to ensure that the eBPF program is safe to run. Checking for infinite loops, out-of-bound/unsafe memory operations, and within the predefined restraints of complexity/code size.

JIT Compiler: Compile the generic byte code to CPU native byte code (i.e. x86). Now you’re able to execute a program that runs at the same speed as natively compiled code. 

Attach: eBPF attaches itself to the right place, so we’re able to observe events that we’ve “hooked” our eBPF program to and send data to the app in user space to action (if needed).  

  • Uprobes (user space), kProbes (kernel space), tracepoints (predefined locations only)

Example: Below is a more detailed example of an eBPF program monitoring a specific syscall “open()”. Once open() is called on a file our eBPF program jumps into action and sends the event data to an “events map”, which is picked up by our helper tool “opensnoop”.

  • Side note: “events map” is basically a way to maintain state in-between different events we’re tracking with our eBPF program, as well as pass data in-between user/kernel space acting as a translation layer.

Flow: Life-cycle of an eBPF program

Why does it matter?

Our security tooling needs to keep up with a world that’s being gobbled up by the cloud-native ecosystem. Traditional security detection tools revolve around IP addresses and are incapable of monitoring processes within a kernel namespace. That’s no good., 

Kubernetes and containers are what the cool kids are doing today, which means the IP addresses to the containers in each cluster are ephemeral (or should be). If the IPs are popping in and out of existence, we need a tool that can identify and monitor each container at a deeper level. 

In walks in eBPF…

In this chart, we can see the difference between observability for old-school monitoring and eBPF monitoring. 

What are the security use cases?

Catching Crypto-jackers

In a cloud-native world, many attackers are hijacking computation from their victims and mining crypto for a quick buck. The ability to spin up an asset and start mining has never been easier thanks to Kubernetes and containerization. As a defender, our ability to detect this “cryptojacking” behavior is critical to avoid the outsized cost and reputational damage. eBPF can help us here (i.e. Faclo). 

Here’s the scenario (Pg. 32) – “Imagine that an attacker manages to compromise one of your hosts and starts a separate pod to run, say, a cryptocurrency miner. They are unlikely to do you the courtesy of instrumenting their mining pod with your sidecar observability or security tools. You’ll need a separate system to be aware of this activity.”

Observation-based Policies (source)

This is when we’re able to directly translate observed events to security policies based on real-world behavior. The recommended approach is to monitor a freshly initiated application, then whitelist all the behaviors, blocking everything else. 

Similar to “seccomp”, but improved. With eBPF we’re able to limit specific process IDs (PIDs) to certain files, kind of like RBAC policies for PIDs and files. 

For example with Cilium Tetragon we can block every container (or Linux process) that starts with the CAP_SYS_ADMIN capability with the deny-privileged-pod.yaml tracing policy.

When a pod is created and includes CAP_SYS_ADMIN, it’s essentially the “new root” in Linux and gives direct access to the node. This gives us access to breaking out all container namespaces and exploiting processes or filesystems on the underlying node where the privileged pod is deployed.

DDoS Prevention (Pg. 41)

“One of the early uses of eBPF in production at scale was for DDoS protection at Cloudflare. A DDoS attacker floods a target machine with many network messages, hoping that the target can’t process them quickly enough and becomes so busy handling these messages that it can’t do useful work. Cloudflare engineers use eBPF programs to examine packets as soon as they arrive, and quickly determine whether a packet is part of such an attack, discarding them if so. The packet doesn’t have to pass through the kernel’s networking stack so it takes far fewer resources to process. The target can cope with a much higher rate of malicious traffic.”

Our eBPF program, in this case, is sitting directly on a network interface card (NIC) thanks to special hooks provided by “eXpress Data Path” or XDP. 

In a test run by Cloudflare back in 2018, they found that eBPF hooked onto a NIC can drop 10 million packets per second. Much faster than alternative options (see below). I’m sure by now (2023) they’ve engineered something even faster. 🤯 

Deceiving sandboxes

As a defender, you could create a malware sandbox with eBPF programs sprinkled throughout that provide false responses to sophisticated malware variants. When malware tries to figure out if it’s in a sandbox or not you’re able to deceive it by tricking each of the checks that usually run (example list below). 

What are the known malicious use cases?

Disclaimer! Most of the malicious use cases mentioned below assume that the attacker has already gained root access to a machine. For example, purchasing credentials via the dark web, using a vulnerability to exploit an internet-facing machine, etc. 

Highly recommend you watch this presentation (demo examples) to get a better understanding of how these superpowers can be manipulated for those with admin access. 

The main highlights would be… 

  1. Data Illusions 🪄: Alter the data for a specific shell when “catting” out data for a file and changing what that data says on the fly. An eBPF program is looking for a specific process ID (PID) and whenever that PID runs a specific command (i.e. cat), eBPF intercepts the data from the file, alters it, then sends it back to that shell. 
  2. Invisible Root 🧙: A non-root user can become root without having to use a password or anything else. A specific eBPF program is deployed and looks for a specific user and every time that user inputs “sudo” they’re immediately given root privileges. The interesting thing here is that when checking their privileges from a separate session they show not having “root”.
  3. You can’t see me 🫥: By setting up a specific eBPF program the defender won’t be able to audit existing eBPF programs running. Every time the defender checks to see what eBPF programs are running with the “bpftool” it immediately kills that bpftool, blinding the defender. 

Resources in the recommended order for n00bs