Sitemap
Towards Dev

A publication for sharing projects, ideas, codes, and new theories.

Follow publication

Supercharge Linux Observability with eBPF: A Modern Approach

--

eBPF (Extended Berkeley Packet Filter) has evolved into a key technology in modern observability. It offers the ability to gather deep insights into Linux systems without the need for significant overhead. Traditional Linux tools like top, htop, and strace can provide information, but they often miss the full picture and may require invasive techniques that impact performance. eBPF allows us to observe, monitor, and trace applications and kernel events with precision, all without making changes to code or causing disruption.

This article explores how you can supercharge your Linux observability using eBPF, along with practical examples to show how it’s implemented in real-world scenarios.

What is eBPF?

eBPF is a powerful technology built into the Linux kernel that enables users to run custom programs in response to events such as system calls, network packets, and file system changes. These programs are run in a sandboxed environment within the kernel, ensuring they are safe and don’t crash the system.

Historically, BPF was created to filter network packets efficiently. Over time, this technology has been extended to a wide range of kernel and user-space monitoring. Developers gather detailed insights into system performance, security, and application behavior without requiring intrusive agents or modifying the underlying application by using eBPF (Extended Berkeley Packet Filter).

Why Use eBPF for Observability?

  • Low Overhead: eBPF runs within the kernel and gathers data without interrupting running processes, making it ideal for production environments.
  • Deep Insights: It can track system calls, network packets, I/O events, and other kernel activities, providing more granular details than traditional monitoring tools.
  • Dynamic Instrumentation: You can attach eBPF programs to live kernel events and tracepoints dynamically.

With these features, eBPF has become a cornerstone for modern observability and performance monitoring.

Examples of Using eBPF for System Monitoring

One of the most common use cases for eBPF is system monitoring. eBPF can trace system calls, measure performance metrics, or analyze I/O operations. Let’s dive into an example where we monitor file system access.

Example: Monitoring File Access

In this example, we will use an eBPF program to monitor open() system calls, which are triggered whenever a process attempts to open a file.

We will use Python and the bcc (BPF Compiler Collection) to write and run our eBPF program. First, ensure you have bcc installed:

sudo apt-get install bpfcc-tools linux-headers-$(uname -r)
pip install bcc

Here’s the eBPF code that monitors file accesses:

from bcc import BPF

# Define eBPF program to trace file open system call
bpf_program = “””
int trace_open(struct pt_regs *ctx, const char __user *filename, int flags) {
bpf_trace_printk(“File opened: %s\\n”, filename);
return 0;
}
“””

# Attach eBPF program to the 'open' system call
bpf = BPF(text=bpf_program)
bpf.attach_kprobe(event="sys_open", fn_name="trace_open")

print("Tracing file open system calls… Hit Ctrl-C to end.")

# Print output from eBPF
while True:
try:
bpf.trace_print()
except KeyboardInterrupt:
exit()

Explanation:

  1. Defining the eBPF Program: We use C syntax inside the bpf_program string to define a program that attaches to the sys_open system call. Whenever a file is opened, this program logs the file name.
  2. Attaching to a Kernel Probe: The attach_kprobe() function attaches the eBPF program to the kernel probe for the sys_open system call, so every time a file is opened, the eBPF program is triggered.
  3. Printing Output: We continuously monitor the output of the trace, printing it to the console.

This small script shows how easy it is to hook into kernel-level events using eBPF, allowing us to monitor file operations with minimal overhead.

Practical Usage

This type of monitoring is particularly useful for debugging issues in production environments. For example, you could use this to detect which files are being accessed when a particular process is slow or misbehaving. With eBPF, you get real-time insights without stopping the process or modifying the application.

Examples of Security Monitoring with eBPF

eBPF also plays a significant role in enhancing system security by providing fine-grained insights into potentially malicious activity. You can use eBPF to trace suspicious system calls, track unusual network traffic patterns, or even monitor file access behavior to detect unauthorized access.

eBPF can be used to create a security auditing system that operates at the kernel level, providing a powerful defense mechanism against threats. Below is an example of how you can leverage eBPF for security monitoring by tracing suspicious execution of binaries.

Example: Monitoring Suspicious Executions

In this example, we will monitor the execution of binaries in /tmp, which is a common target for attackers trying to execute malicious scripts.

from bcc import BPF

# Define eBPF program to trace execve system calls
bpf_program = “””
int trace_execve(struct pt_regs *ctx, const char __user *filename) {
if (bpf_strncmp(filename, “/tmp”, 4) == 0) {
bpf_trace_printk(“Suspicious execution: %s\\n”, filename);
}
return 0;
}
“””

# Attach eBPF program to 'execve' system call
bpf = BPF(text=bpf_program)
bpf.attach_kprobe(event="sys_execve", fn_name="trace_execve")

print("Monitoring executions in /tmp… Hit Ctrl-C to end.")

# Print output from eBPF
while True:
try:
bpf.trace_print()
except KeyboardInterrupt:
exit()

Explanation:

  1. Tracing the execve System Call: The execve system call is used when executing binaries in Linux. The eBPF program hooks into this system call to monitor any process attempting to execute a binary.
  2. Monitoring /tmp: The program checks whether the binary being executed is located in the /tmp directory. If it is, it logs it as a “suspicious execution,” since /tmp is often a temporary location for untrusted files or scripts.

Examples of Network Observability with eBPF

eBPF can also provide deep insights into network traffic by attaching to network events, sockets, or packet-processing functions. This makes it perfect for monitoring network performance, tracing packet flows, and detecting anomalies in real-time.

Example: Tracing Network Connections

Let’s consider another example where we use eBPF to trace outgoing TCP connections made by applications on the system. This is useful for tracking which applications are establishing connections and where the connections are going.

from bcc import BPF

# Define eBPF program to trace TCP connect
bpf_program = “””
int trace_connect(struct pt_regs *ctx, struct sock *sk) {
u32 pid = bpf_get_current_pid_tgid() >> 32;
bpf_trace_printk(“PID %d made a connection\\n”, pid);
return 0;
}
“””

# Attach eBPF program to the 'tcp_connect' tracepoint
bpf = BPF(text=bpf_program)
bpf.attach_kprobe(event="tcp_v4_connect", fn_name="trace_connect")

print("Tracing TCP connections… Hit Ctrl-C to end.")

# Print output from eBPF
while True:
try:
bpf.trace_print()
except KeyboardInterrupt:
exit()

Explanation:

  1. Tracing TCP Connections: The tcp_v4_connect tracepoint is triggered whenever an application attempts to create an outgoing TCP connection. We use the trace_connect function to log the process ID (PID) of the application initiating the connection.
  2. Real-time Monitoring: This program continuously monitors outgoing connections and logs them in real-time. It’s useful for detecting which processes are trying to access the network and could help identify malicious behavior or unintentional traffic.

This kind of network observability is invaluable for security monitoring and performance troubleshooting. Whether you’re trying to optimize network usage or detect unauthorized access, eBPF gives you the detailed insights necessary.

Examples of Performance Monitoring with eBPF

Another key use case for eBPF is performance monitoring. eBPF can track CPU usage, memory allocation, disk I/O, and more. Since eBPF operates at the kernel level, it can provide system-wide performance metrics without affecting the applications running on the system.

Example: Measuring CPU Usage per Process

Here’s an example where we measure CPU usage for individual processes:

from bcc import BPF

# Define eBPF program to track CPU time
bpf_program = """
BPF_HASH(start, u32);

int trace_start(struct pt_regs *ctx) {
u32 pid = bpf_get_current_pid_tgid();
u64 ts = bpf_ktime_get_ns();
start.update(&pid, &ts);
return 0;
}

int trace_finish(struct pt_regs *ctx) {
u32 pid = bpf_get_current_pid_tgid();
u64 *tsp = start.lookup(&pid);
if (tsp != 0) {
u64 delta = bpf_ktime_get_ns() - *tsp;
bpf_trace_printk("PID %d used %llu ns of CPU time\\n", pid, delta);
start.delete(&pid);
}
return 0;
}
"""


# Attach to scheduler functions for process start and finish
bpf = BPF(text=bpf_program)
bpf.attach_kprobe(event="sched_switch", fn_name="trace_start")
bpf.attach_kretprobe(event="sched_switch", fn_name="trace_finish")

print("Measuring CPU usage per process… Hit Ctrl-C to end.")

# Print output from eBPF
while True:
try:
bpf.trace_print()
except KeyboardInterrupt:
exit()

Explanation:

  1. Tracking Process Start and Finish: We attach the eBPF program to the sched_switch function, which is triggered whenever the scheduler switches processes. The trace_start function records the start time for a process, and the trace_finish function calculates how much CPU time was used.
  2. Output: This program logs the amount of CPU time used by each process. It’s useful for identifying resource-intensive processes and understanding system performance at a granular level.

This approach to performance monitoring helps system administrators and developers pinpoint bottlenecks and optimize resource usage.

Conclusion

eBPF’s ability to trace system calls, monitor network activity, and measure performance metrics — without the overhead of traditional methods — makes it an essential tool for modern infrastructure and application monitoring.

Whether you are debugging performance issues, optimizing system resources, or improving security, eBPF offers a low-overhead, highly flexible way to gain deep insights into your systems. The practical examples provided demonstrate how easily you can integrate eBPF into your workflows, giving you real-time observability with minimal impact on system performance.

--

--

Towards Dev
Towards Dev

Published in Towards Dev

A publication for sharing projects, ideas, codes, and new theories.

Supratip Banerjee
Supratip Banerjee

Written by Supratip Banerjee

Supratip is a Sol Arch and thought leader in technology space. His specializations are enterprise application design, DevOp and Cloud. He's a AWS Comm Builder.

No responses yet

Write a response