BPF token support and testing infrastructure by dylandreimerink · Pull Request #1953 · cilium/ebpf

dylandreimerink · 2026-02-17T15:10:34Z

This PR introduces test utilities that allows us to write tests for BPF tokens. This RunWithToken utility function is largely based on the test code from the kernel selftests but modified to work with Go and its runtime.

The individual commit messages contain more details. The high level is that we can write tests (and subtests) where we start by calling RunWithToken with a set of delegated permissions. The rest of the test body will be executed in an environment with a BPFFS mounted with those permissions.

This PR also includes BPF token support. This takes the "ambient token" approach, where we do not add any new API surface, but we automatically use tokens when they are available. This allows us to experiment with tokens without committing to an API. We can always add more controls in the future.

Instead of relying on hardcoding search paths or environment variables, this implementation scans the /proc/self/mountinfo procfile to find all BPFFS instances mounted in the processes mount namespace. This means we are able to use any BPFFS path, and even multiple BPFFS'es, for example if one FS has no permissions but holds pinned objects, and a secondary BPFFS is used to the delegated tokens. However, there is no method to select a preferred one, if multiple BPFFS'es with delegated privileges exist (which should be uncommon).

A nice side effect of reading the mount info is that we also know which permissions are delegated which is used in this implementation to enhance error messages when using tokens but the token lacks certain permissions.

The main purpose of this PR was to implement the testing utilities, and the PR is structured such that we can drop the actual token implementation if any of the other proposals is more appealing then this one.

Note: This PR currently has cherry-picked a commit from #1950 for its capability syscalls. This PR should be rebased once #1950 is merged.

javiercardona-work · 2026-02-17T22:18:30Z

Thanks for doing this. If you end up landing this, I will withdraw #1948.

I tested this PR in our environment, and it works as documented. In case it can be useful, I'm attaching the test program we used: ambient_token_test.go.zip

This runs in our namespaced container with CAP_BPF permissions and a delegated BPF mountpoint (/run/tw/bpf). The mountpoint is automatically detected:

$ /tmp/ambient-token-test            
ambient-token-test v7

=== Diagnostics ===
  user namespace: user:[4026535428]
  unprivileged_bpf_disabled: 1

  Capabilities before:
  CapInh:       0000000000000000
  CapPrm:       000001c7a9ac7dff
  CapEff:       000001c7a9ac7dff
  CapBnd:       000001c7a9ac7dff
  CapAmb:       0000000000000000

=== Token Create ===
  token.Create(): OK (fd=6, mount=/run/tw/bpf)

=== Feature Probes (ambient token) ===
  prog SocketFilter         supported
  prog Kprobe               supported
  prog SchedCLS             supported
  prog XDP                  supported
  prog TracePoint           supported
  prog CGroupSKB            supported
  map  Hash                 supported
  map  Array                supported
  map  PerfEventArray       supported
  map  LRUHash              supported
  map  RingBuf              supported

=== Map Create (ambient token) ===
  map create: OK (fd=3)

=== Program Load (ambient token) ===
  program load: OK (fd=3)

Done.

When running it without CAP_BPF, token creation and all subsequent bpf operations fail, as expected:

$ /tmp/ambient-token-test --drop-caps
ambient-token-test v7

=== Diagnostics ===
  user namespace: user:[4026535428]
  unprivileged_bpf_disabled: 1

  Capabilities before:
  CapInh:       0000000000000000
  CapPrm:       000001c7a9ac7dff
  CapEff:       000001c7a9ac7dff
  CapBnd:       000001c7a9ac7dff
  CapAmb:       0000000000000000

  Dropping CAP_BPF, CAP_SYS_ADMIN, CAP_PERFMON...
  Capabilities after:
  CapInh:       0000000000000000
  CapPrm:       00000107a98c7dff
  CapEff:       00000107a98c7dff
  CapBnd:       00000107a98c7dff
  CapAmb:       0000000000000000

=== Token Create ===
  token.Create(): BPF Token is not available

=== Feature Probes (ambient token) ===
  prog SocketFilter         detect support for SocketFilter: load program: operation not permitted
  prog Kprobe               detect support for Kprobe: load program: operation not permitted
  prog SchedCLS             detect support for SchedCLS: load program: operation not permitted
  prog XDP                  detect support for XDP: load program: operation not permitted
  prog TracePoint           detect support for TracePoint: load program: operation not permitted
  prog CGroupSKB            detect support for CGroupSKB: load program: operation not permitted
  map  Hash                 detect support for Hash: operation not permitted
  map  Array                detect support for Array: operation not permitted
  map  PerfEventArray       detect support for PerfEventArray: operation not permitted
  map  LRUHash              detect support for LRUHash: operation not permitted
  map  RingBuf              detect support for RingBuf: operation not permitted

=== Map Create (ambient token) ===
  map create: FAILED (creating map: map create: operation not permitted (MEMLOCK may be too low, consider rlimit.RemoveMemlock))

=== Program Load (ambient token) ===
  program load: FAILED (load program: operation not permitted (MEMLOCK may be too low, consider rlimit.RemoveMemlock))

Done.

anryko

Thank you for working on this!

internal/testutils/token_linux.go

map.go

prog.go

internal/testutils/token_linux.go

internal/token/token.go

dylandreimerink · 2026-02-23T14:30:12Z

Rebased on main and updated to match changes made in #1950. Marking ready for review.

javiercardona-work

The updated version of this branch still passes our tests in our containerized environment.

[09:47:25 root@tsp_prn/mks/mks-system-pool-jcardona-sparklemuffin-kubelet/0 ~]$ /tmp/ambient-token-test 
ambient-token-test v9

=== Diagnostics ===
  user namespace: user:[4026535252]
  unprivileged_bpf_disabled: 1

  Capabilities before:
  CapInh:       0000000000000000
  CapPrm:       000001c7a9ac7dff
  CapEff:       000001c7a9ac7dff
  CapBnd:       000001c7a9ac7dff
  CapAmb:       0000000000000000

=== Token Create ===
  token.Create(): OK (fd=6, mount=/run/tw/bpf)

=== Feature Probes (ambient token) ===
  prog SocketFilter         supported
  prog Kprobe               supported
  prog SchedCLS             supported
  prog XDP                  supported
  prog TracePoint           supported
  prog CGroupSKB            supported
  map  Hash                 supported
  map  Array                supported
  map  PerfEventArray       supported
  map  LRUHash              supported
  map  RingBuf              supported

=== Map Create (ambient token) ===
  map create: OK (fd=3)

=== Program Load (ambient token) ===
  program load: OK (fd=3)

Done.

internal/testutils/token_linux.go

javiercardona-work · 2026-03-02T17:43:44Z

@rgo3 I there something I can do to help merge this PR? Happy to run additional tests on my environment.

dylandreimerink · 2026-03-03T13:47:49Z

I there something I can do to help merge this PR?

We are waiting for @ti-mo to review, since I made the PR, and he is the only other project maintainer. But Timo is a busy man, with a very long todo list, so this just takes some time.

All reviews are welcome, but Timo's approval determines merge-ability of the PR.

javiercardona-work · 2026-03-11T15:51:46Z

Hi @ti-mo,

Could you provide a rough estimate on when this might be ready for merge? I’m curious if there are any outstanding design concerns we should address, or if it’s primarily a matter of finding an opening in your schedule. Our team is excited to move away from our internal fork and adopt @dylandreimerink better implementation for this feature, so we’re just looking to coordinate our timeline.

Thanks!
Javier

ti-mo · 2026-03-13T16:15:05Z

Hi Javier,

This PR is the result of internal discussions, so don't expect any departures from the overall architecture, if that makes things a bit more predictable for you.

For the initial implementation, we want there to not be any API changes, which should make this branch easy to carry in your Cilium fork since there should be no token-related Cilium changes required. I'll start working on this on Monday and it will be merged when it's done.

ti-mo

Thanks! This is all I got for now, will do a 2nd round when these are discussed/addressed.

internal/testutils/token_linux.go

internal/cmd/gentypes/main.go

internal/token/token.go

internal/testutils/token_linux.go

internal/sys/types.go

dylandreimerink · 2026-03-19T16:08:39Z

I figured out that you can request object info on tokens as well, which returns the delegated permissions as numbers, which means we don't have to have all of the from string and parsing logic. I still need to address a few of the outstanding comments with regards to changing the API of the tokens testutils. Just pushing what I have so far.

Setting up the environment to be able to use BPF tokens is a bit involved. This commit adds the `RunWithToken` utility function to do this setup. Using tokens requires a few things: * The process must run in a user namespace that is not the initial user namespace of the system. * The process must have the CAP_BPF capability in that user namespace. * A BPFFS must be available that has been mounted with delegated permissions. * The BPFFS must be owned by the user namespace of the process. In addition we want to run some tests with tokens and other without. Some actions involved in setting up the environment for tokens are irreversible, so we have to spawn a subprocess to run the tests with tokens. The `RunWithToken` helper does all of the required setup. It creates a UNIX socket pair to communicate with the subprocess, then executes a new instance of the current test binary. The new process is executed in its own user NS and mount NS. Only the current test is executed in the subprocess. An environment variable is used to detect if we are executing in a subprocess or not. The parent will wait for a BPFFS context to be send over the socket pair and will delegate the permissions specified as arguments to `RunWithToken`. When the subprocess first starts, it check the env var, and does the child side of the setup. It creates a BPFFS context, and sends it to the parent via the UNIX socket. After getting confirmation that the delegation is done, the child mounts the BPFFS, drops capabilities, and does a execve to re-execute itself. This ensures that in the final stage tokens are setup, even during init and global variable initialization. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>

This commit changes a few things to do with type generation. First the addition of type generation for the token create command and the TokenInfo type. Second is that we generate if any enum has a value with the __MAX prefix, we will remove the underscores so this max value can be used to iterate over all valid enum values. Lastly, we add stringer generation for the Cmd, MapType, ProgType and AttachType enums, so these can be used in error messages. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>

This commit adds logic to create BPF tokens. We parse the mountinfo proc file to discover all bpffs mounts. We then attempt to create a token from each mount until we find one that works. This means that we do not have to hard code any paths or pass them via environment variables. A side effect is that parsing the mountinfo tells us exactly what the delegated permissions are of the token. This allows us to create more informative error messages or to skip trying to execute syscalls we know will fail. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>

This commit adds token support for programs. It uses the "ambient" token approach, which means that the token is not explicitly passed by a user but instead we always attempt to create a token when loading programs and fall back to loading without a token if token creation fails. This commit also adds rich errors for token related failure. If loading a program fails with EPERM, and without a verifier log we check if the token is missing a BPF command, program type, or attach type which would explain the failure. If we find an issue, we enrich the error to assist users in debugging token related issues. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>

This commit adds token support for maps. It uses the "ambient" token approach, which means that the token is not explicitly passed by a user but instead we always attempt to create a token when loading maps and fall back to loading without a token if token creation fails. This commit also adds rich errors for token related failure. If loading a map fails with EPERM we check if the token is missing a BPF command or map type which would explain the failure. If we find an issue, we enrich the error to assist users in debugging token related issues. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>

This commit adds token support for BTF. It uses the "ambient" token approach, which means that the token is not explicitly passed by a user but instead we always attempt to create a token when loading BTF and fall back to loading without a token if token creation fails. This commit also adds rich errors for token related failure. If loading a BTF fails with EPERM we check if the token is missing the BPF command permission which would explain the failure. If we find an issue, we enrich the error to assist users in debugging token related issues. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>

This commit changes `RunWithToken` to use a subtest API style. This is more intuitive to use, over the old style where the caller had to do an early return if `RunWithToken` returned true. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>

dylandreimerink force-pushed the feature/tokens-with-tests branch 2 times, most recently from e80d8a8 to fa9614c Compare February 17, 2026 15:43

dylandreimerink force-pushed the feature/tokens-with-tests branch 2 times, most recently from 8f7f344 to c2149ea Compare February 18, 2026 12:51

anryko reviewed Feb 19, 2026

View reviewed changes

anryko mentioned this pull request Feb 19, 2026

Adding support for privilege delegation using BPF Token #1921

Open

javiercardona-work mentioned this pull request Feb 19, 2026

Add BPF token support for program and map loading #1948

Closed

dylandreimerink force-pushed the feature/tokens-with-tests branch 2 times, most recently from 50c6abc to 0209971 Compare February 20, 2026 12:19

ti-mo changed the title ~~Add BPF token testing infa and BPF token support~~ BPF token support and testing infrastructure Feb 23, 2026

dylandreimerink force-pushed the feature/tokens-with-tests branch 2 times, most recently from d29e5d5 to a416a64 Compare February 23, 2026 14:22

dylandreimerink marked this pull request as ready for review February 23, 2026 14:30

dylandreimerink requested review from a team and rgo3 as code owners February 23, 2026 14:30

javiercardona-work approved these changes Feb 23, 2026

View reviewed changes

internal/testutils/token_linux.go Show resolved Hide resolved

internal/testutils/token_linux.go Show resolved Hide resolved

ti-mo reviewed Mar 17, 2026

View reviewed changes

ti-mo mentioned this pull request Mar 18, 2026

Add ErrNotPermitted error for permission denied in feature probes #1949

Open

dylandreimerink force-pushed the feature/tokens-with-tests branch 2 times, most recently from 1bff0ee to cdf4dd8 Compare March 19, 2026 16:03

dylandreimerink added 3 commits March 20, 2026 11:45

dylandreimerink added 4 commits March 20, 2026 12:49

dylandreimerink force-pushed the feature/tokens-with-tests branch from cdf4dd8 to 2da766f Compare March 20, 2026 11:51

dylandreimerink requested a review from ti-mo March 20, 2026 12:04

Conversation

dylandreimerink commented Feb 17, 2026

Uh oh!

javiercardona-work commented Feb 17, 2026

Uh oh!

anryko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dylandreimerink commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

javiercardona-work left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

javiercardona-work commented Mar 2, 2026

Uh oh!

dylandreimerink commented Mar 3, 2026

Uh oh!

javiercardona-work commented Mar 11, 2026

Uh oh!

ti-mo commented Mar 13, 2026

Uh oh!

ti-mo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dylandreimerink commented Mar 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dylandreimerink commented Feb 23, 2026 •

edited

Loading