Two technical questions towards GPU/CDI support: AppArmor and manifest capabilities

[LoudLemur]

We are working on a concrete, evidence-backed proposal for GPU (Graphical Processor Unit) device access for apps, following on from the long-running request in https://forum.cloudron.io/topic/12401 and the CDI (Container Device Interface) (groundwork described there.

The short version of the context: CDI has been enabled by default in Docker since 28.2.0, the docker-ce 29.6.2 that the current installer pins now routes even its own --gpus flag through CDI for both NVIDIA and AMD, and I have working proofs of a real GPU via CDI with the NVIDIA toolkit and of stock docker-ce 29.6.2 consuming a synthetic CDI spec through the same Engine API payload shape dockerode sends.

Writing the design up against the box source left us with a small number of questions that only Team Cloudron can answer:

Question 1: the docker-cloudron-app AppArmor profile.

A CDI spec injects host paths into the container at create time, typically the driver libraries and a few binaries such as nvidia-smi, as bind mounts described by the spec.

Does the docker-cloudron-app profile (setup/start/docker-cloudron-app.apparmor in the box source) permit reads and mmap of libraries mounted in from host driver paths, or would it deny them?

We can test this empirically on our own hardware if that is useful, but the profile's intended scope is something only its authors can state, and it determines whether a GPU grant works under the standard app confinement or needs a profile amendment.

Question 2: where manifest capabilities values are validated.

Reading the box source, manifest.capabilities appears to be consumed only in src/docker.js, with unrecognised values silently ignored, which would make adding a new capability name purely additive.

Is there a whitelist elsewhere, for example in the CLI's manifest validation or in app store submission checks, that would reject a manifest declaring an unknown capability such as gpu?

Question 3: which shape would you prefer, if any?

The two natural shapes I found are, first, widening the existing per-app Devices feature (the dashboard Resources form backed by setDevices) so it accepts CDI device names like nvidia.com/gpu=all alongside /dev/ paths, and second, a new manifest capability mirroring how vaapi is implemented, with a guard so it no-ops on hosts without a CDI spec.

The first reuses the persistence, REST, dashboard, audit and recreate machinery unchanged and keeps the decision with the administrator rather than the packager; the second matches the existing capability precedent.

If the team has a preference, or a reason both are wrong, knowing it before anything is written up further would save everyone time.

We have a full design sketch with source references, failure modes and non-goals, plus reproducible test transcripts, and are happy to share any of it; there is also an Nvidia machine available for any test the team would find persuasive.

[girish]

Does this mean we don't need to install nvidia's special docker anymore for the GPU integration to work?

For 1. Not sure, you will have to try None of our apps so far need that.

For 2,3. Correct, unknown values are ignored. I guess we can add a capability called gpu yeah similar to vaapi

[LoudLemur]

@girish

Thank you, that is what we needed.

Your question: no, nvidia's special docker is not needed any more

• Gone: the nvidia runtime registration in /etc/docker/daemon.json and --runtime=nvidia. This is precisely what CDI (Container Device Interface) replaces. Nothing on the Cloudron side would touch the daemon configuration.
• Still needed, and still the administrator's job: the NVIDIA driver, and the nvidia-container-toolkit package. nvidia-ctk cdi generate writes the specification file, and the generated specification calls /usr/bin/nvidia-cdi-hook from that same package when a container starts.
• New since the 2024 discussion: the toolkit keeps the specification current by itself. Version 1.19 ships nvidia-cdi-refresh.path, a systemd path unit that regenerates it whenever the driver or the toolkit is upgraded. The refresh-on-driver-update automation that made this look expensive in 2024 is now upstream's, not yours.
• Left on the Cloudron side: one HostConfig.DeviceRequests entry, { "Driver": "cdi", "DeviceIDs": ["nvidia.com/gpu=all"] }, plus a guard so the capability does nothing on a host with no specification, in the spirit of vaapi's check for /dev/dri.

Question 1, AppArmor: we tried, as suggested, and it passes

We built an Ubuntu 24.04 virtual machine, installed docker-ce 29.6.2 and containerd 2.2.6 as your installer pins them, loaded docker-cloudron-app verbatim from the box source, and ran a container in the shape the box creates: confined by the profile, read-only root filesystem, CDI device attached.

The specification was synthetic, since there is no GPU (Graphics Processing Unit) in the virtual machine, but it reproduces what the profile actually mediates: driver libraries and a binary bind mounted in from host paths, a device node the image does not have, and a createContainer hook that writes into the container root filesystem.

• Reading, executing and mmapping the injected files: all fine. The injected library was loaded through LD_PRELOAD and appears as five mapped regions in /proc/self/maps, which is the operation the question was really about.
• The injected device node opened and accepted a write.
• Zero kernel denials. dmesg | grep apparmor.*DENIED is empty after every run.
• Identical results unconfined, so the profile changes nothing about CDI injection.
• Confirmed with --user 1000:1000, since app processes are not root, and /proc/self/attr/current inside the container reads docker-cloudron-app (enforce), so the profile really was applied.

What is not proven: no real GPU was involved, so the ioctls nvidia-smi performs on /dev/nvidia* are formally untested. Everything the profile mediates, which is file access, is covered. Full transcript in the next post.

ReadonlyRootfs also passes, which was the sharper risk

An NVIDIA specification does not only bind mount libraries. It runs createContainer hooks that write into the container root filesystem, creating the libcuda.so.1 style symlinks and running ldconfig.

• Result: the hooks succeed on overlayfs, under the profile, as an unprivileged user, while the container process itself is still refused writes on the same filesystem.
• Why: the runtime applies the read-only remount after the createContainer hooks have run. No writable-rootfs carve-out is needed for app containers.

The presence guard turns out to be one API call

vaapi can use existsSync('/dev/dri'), but a CDI name is a lookup key rather than a path, which looked like the one piece of genuinely new logic. It is not:

docker info --format '{{json .DiscoveredDevices}}'
[{"Source":"cdi","ID":"example.com/gpu=all"}]

The same field comes back from GET /info on the socket, so from the box it is a docker.info() call and a search for an entry whose Source is cdi. No directory scanning and no specification parsing. It is available from Engine API 1.50; the daemon under test reported 1.55.

On the name, we would suggest gpu rather than nvidia

• AMD's container toolkit now generates CDI specifications too, under amd.com/gpu, and Docker 29.3.0 routes its own --gpus flag through CDI for AMD, as 29.2.2 did for NVIDIA.
• A single gpu capability, guarded by a check for any GPU device in the loaded specifications rather than for the NVIDIA vendor prefix, would cover a second vendor with no further code.

---

Open questions for you

1. Merge request, or specification first? We can prepare a patch against master limited to the guard, the DeviceRequests entry in src/docker.js and the manifest documentation. If you would rather read a written specification before reviewing code, we will send that instead.
2. Is gpu the name you want? You suggested it, we agree, and the AMD point above is our reason for preferring it over nvidia. Worth confirming before it becomes a documented manifest key.
3. Silent no-op, or a visible error? vaapi silently does nothing on a host without /dev/dri. We would mirror that for gpu, but an app that quietly runs without its GPU may be worse than one that refuses to start. Your call, since it sets the precedent.

There is an RTX 5090 machine here for any test that would help, and the full design sketch with source references and test transcripts is available whenever it is useful.

If anyone with a GPU box wants to confirm the AppArmor result against real hardware, it is three commands and we have written them out in the next post.

[LoudLemur]

The AppArmor transcript, for the record, and the short version for anyone who wants to confirm it against real hardware.

What was run

• Host: a throwaway Ubuntu 24.04.4 virtual machine, kernel 6.8.0-136, AppArmor enforcing.
• Docker: docker-ce 29.6.2 with containerd 2.2.6, the versions scripts/installer.sh pins, from the same Docker repository debs. Storage driver overlayfs, Engine API 1.55.
• Profile: setup/start/docker-cloudron-app.apparmor copied verbatim from the box source and loaded with apparmor_parser -r -W.
• Device: a synthetic CDI (Container Device Interface) specification, because there is no GPU (Graphics Processing Unit) in the virtual machine. It reproduces what the profile mediates: a real shared library bind mounted to /usr/lib/x86_64-linux-gnu/libcuda.so.610.43.03, a binary mounted as a stand-in for nvidia-smi, a character device node absent from the image, an injected environment variable, and a createContainer hook doing what create-symlinks and update-ldcache do.

Result: no denials anywhere

docker run --rm --read-only \
    --security-opt apparmor=docker-cloudron-app \
    --device example.com/gpu=all ubuntu:24.04 sh -c '...'

-- injected env:            CDI_PROOF=it-works
-- read injected library:   0000000 177   E   L   F
-- mmap injected library:   mapped regions: 5
-- execute injected binary: EXEC_OK
-- open injected device:    DEVICE_WRITE_OK
-- hook artefacts:          libcuda.so.1 -> libcuda.so.610.43.03
-- rootfs writable?:        read-only as expected
--- kernel denials:         NONE

• The mmap line is the load-bearing one: the injected library is loaded through LD_PRELOAD and counted in /proc/self/maps, so the dynamic loader opened and mapped a library that exists in the container only because the specification put it there, under confinement.
• The same run unconfined gives byte-for-byte the same output, so the profile changes nothing about CDI injection.
• A third run with --user 1000:1000, since app processes are not root, also passes, and /proc/self/attr/current inside the container reads docker-cloudron-app (enforce), confirming the profile was genuinely applied rather than quietly ignored.

To confirm it on real hardware, three commands

For anyone with an NVIDIA box, the toolkit installed and Docker 28.2.0 or newer:

# 1. check the pieces
ls /etc/cdi /var/run/cdi                  # a nvidia.yaml should be in one of them
sudo aa-status | grep docker-cloudron-app # the profile should be loaded

# 2. run it in the shape the box creates
sudo docker run --rm --read-only \
    --security-opt apparmor=docker-cloudron-app \
    --device nvidia.com/gpu=all \
    ubuntu:24.04 nvidia-smi

# 3. look for denials
sudo dmesg | grep -i 'apparmor.*DENIED'

The image contains no NVIDIA userspace at all, so every part of the nvidia-smi output comes from the specification. If step 3 prints nothing, the profile blocked nothing.

If a specification needs generating first, that is sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml, and adding --user 1000:1000 to step 2 is the closest thing to what an app container actually does.

The one gap we cannot close here is the ioctls that nvidia-smi performs on /dev/nvidia*, since no GPU was in the virtual machine. Opening those device nodes is what the profile mediates and that passed, so we do not expect a surprise, but a run on real hardware would settle it.

[LoudLemur]

@girish

Understood, and thank you for the straight answer. We will not send a patch.

Since the capability itself sounded acceptable, here is everything needed to implement it, so it costs whoever picks it up an afternoon rather than a week. No code below, only behaviour and the results of the testing we have already done.

The change, in words

• In the same place vaapi is handled at container creation, a gpu capability sets HostConfig.DeviceRequests to a single entry: driver cdi, device IDs a one-element list holding the resolved device name, for example nvidia.com/gpu=all. Nothing else in the container options changes.
• Guard it so it does nothing on a host with no GPU (Graphics Processing Unit), exactly as vaapi no-ops without /dev/dri.

The guard is one API call, not a specification parser

• GET /info reports what the daemon resolved, as DiscoveredDevices. On our test daemon that reads [{"Source":"cdi","ID":"example.com/gpu=all"}].
• So: take entries whose Source is cdi and whose ID matches <vendor>/gpu=<name>, prefer the one ending =all, otherwise take the first, and if there are none, do nothing.
• Match the device class rather than the vendor prefix. AMD's toolkit generates amd.com/gpu and Docker 29.3.0 already routes --gpus through CDI (Container Device Interface) for AMD, so vendor-agnostic matching costs nothing and covers the second vendor for free. This is also why we would suggest the name gpu rather than nvidia.
• Do not read or compare cdiVersion. The upstream specification is at 1.1.0 while NVIDIA emits 0.7.0 and AMD emits its own, and consumers accept anything up to the version they support.
• DiscoveredDevices needs Engine API 1.50 or newer. On anything older, scanning /etc/cdi and /var/run/cdi for a specification whose kind ends in /gpu is an adequate fallback.

What we tested, so nobody has to repeat it

• The AppArmor profile does not interfere. Ubuntu 24.04, docker-ce 29.6.2 and containerd 2.2.6 as your installer pins them, docker-cloudron-app loaded verbatim, container run confined with a read-only root filesystem and an unprivileged user. Injected libraries were read, executed and mmapped, the injected device node opened, and dmesg showed no denials at all. Results were identical unconfined.
• ReadonlyRootfs: true is not a problem. An NVIDIA specification runs createContainer hooks that write into the container root filesystem, creating the libcuda.so.1 symlinks and running ldconfig. Those hooks run before the runtime applies the read-only remount, so they succeed while the app process itself still cannot write. No carve-out needed.
• Host prerequisites stay with the administrator. The NVIDIA driver and nvidia-container-toolkit, nothing else. No nvidia runtime registration and no daemon configuration change. Toolkit 1.19 ships nvidia-cdi-refresh.path, which regenerates the specification whenever the driver or toolkit is upgraded, so the refresh automation that made this look expensive in 2024 is now upstream's.

Two decisions we would leave to you

• Whether an unmet gpu capability should stay silent like vaapi, or fail visibly. An app quietly running without its GPU may be worse than one that refuses to start.
• Whether to gate it as experimental for a first release.

If a written specification with source references, failure modes and non-goals would be useful, we have one and will post or send it in whatever form suits. There is also an Nvidia machine here for any test that would help, including confirming the AppArmor result against real hardware, which is the one thing our virtual machine could not do.

[girish]

Yeah, feel free to share your patches here though, it's not a problem. As @james said elsewhere, the project is not set up to accept external contribution. Licensing issues aside, there is a lot of testing that goes on (we test each release in over 15 VPS providers, have e2e tests for all features), documentation, product messaging and so on which at the moment all not public. Instead of forking, you can just clone and push your repo.

[LoudLemur]

Great, and thanks girish. Cloudron is brilliant!