Packaging a tiny static-binary app without `cloudron/base` — anything wrong with this approach?

[LoudLemur]

We are packaging a small network service and ran into a size problem would like a sanity check on before I go further with it.

The situation: the app itself is a single statically-linked binary,
under 10MB, with no runtime dependencies. Following the tutorial /
cheat-sheet pattern, the final Docker stage is cloudron/base, and the resulting image comes out to roughly 2.5GB — almost entirely the base image itself (a large apt-get layer plus a bundled language runtime the app never touches), not the app. That size turned out to matter in practice: even with the image built ahead of time and installed via cloudron install --image (so there's no server-side compile), the install is still slow, because the whole 2.5GB has to be pushed to a registry and then pulled onto the Cloudron host every time.

What we tried instead: We re-read the packaging cheat-sheet looking for anything that actually requires cloudron/base specifically, and couldn't find one — the documented contract (read-only rootfs with /app/data//run//tmp writable, the CLOUDRON_* env vars, HTTP health checks, stdout/stderr logging, addon file injection) reads as enforced by the platform at runtime, not baked into that particular base image. So I swapped the final stage for a minimal base instead,
just enough to provide a cloudron user at uid:gid 1000:1000, a
gosu-equivalent privilege-drop tool, and a shell to run start.sh
(kept bash specifically, not the base's default lighter shell, so
the dashboard's web terminal and file manager, which exec bash intothe container, still work). Final image: about 20MB, down from 2.5GB.

We tested it as thoroughly as we could from outside actually installing on a Cloudron: ran the built image as a container with the same env vars Cloudron sets, confirmed the app starts and serves correctly, confirmed start.sh's root→drop-privilege sequence still works and the process ends up running as the unprivileged user, confirmed bash is present and functional inside the running container, and simulated
a permissions reset on the data directory (as the docs say backup
restores can cause) to confirm the startup script's chown still
recovers correctly. It also installed and ran fine on a real (personal,
non-Store) Cloudron instance.

What we are unsure about: We went looking for a precedent and couldn't
find any. Every app Dockerfile we could find under the official
Cloudron GitHub org uses cloudron/base, including at least one that's about as minimal in spirit as mine (a simple single-purpose file server). Zero hits for alternate/minimal base images anywhere in that org. That's made us second-guess it even though I can't find a documented reason it shouldn't work:

• Is there a real technical reason every app uses cloudron/base that
  isn't written down in the packaging docs, something the base image provides that we haven't hit yet (backup/restore internals, the update mechanism, something the App Store review process execs into the container and expects to find)?
• Would this be looked at unfavorably for an eventual Store submission
• Is there a reason apps this size still ship on the big base that
  we are missing, or is it genuinely just "nobody's wanted to optimize this because most apps actually need a language runtime anyway"?

Might this bite us later?

[fbartels]

@LoudLemur said:

the install is still slow, because the whole 2.5GB has to be pushed to a registry and then pulled onto the Cloudron host every time.

Your ai is not reasoning correctly here. Since all cloudron app share the same base image these 2,5gb are already sitting on the cloudron system and only the layers you add during your app build are downloaded.

[LoudLemur]

@fbartels

Thank you for looking at this. You are right that content-addressable layer dedup is how this is supposed to work, however we don't think it explains what we actually saw:

Two things:

1. The base image isn't actually one constant across a Cloudron host. We pulled cloudron/base locally while investigating this and ended up with three distinct cached versions: 4.2.0, 5.0.0, and an untagged/latest one, at 2.25–2.48GB each, none byte-identical.

If installs on a given host accumulate whatever base version was current when each app was last built or updated, rather than all sharing one pinned digest, then a newer pin (we are on 5.1.0) isn't guaranteed to already be sitting there just because other apps on the same host use a cloudron/base of one version or another.

2. The actual install we watched doesn't fit "already cached." We built the exact same image with the old base and installed it fresh on a real (personal) Cloudron via cloudron install --image, timed it: ~26 minutes, with a Downloading image step that sat there for most of that time. If the 2.5GB of base layers were already resident on that host and only our ~9MB diff needed to move, that step should have finished in seconds, not half an hour. After switching to the ~20MB image, the same install (uninstall + reinstall, same host) took 1m40s total, download step included.

So either that particular host genuinely didn't have our exact base-image digest cached yet, or something else was slow that we haven't identified, but "the base image was already shared, so size wasn't the bottleneck" doesn't match the timing we actually measured on that install.

Is there a piece of the caching behaviour we are missingthat would explain the ~25-minute gap, not just the general mechanism?

[fbartels]

@LoudLemur said:

that particular host genuinely didn't have our exact base-image digest cached yet

This was probably the reason for the long download.

[robi]

The base image was recently updated too