How can we optimize/clean disk usage from Docker and more?

[girish]

@d19dotca speak of the devil... there is a new version of umami, let me check why it's so big.

[jdaviescoates]

@d19dotca said in How can we optimize/clean disk usage from Docker and more?:

docker image prune -a it still gave me a 0 byte response showing it cleaned up nothing.

Just to say, I'm pretty sure that in the past I've done this and got a 0 byte response too, but then there actually was more space available.

[girish]

Just the node modules is 1.4G

root@aaa028f2-b8bc-41cf-82be-a23f59293d39:/app/code# du -hcs *
4.0K	Dockerfile
4.0K	LICENSE
4.0K	README.md
4.0K	app.json
36K	cypress
4.0K	cypress.config.ts
160K	db
8.0K	docker
4.0K	docker-compose.yml
51M	geo
4.0K	jest.config.ts
4.0K	jsconfig.json
4.0K	lang-ignore.json
4.0K	netlify.toml
4.0K	next-env.d.ts
4.0K	next.config.js
1.3G	node_modules
4.0K	package.components.json
8.0K	package.json
4.0K	postcss.config.js
68K	prisma
4.5M	public
4.0K	rollup.components.config.mjs
4.0K	rollup.tracker.config.mjs
60K	scripts
3.2M	src
4.0K	tsconfig.json
532K	yarn.lock
1.4G	total

[d19dotca]

@girish does that mean the image should only be about 1.4 GB in size rather than the 6+GB I’m seeing currently? Very odd if that’s the case, I wonder how that exploded in size.

[d19dotca]

FWIW, I uninstalled the app so that the image would be removed, confirmed it was removed, and then re-installed. The image size changed slightly from 6.57 GB to 6.51 GB, but still quite large. I may be doing in the wrong area though if we don't think this is a huge concern, just really strange to me why it's so much larger than all the other images when it's supposedly a fairly lean app.

For the life of me I cannot seem to reclaim the over 2 GB of disk space from images in Docker, I am puzzled as to why that is happening.

[girish]

@d19dotca it seems the yarn cache was note cleared. Clearing it brought down the size . It's in the new package.

[andreasb]

cleaning up the docker images as mentioned above basically works.
On my instance, there however are two app docker images that are much larger than the others:
rallly (almost 6GB), loomio (>6GB)
Is this size "normal"?
Is the "normal" size of the apps' docker images documented anywhere?

[joseph]

@andreasb you can check the sizes of docker images here - https://hub.docker.com/u/cloudron . But looks like it lists only the compressed size. Size of docker images is not documented anywhere . The 6GB includes many layers and the layers are "shared" across apps (the way docker works).

We also hvae images on quay - https://quay.io/organization/cloudron . But I think this requires login, despite images being public . These sizes differ from dockerhub

[robi]

After being inspired by @timconsidine's AppsMonitor and noticing higher than usual disk usage, here is the perplexing af play-by-play of how I saved over 20GB of space in terminal commands.

54GB seems high for this smaller server. d;-/

root@zvuk:~# docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          53        49        54.14GB   393.4MB (0%)
Containers      130       110       625.5MB   625.5MB (100%)
Local Volumes   244       244       9.686GB   0B (0%)
Build Cache     0         0         0B        0B

root@zvuk:~# docker image prune -f
Total reclaimed space: 0MB

root@zvuk:~# docker container prune -f
Total reclaimed space: 625.5MB
root@zvuk:~# docker volume prune -f
Deleted Volumes:
...

Total reclaimed space: 3.422GB
root@zvuk:~# docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          53        42        54.14GB   17.79GB (32%)
Containers      110       110       0B        0B
Local Volumes   224       224       6.263GB   0B (0%)
Build Cache     0         0         0B        0B

root@zvuk:~# docker image prune -f
Deleted Images:
...

Total reclaimed space: 2.36kB

What? That's it?

How did volume prune find 3+GB when 0% was reclaimable?

Then this took a while of staring & pondering:

root@zvuk:~# docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          51        42        54.14GB   17.99GB (33%)
Containers      110       110       0B        0B
Local Volumes   224       224       6.263GB   0B (0%)
Build Cache     0         0         0B        0B


root@zvuk:~# docker image prune -af
Deleted Images:
... (long list)

Total reclaimed space: 20.16GB

root@zvuk:~# docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          42        42        33.97GB   0B (0%)
Containers      110       110       0B        0B
Local Volumes   224       224       6.263GB   0B (0%)
Build Cache     0         0         0B        0B

I learned that the -af flag is important at least for images.

The long list of deleted images contained old surfer images, wordpress.unmanaged images, many untagged images, including the cloudron 5.0.0 base image, custom apps and apps I tried once and deleted long ago.

@girish Do we need box to include more checks across volumes, containers and images now that we're dealing with so many custom apps and base images updates?

[jdaviescoates]

@robi said:

I learned that the -af flag is important at least for images.

This should probably make it's way into the docs somewhere.

[huehner]

@girish Hi, i was looking at the umami Dockerfile and i think we could save another 200MB here on top.

The pnpm install is not adding --prod here https://git.cloudron.io/packages/umami-app/-/blob/master/Dockerfile?ref_type=heads#L23
So it's also adding all the devDependencies to node_modules (i.e. vitest, etc)
Skipping those slims the node_modules slightly (200MB)

There's one small problem though the "pnpm run build:geo" part fails then as it uses the tar module which is only in devDependencies. Adding tar manually there as a workaround get's it to work.
I did a quick "cloudron builder build" and also cloudron install (as quite a bit of the build process is on start time) and with just that tar workaround it went through and can login to the app.

I didn't report that tar bug upstream or did any more testing (as i don't actually use umami). Was just curious myself if we could same some more space here.