v3.24.0 constantly runs out of memory

[timconsidine]

Cloudron Team : I am not clear whether cloudron 3.24.0 used a different upstream base 3.1.0 to previous, but my impression is not. So the issue is in cloudron v3.24.0 ? Can it be fixed ?

[james]

Hello @timconsidine
These are the changes made https://git.cloudron.io/packages/umami-app/-/commit/4e3d31c2f37285960d017fd157956b5aae9138ef

One thing that jumps at me is --max-old-space-size=4096.

The container runs a full Next.js production yarn build on every start (see start.sh:L35).
Next.js builds are notoriously memory-hungry; they scale with the size of the app bundle.
Node is told --max-old-space-size=4096 (4 GiB V8 heap) while the cgroup hard cap from CloudronManifest.json:L13 is 2 GiB by default.
Node happily tries to grow past 2 GiB, the kernel OOM-kills the container.

This value should not be hard coded, instead should reflect the memory limit set by the user.
An app update will be available soon.

[timconsidine]

@james thank you, I appreciate you looking into and resolving it

[james]

Fixed with the latest update https://forum.cloudron.io/post/123645

[ekevu123]

I have just run into the same issue.

Isn't the issue that the build process forks into different processes per worker and the sum of it takes too much memory? I had to set my memory limit to 5 GB to get the app to start again. It looks to me like the workers are given 75 % of the memory assigned to umami each, which leads to the issue. Also, does the app have to recompile every single time it restarts?
Maybe the default memory could also be raised above 2 GB considering that several people (also in another thread) faced memory issues before.

[d19dotca]

I too have just run into this with the most recent package/image from Cloudron for Umami... it was constantly hitting OOM and failing to respond until I lifted it from 2 GB to 5 GB temporarily and then could reduce it back down again. I think the issue was that it temporarily needed much more than 2 GB of memory to perform what appeared to be some database updates for Umami.

This was for package 3.27.0, which contains Umami 3.4.0.

[timconsidine]

@d19dotca FWIW I have switched from Umami to Rybbit (Community App).

Umami performs well, I just prefer Rybbit.

[d19dotca]

@timconsidine Yeah I think I prefer Rybbit too, right now I haven’t switched over yet though and am running both in parallel, with Rybbit taking a few sites too.

[robi]

@d19dotca Yeah, I had it OOM for a while and it was already at 2GB RAM, It needed 3GB just to do a rebuild and it took an hour.

The new default needs to be 3GB @james

All that to find out i don't want it anymore ;-/

Time to install Rybbit. Thanks Tim.

[ekevu123]

I keep running into this issue again. Oddly, it doesn't matter how high my memory limit is (it's set to 5 GB at the moment), it still keeps happening. The usage graph doesn't show any spike. This happens when 11 workers are being spawned simultaneously using next build apparently.

EDIT: I did some digging (as I did before), and this happens during auto-update and restart. One theory right now is that during auto-update, when multiple apps are updated, it causes a spike on the server, that's why an ordinary mid-day restart can't reproduce it.

Can Cloudron change the build process and perhaps have fewer workers run in parallel, and perhaps not rebuild the app with every restart?

[joseph]

@ekevu123 said:

Can Cloudron change the build process and perhaps have fewer workers run in parallel, and perhaps not rebuild the app with every restart?

Apps are not rebuilt on every restart. Do you mean creating a container ?

[ekevu123]

I might be wrong, but see below, I get this upon every restart. Isn't that an unnecessary build process for a restart without any update?

Restart requested:

apptask: run: startTask installationState: pending_restart runState: running
tasks: updating task 16772 with: {"percent":30,"message":"Restarting container"}

Build pipeline starts inside the container:

=> Running build script that generates the migrations
$ npm-run-all check:env build:db check:db build:tracker build:recorder build:geo build:openapi build:packages build:app

Cloudron reports success while the build is still running:

tasks: updating task 16772 with: {"percent":100,"message":"Done"}
taskworker: Task took 11.522 seconds

Nothing listens on port 3000 during the build (repeats every 10 seconds):

=> Healthcheck error: Error: connect ECONNREFUSED 172.18.18.146:3000

Next.js production build:

$ next build --turbo
▲ Next.js 16.3.4 (Turbopack)
✓ Compiled successfully in 12.0s

Post-build and start:

$ node scripts/postbuild.js
=> Running migrations
No pending migrations to apply.
==> Starting Umami
✓ Ready in 154ms