Mastodon pre-update backup repeatedly OOMs — Cloudron 10.0.5 / DigitalOcean Spaces

[stormgrass]

Problem description

My Mastodon pre-update backup repeatedly fails, blocking the update from package 1.19.1 to 1.19.2. Mastodon itself remains running and healthy. I’m posting in the Cloudron/server category because the failing process is Cloudron’s backup task.

The server has 16 GiB RAM and 4 GiB swap. My backup destination is DigitalOcean Spaces using rsync, with upload concurrency 10 and upload part size 10 MiB.

I moved an unused cache directory outside the app’s backup scope, preserving it for rollback. This substantially reduced the file count, but backups still fail:

• Task 24234: with a 6 GiB backup memory limit, the host ran out of memory. The kernel killed the backup worker at approximately 3.37 GiB anonymous RSS, reporting global_oom.
• Task 24245: after rebooting and lowering the backup memory limit to 3 GiB, the worker hit its own cgroup limit (CONSTRAINT_MEMCG). It stopped after uploading 9,466/28,032 files. Cloudron reported “Task stopped” / exit code 70.

My latest retained successful Mastodon backup is from September 3. The update has not completed, and the app remains in installationState=error despite being healthy.

cloudron-support --troubleshoot

Some details are redacted below; diagnostic results are otherwise unchanged.

═══ Host ═══
Vendor: DigitalOcean Product: Droplet
Linux: 5.15.0-191-generic
Ubuntu: jammy 22.04
Cloudron: 10.0.5
Execution environment: kvm
Processor: DO-Premium-AMD x 4
RAM: 16373036KB
Disk: /dev/vda1        60G

═══ System ═══
[OK] Root disk usage is OK (70%)
[OK] Memory usage is OK (74%)
[WARN] Swap is 99% used — the box may be undersized
[OK] Disk I/O on /home/yellowtent (vda, HDD): 571 MB/s write, iowait 6%
[OK] Clock is NTP-synchronized
[OK] node version is correct
[OK] IPv6 is enabled in kernel. Public IPv6 address detected
[OK] docker is running
[OK] docker version is correct

═══ Networking ═══
[OK] no incompatible persistent-firewall packages installed
[OK] DNS is resolving via systemd-resolved
[OK] unbound is running
[OK] IPv4 HTTPS to api.cloudron.io/api/v1/helper/public_ip
[OK] IPv6 HTTPS to api.cloudron.io/api/v1/helper/public_ip
[OK] IPv4 HTTPS to auth.docker.io/token
[OK] IPv6 HTTPS to auth.docker.io/token
[OK] IPv4 HTTPS to acme-v02.api.letsencrypt.org
[OK] IPv6 HTTPS to acme-v02.api.letsencrypt.org
[OK] netplan is good

═══ Cloudron Platform ═══
[OK] MySQL is running
[OK] nginx is running
[OK] Dashboard cert is valid [expiry details redacted]
[OK] dashboard is reachable via loopback
[OK] No pending database migrations
[OK] Service 'mysql' is running and healthy
[OK] Service 'postgresql' is running and healthy
[OK] Service 'mongodb' is lazy-stopped (no installed Cloudron app uses the 'mongodb' addon)
[OK] Service 'mail' is running and healthy
[OK] Service 'graphite' is running and healthy
[OK] Service 'sftp' is running and healthy
[OK] box v10.0.5 is running
[OK] Dashboard is reachable via IPv4 [URL redacted]
[OK] Dashboard is reachable via IPv6 [URL redacted]
[OK] Domain [redacted] is valid [expiry details redacted]

======== Summary ========
PASS: 33
WARN: 1
FAIL: 0
SKIP: 0

Warnings:
  - Swap is 99% used — the box may be undersized

For troubleshooting tips, see https://docs.cloudron.io/troubleshooting
To share a full diagnostic dump on the forum, run: cloudron-support --send-diagnostics

Relevant errors from box.log

Selected lines from /home/yellowtent/platformdata/logs/box.log:

2026-09-20T07:18:51.297Z tasks: startTask: 24245 done. error: { message: 'Task 24245 stopped', code: 'stopped' }
2026-09-20T07:18:51.311Z apps: Apptask crashed/stopped: Task 24245 stopped

The same log records this task-runner output:

Finished with result: oom-kill
Main processes terminated with: code=exited/status=70
Service runtime: 8min 37.392s
CPU time consumed: 3min 31.136s

Relevant kernel log excerpts, with the hostname omitted:

2026-09-20T07:18:50+0000 kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=box-task-24245.service,mems_allowed=0,oom_memcg=/system.slice/box-task-24245.service,task_memcg=/system.slice/box-task-24245.service,task=MainThread,pid=389975,uid=0
2026-09-20T07:18:50+0000 kernel: Memory cgroup out of memory: Killed process 389975 (MainThread) total-vm:13140380kB, anon-rss:2949476kB, file-rss:27000kB, shmem-rss:0kB, UID:0 pgtables:16832kB oom_score_adj:0

Questions

1. Is this a known issue with the rsync/S3 backup path in Cloudron 10.0.5? What configuration would you recommend, given that a larger task limit previously exhausted the host?
2. I’m considering a separate Mastodon-only tgz backup site, using the same Space under a unique prefix while preserving existing rsync backups. What is the supported way to return the healthy app from installationState=error to a state that permits a standalone manual backup, without restoring data or skipping the update safety backup?
3. Is large-tarball backup rotation on Spaces supported in 10.0.5? Cloudron’s documentation warns about copying objects over 5 GB, but the installed code uses multipart UploadPartCopy, which DigitalOcean currently documents as supported.

I’d like to verify a fresh backup before changing the update-backup routing or attempting another update. I can provide additional logs to support if needed.

Thanks!

[james]

Hello @stormgrass
I would go with the Red Hat guide https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/storage_administration_guide/ch-swapspace
So at 16 GiB memory 8 GiB of SWAP.

[james]

Hello @stormgrass
Thanks for reporting this issue with all the details.

@stormgrass said:

Is large-tarball backup rotation on Spaces supported in 10.0.5? Cloudron’s documentation warns about copying objects over 5 GB, but the installed code uses multipart UploadPartCopy, which DigitalOcean currently documents as supported.

It should work. Why should?
We recently found an issue with IONOS S3 and large multipart UploadPartCopy which IONOS is now working on to fix it.

---

The kernal log shows it killed PID 389975 since the log of the backup task itself was not provided I am not 100% sure if PID 389975 is actually the backup process itself.
It could be that the MySQL service run out of memory and that was killed.
Also the close proximity of

@stormgrass said:

The kernel killed the backup worker at approximately 3.37 GiB anonymous RSS, reporting global_oom.

to your 4 GiB swap size makes me a little suspicous that it might be running out of SWAP memory and not actual memory.
Could you try increasing the MySQL service memory limit and maybe also increasing the SWAP memory?

[stormgrass]

Thanks for the quick reply, and for clarifying the Spaces multipart-copy support.

I’ve now checked the backup task’s own log alongside the kernel and systemd records. For the latest attempt, PID 389975 appears to belong to the backup task rather than MySQL:

oom_memcg=/system.slice/box-task-24245.service
task_memcg=/system.slice/box-task-24245.service
task=MainThread,pid=389975

Systemd reports:

Result=oom-kill
ExecMainStatus=70
MemoryMax=3221225472

The corresponding task log identifies task 24245, shows it uploading files until 07:18:44 UTC, and then records:

2026-09-20T07:18:50.714Z Exiting with code 70

There were two different failures: the earlier attempt hit global OOM with the backup limit set to 6 GiB; the latest hit CONSTRAINT_MEMCG with the backup limit set to 3 GiB.

Swap was indeed exhausted, so I’m open to increasing it. What swap size and backup-task memory limit would you recommend testing together? Is there a cgroup swap limit I should check first?

Given the process mapping above, would you still recommend increasing MySQL’s limit? This Mastodon app uses PostgreSQL.

I can provide the redacted backup-task log as well.

[james]

Hello @stormgrass
I would go with the Red Hat guide https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/storage_administration_guide/ch-swapspace
So at 16 GiB memory 8 GiB of SWAP.

[stormgrass]

Thanks, that did it!

I increased total swap from 4 GiB to 8 GiB, leaving the backup task’s memory limit at 3 GiB and all other backup settings unchanged.

The next attempt completed the backup successfully, then updated Mastodon from package 1.19.1 to 1.19.2.

Thanks for pointing me in the right direction!