<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Mastodon pre-update backup repeatedly OOMs — Cloudron 10.0.5 / DigitalOcean Spaces]]></title><description><![CDATA[<h2>Problem description</h2>
<p dir="auto">My Mastodon pre-update backup repeatedly fails, blocking the update from package 1.19.1 to 1.19.2. Mastodon itself remains running and healthy. I’m posting in the Cloudron/server category because the failing process is Cloudron’s backup task.</p>
<p dir="auto">The server has <strong>16 GiB RAM and 4 GiB swap</strong>. My backup destination is <strong>DigitalOcean Spaces using rsync</strong>, with upload concurrency <strong>10</strong> and upload part size <strong>10 MiB</strong>.</p>
<p dir="auto">I moved an unused cache directory outside the app’s backup scope, preserving it for rollback. This substantially reduced the file count, but backups still fail:</p>
<ul>
<li><strong>Task 24234:</strong> with a <strong>6 GiB backup memory limit</strong>, the host ran out of memory. The kernel killed the backup worker at approximately <strong>3.37 GiB anonymous RSS</strong>, reporting <code>global_oom</code>.</li>
<li><strong>Task 24245:</strong> after rebooting and lowering the backup memory limit to <strong>3 GiB</strong>, the worker hit its own cgroup limit (<code>CONSTRAINT_MEMCG</code>). It stopped after uploading <strong>9,466/28,032 files</strong>. Cloudron reported “Task stopped” / exit code 70.</li>
</ul>
<p dir="auto">My latest retained successful Mastodon backup is from <strong>September 3</strong>. The update has not completed, and the app remains in <code>installationState=error</code> despite being healthy.</p>
<h2><code>cloudron-support --troubleshoot</code></h2>
<p dir="auto">Some details are redacted below; diagnostic results are otherwise unchanged.</p>
<pre><code class="language-text">═══ Host ═══
Vendor: DigitalOcean Product: Droplet
Linux: 5.15.0-191-generic
Ubuntu: jammy 22.04
Cloudron: 10.0.5
Execution environment: kvm
Processor: DO-Premium-AMD x 4
RAM: 16373036KB
Disk: /dev/vda1        60G

═══ System ═══
[OK] Root disk usage is OK (70%)
[OK] Memory usage is OK (74%)
[WARN] Swap is 99% used — the box may be undersized
[OK] Disk I/O on /home/yellowtent (vda, HDD): 571 MB/s write, iowait 6%
[OK] Clock is NTP-synchronized
[OK] node version is correct
[OK] IPv6 is enabled in kernel. Public IPv6 address detected
[OK] docker is running
[OK] docker version is correct

═══ Networking ═══
[OK] no incompatible persistent-firewall packages installed
[OK] DNS is resolving via systemd-resolved
[OK] unbound is running
[OK] IPv4 HTTPS to api.cloudron.io/api/v1/helper/public_ip
[OK] IPv6 HTTPS to api.cloudron.io/api/v1/helper/public_ip
[OK] IPv4 HTTPS to auth.docker.io/token
[OK] IPv6 HTTPS to auth.docker.io/token
[OK] IPv4 HTTPS to acme-v02.api.letsencrypt.org
[OK] IPv6 HTTPS to acme-v02.api.letsencrypt.org
[OK] netplan is good

═══ Cloudron Platform ═══
[OK] MySQL is running
[OK] nginx is running
[OK] Dashboard cert is valid [expiry details redacted]
[OK] dashboard is reachable via loopback
[OK] No pending database migrations
[OK] Service 'mysql' is running and healthy
[OK] Service 'postgresql' is running and healthy
[OK] Service 'mongodb' is lazy-stopped (no installed Cloudron app uses the 'mongodb' addon)
[OK] Service 'mail' is running and healthy
[OK] Service 'graphite' is running and healthy
[OK] Service 'sftp' is running and healthy
[OK] box v10.0.5 is running
[OK] Dashboard is reachable via IPv4 [URL redacted]
[OK] Dashboard is reachable via IPv6 [URL redacted]
[OK] Domain [redacted] is valid [expiry details redacted]

======== Summary ========
PASS: 33
WARN: 1
FAIL: 0
SKIP: 0

Warnings:
  - Swap is 99% used — the box may be undersized

For troubleshooting tips, see https://docs.cloudron.io/troubleshooting
To share a full diagnostic dump on the forum, run: cloudron-support --send-diagnostics
</code></pre>
<h2>Relevant errors from <code>box.log</code></h2>
<p dir="auto">Selected lines from <code>/home/yellowtent/platformdata/logs/box.log</code>:</p>
<pre><code class="language-text">2026-09-20T07:18:51.297Z tasks: startTask: 24245 done. error: { message: 'Task 24245 stopped', code: 'stopped' }
2026-09-20T07:18:51.311Z apps: Apptask crashed/stopped: Task 24245 stopped
</code></pre>
<p dir="auto">The same log records this task-runner output:</p>
<pre><code class="language-text">Finished with result: oom-kill
Main processes terminated with: code=exited/status=70
Service runtime: 8min 37.392s
CPU time consumed: 3min 31.136s
</code></pre>
<p dir="auto">Relevant kernel log excerpts, with the hostname omitted:</p>
<pre><code class="language-text">2026-09-20T07:18:50+0000 kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=box-task-24245.service,mems_allowed=0,oom_memcg=/system.slice/box-task-24245.service,task_memcg=/system.slice/box-task-24245.service,task=MainThread,pid=389975,uid=0
2026-09-20T07:18:50+0000 kernel: Memory cgroup out of memory: Killed process 389975 (MainThread) total-vm:13140380kB, anon-rss:2949476kB, file-rss:27000kB, shmem-rss:0kB, UID:0 pgtables:16832kB oom_score_adj:0
</code></pre>
<h2>Questions</h2>
<ol>
<li>Is this a known issue with the rsync/S3 backup path in Cloudron 10.0.5? What configuration would you recommend, given that a larger task limit previously exhausted the host?</li>
<li>I’m considering a separate Mastodon-only <strong>tgz backup site</strong>, using the same Space under a unique prefix while preserving existing rsync backups. What is the supported way to return the healthy app from <code>installationState=error</code> to a state that permits a standalone manual backup, without restoring data or skipping the update safety backup?</li>
<li>Is large-tarball backup rotation on Spaces supported in 10.0.5? Cloudron’s documentation warns about copying objects over 5 GB, but the installed code uses multipart <code>UploadPartCopy</code>, which DigitalOcean currently documents as supported.</li>
</ol>
<p dir="auto">I’d like to verify a fresh backup before changing the update-backup routing or attempting another update. I can provide additional logs to support if needed.</p>
<p dir="auto">Thanks!</p>
]]></description><link>https://forum.cloudron.io/topic/15987/mastodon-pre-update-backup-repeatedly-ooms-cloudron-10.0.5-digitalocean-spaces</link><generator>RSS for Node</generator><lastBuildDate>Sat, 26 Sep 2026 09:22:48 GMT</lastBuildDate><atom:link href="https://forum.cloudron.io/topic/15987.rss" rel="self" type="application/rss+xml"/><pubDate>Sun, 20 Sep 2026 08:22:48 GMT</pubDate><ttl>60</ttl><item><title><![CDATA[Reply to Mastodon pre-update backup repeatedly OOMs — Cloudron 10.0.5 / DigitalOcean Spaces on Sun, 20 Sep 2026 10:32:09 GMT]]></title><description><![CDATA[<p dir="auto">Hello <a class="plugin-mentions-user plugin-mentions-a" href="/user/stormgrass" aria-label="Profile: stormgrass">@<bdi>stormgrass</bdi></a><br />
I would go with the Red Hat guide <a href="https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/storage_administration_guide/ch-swapspace" target="_blank" rel="noopener noreferrer nofollow ugc">https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/storage_administration_guide/ch-swapspace</a><br />
So at 16 GiB memory 8 GiB of SWAP.</p>
]]></description><link>https://forum.cloudron.io/post/129748</link><guid isPermaLink="true">https://forum.cloudron.io/post/129748</guid><dc:creator><![CDATA[james]]></dc:creator><pubDate>Sun, 20 Sep 2026 10:32:09 GMT</pubDate></item><item><title><![CDATA[Reply to Mastodon pre-update backup repeatedly OOMs — Cloudron 10.0.5 / DigitalOcean Spaces on Sun, 20 Sep 2026 13:09:08 GMT]]></title><description><![CDATA[<p dir="auto">Thanks, that did it!</p>
<p dir="auto">I increased total swap from 4 GiB to 8 GiB, leaving the backup task’s memory limit at 3 GiB and all other backup settings unchanged.</p>
<p dir="auto">The next attempt completed the backup successfully, then updated Mastodon from package 1.19.1 to 1.19.2.</p>
<p dir="auto">Thanks for pointing me in the right direction!</p>
]]></description><link>https://forum.cloudron.io/post/129753</link><guid isPermaLink="true">https://forum.cloudron.io/post/129753</guid><dc:creator><![CDATA[stormgrass]]></dc:creator><pubDate>Sun, 20 Sep 2026 13:09:08 GMT</pubDate></item><item><title><![CDATA[Reply to Mastodon pre-update backup repeatedly OOMs — Cloudron 10.0.5 / DigitalOcean Spaces on Sun, 20 Sep 2026 10:32:09 GMT]]></title><description><![CDATA[<p dir="auto">Hello <a class="plugin-mentions-user plugin-mentions-a" href="/user/stormgrass" aria-label="Profile: stormgrass">@<bdi>stormgrass</bdi></a><br />
I would go with the Red Hat guide <a href="https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/storage_administration_guide/ch-swapspace" target="_blank" rel="noopener noreferrer nofollow ugc">https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/storage_administration_guide/ch-swapspace</a><br />
So at 16 GiB memory 8 GiB of SWAP.</p>
]]></description><link>https://forum.cloudron.io/post/129748</link><guid isPermaLink="true">https://forum.cloudron.io/post/129748</guid><dc:creator><![CDATA[james]]></dc:creator><pubDate>Sun, 20 Sep 2026 10:32:09 GMT</pubDate></item><item><title><![CDATA[Reply to Mastodon pre-update backup repeatedly OOMs — Cloudron 10.0.5 / DigitalOcean Spaces on Sun, 20 Sep 2026 09:47:05 GMT]]></title><description><![CDATA[<p dir="auto">Thanks for the quick reply, and for clarifying the Spaces multipart-copy support.</p>
<p dir="auto">I’ve now checked the backup task’s own log alongside the kernel and systemd records. For the latest attempt, PID 389975 appears to belong to the backup task rather than MySQL:</p>
<pre><code>oom_memcg=/system.slice/box-task-24245.service
task_memcg=/system.slice/box-task-24245.service
task=MainThread,pid=389975
</code></pre>
<p dir="auto">Systemd reports:</p>
<pre><code>Result=oom-kill
ExecMainStatus=70
MemoryMax=3221225472
</code></pre>
<p dir="auto">The corresponding task log identifies task 24245, shows it uploading files until 07:18:44 UTC, and then records:</p>
<pre><code>2026-09-20T07:18:50.714Z Exiting with code 70
</code></pre>
<p dir="auto">There were two different failures: the earlier attempt hit global OOM with the backup limit set to 6 GiB; the latest hit CONSTRAINT_MEMCG with the backup limit set to 3 GiB.</p>
<p dir="auto">Swap was indeed exhausted, so I’m open to increasing it. What swap size and backup-task memory limit would you recommend testing together? Is there a cgroup swap limit I should check first?</p>
<p dir="auto">Given the process mapping above, would you still recommend increasing MySQL’s limit? This Mastodon app uses PostgreSQL.</p>
<p dir="auto">I can provide the redacted backup-task log as well.</p>
]]></description><link>https://forum.cloudron.io/post/129745</link><guid isPermaLink="true">https://forum.cloudron.io/post/129745</guid><dc:creator><![CDATA[stormgrass]]></dc:creator><pubDate>Sun, 20 Sep 2026 09:47:05 GMT</pubDate></item><item><title><![CDATA[Reply to Mastodon pre-update backup repeatedly OOMs — Cloudron 10.0.5 / DigitalOcean Spaces on Sun, 20 Sep 2026 09:18:25 GMT]]></title><description><![CDATA[<p dir="auto">Hello <a class="plugin-mentions-user plugin-mentions-a" href="/user/stormgrass" aria-label="Profile: stormgrass">@<bdi>stormgrass</bdi></a><br />
Thanks for reporting this issue with all the details.</p>
<blockquote>
<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/user/stormgrass" aria-label="Profile: stormgrass">@<bdi>stormgrass</bdi></a> <a href="/post/129739">said</a>:</p>
<p dir="auto">Is large-tarball backup rotation on Spaces supported in 10.0.5? Cloudron’s documentation warns about copying objects over 5 GB, but the installed code uses multipart UploadPartCopy, which DigitalOcean currently documents as supported.</p>
</blockquote>
<p dir="auto">It should work. Why <em><strong>should</strong></em>?<br />
We recently found an issue with IONOS S3 and large multipart UploadPartCopy which IONOS is now working on to fix it.</p>
<hr />
<p dir="auto">The kernal log shows it killed PID <code>389975</code> since the log of the backup task itself was not provided I am not 100% sure if PID <code>389975</code> is actually the backup process itself.<br />
It could be that the MySQL service run out of memory and that was killed.<br />
Also the close proximity of</p>
<blockquote>
<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/user/stormgrass" aria-label="Profile: stormgrass">@<bdi>stormgrass</bdi></a> <a href="/post/129739">said</a>:</p>
<p dir="auto">The kernel killed the backup worker at approximately 3.37 GiB anonymous RSS, reporting global_oom.</p>
</blockquote>
<p dir="auto">to your 4 GiB swap size makes me a little suspicous that it might be running out of SWAP memory and not actual memory.<br />
Could you try increasing the MySQL service memory limit and maybe also increasing the SWAP memory?</p>
]]></description><link>https://forum.cloudron.io/post/129743</link><guid isPermaLink="true">https://forum.cloudron.io/post/129743</guid><dc:creator><![CDATA[james]]></dc:creator><pubDate>Sun, 20 Sep 2026 09:18:25 GMT</pubDate></item></channel></rss>