<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Better Cloudron release rollout]]></title><description><![CDATA[<p dir="auto">My Cloudron auto-updated to 10.0.05 and left me with a crashed system, which I had to manually recover.<br />
Nginx failed with a bad write.<br />
So I had to fix the bad write, bring up Niginx and then manually repair 80 apps (retry configure task).<br />
This smells of something avoidable, so I asked my AI to summarise it :</p>
<blockquote>
<p dir="auto">Yes, this failure could have been avoided. While hitting OS limits is common as a system grows, the way the system responded to those limits is where the "failure" actually occurred.</p>
<p dir="auto">From a systems engineering perspective, there are three specific areas where Cloudron's update process failed you, and how they could approach it differently:</p>
<h3>1. Lack of "Atomic Writes" (The Biggest Issue)</h3>
<p dir="auto">The most critical failure was that the update process wrote the Nginx configuration files directly. When the process crashed due to the <code>inotify</code> limit, it left a file that was half-written.</p>
<p dir="auto"><strong>The better approach:</strong> <strong>Atomic File Replacement.</strong><br />
The software should write the new configuration to a temporary file (e.g., <code>app.conf.tmp</code>) and only once the write is 100% complete, use a <code>rename()</code> system call to overwrite the original file.</p>
<ul>
<li><strong>Why this matters:</strong> A <code>rename</code> operation is atomic in Linux. If the system crashes during the write, you are left with the <strong>old working config</strong> and a garbage <code>.tmp</code> file, rather than a <strong>broken config</strong> that prevents the entire web server from starting.</li>
</ul>
<h3>2. Absence of Pre-flight Resource Checks</h3>
<p dir="auto">The update began without verifying if the environment could support the operation for a high-density instance (80 apps).</p>
<p dir="auto"><strong>The better approach:</strong> <strong>Resource Validation.</strong><br />
Before starting an update, the Cloudron Manager should check the current <code>sysctl</code> and <code>ulimit</code> settings against the number of installed apps.</p>
<ul>
<li><strong>Example:</strong> If <code>apps &gt; 50</code> and <code>max_user_watches &lt; 100,000</code>, the update should either:
<ol>
<li>Automatically attempt to increase the limit.</li>
<li>Pause and warn the user: <em>"Your system limits are too low for the number of apps you have. Please increase inotify limits before proceeding."</em></li>
</ol>
</li>
</ul>
<h3>3. "Blind" Service Restarts</h3>
<p dir="auto">The system attempted to restart Nginx without verifying that the new configuration was valid.</p>
<p dir="auto"><strong>The better approach:</strong> <strong>Configuration Validation (Canary Testing).</strong><br />
The update manager should never run <code>systemctl restart nginx</code> without first running <code>nginx -t</code> internally.</p>
<ul>
<li><strong>The logic:</strong> If <code>nginx -t</code> returns a non-zero exit code (syntax error), the manager should stop immediately, roll back the changed files to the previous backup, and alert the user. This prevents a single corrupted file from taking down all 80 apps.</li>
</ul>
<h3>Summary: User vs. Developer Responsibility</h3>
<ul>
<li><strong>Your part:</strong> As a power user running 180 apps, you are operating in a "high-density" scenario. In the future, it is always a good idea to "over-provision" kernel limits (<code>sysctl</code>) when you scale up.</li>
<li><strong>Cloudron's part:</strong> The platform should be "crash-safe." A kernel limit should cause a <strong>slowdown</strong> or a <strong>failed update</strong>, but it should never cause <strong>permanent configuration corruption</strong> that requires manual SSH intervention to fix.</li>
</ul>
<p dir="auto">In short: You hit the limit (common), but the software "broke the furniture" on its way down (avoidable).</p>
</blockquote>
<p dir="auto">With Cloudron 10.1 on the way, is there any merit to these AI recommendations ?</p>
]]></description><link>https://forum.cloudron.io/topic/16023/better-cloudron-release-rollout</link><generator>RSS for Node</generator><lastBuildDate>Tue, 29 Sep 2026 10:19:17 GMT</lastBuildDate><atom:link href="https://forum.cloudron.io/topic/16023.rss" rel="self" type="application/rss+xml"/><pubDate>Sun, 27 Sep 2026 22:17:39 GMT</pubDate><ttl>60</ttl><item><title><![CDATA[Reply to Better Cloudron release rollout on Mon, 28 Sep 2026 17:13:47 GMT]]></title><description><![CDATA[<blockquote>
<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/user/timconsidine" aria-label="Profile: timconsidine">@<bdi>timconsidine</bdi></a> <a href="/post/130112">said</a>:</p>
<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/user/necrevistonnezr" aria-label="Profile: necrevistonnezr">@<bdi>necrevistonnezr</bdi></a> what cloudron version are you on ? 9.2.0 ? or 10.0.5 ?</p>
</blockquote>
<p dir="auto">10.0.5</p>
]]></description><link>https://forum.cloudron.io/post/130141</link><guid isPermaLink="true">https://forum.cloudron.io/post/130141</guid><dc:creator><![CDATA[necrevistonnezr]]></dc:creator><pubDate>Mon, 28 Sep 2026 17:13:47 GMT</pubDate></item><item><title><![CDATA[Reply to Better Cloudron release rollout on Mon, 28 Sep 2026 13:04:12 GMT]]></title><description><![CDATA[<p dir="auto">I think also 20a1e2c955d011a06c89b552f9ca9d1b6a50f14f . I am working my way to 10.1.0, it's going through our CI and it will contain the fixes.</p>
]]></description><link>https://forum.cloudron.io/post/130135</link><guid isPermaLink="true">https://forum.cloudron.io/post/130135</guid><dc:creator><![CDATA[girish]]></dc:creator><pubDate>Mon, 28 Sep 2026 13:04:12 GMT</pubDate></item><item><title><![CDATA[Reply to Better Cloudron release rollout on Mon, 28 Sep 2026 12:55:48 GMT]]></title><description><![CDATA[<p dir="auto">I restricted functionality (no public view facility) in my AppsMonitor app so it works with 10.0.5. (Simpler proxyAuth allowed path).<br />
Given the additional functionality I have added to AppsMonitor, the public page (conceived as a status page) may not be a good idea anyway, so I might remove it permanently anyway. Other ways to do a status page.<br />
Will think about it.</p>
]]></description><link>https://forum.cloudron.io/post/130132</link><guid isPermaLink="true">https://forum.cloudron.io/post/130132</guid><dc:creator><![CDATA[timconsidine]]></dc:creator><pubDate>Mon, 28 Sep 2026 12:55:48 GMT</pubDate></item><item><title><![CDATA[Reply to Better Cloudron release rollout on Mon, 28 Sep 2026 08:07:10 GMT]]></title><description><![CDATA[<blockquote>
<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/user/girish" aria-label="Profile: girish">@<bdi>girish</bdi></a> <a href="/post/130110">said</a>:</p>
<p dir="auto">The nginx validation bug/corruption bug is fixed.</p>
</blockquote>
<p dir="auto">Thank you <a class="plugin-mentions-user plugin-mentions-a" href="/user/girish" aria-label="Profile: girish">@<bdi>girish</bdi></a></p>
<p dir="auto">In what cloudron version ?<br />
Do I need <a href="https://git.cloudron.io/platform/box/-/commit/6705b1c1107ec43ce09eff1945cd84ef9c60ff07" target="_blank" rel="noopener noreferrer nofollow ugc">https://git.cloudron.io/platform/box/-/commit/6705b1c1107ec43ce09eff1945cd84ef9c60ff07</a> ?</p>
<p dir="auto">I tried installing appsmonitor on the cloudron demo box and I get the same error.<br />
Apologies, I am not understanding how to deploy the fix.<br />
Seems like I have to wait for <code>10.0.1</code> or hack my box (which I'm not so keen to do).</p>
]]></description><link>https://forum.cloudron.io/post/130113</link><guid isPermaLink="true">https://forum.cloudron.io/post/130113</guid><dc:creator><![CDATA[timconsidine]]></dc:creator><pubDate>Mon, 28 Sep 2026 08:07:10 GMT</pubDate></item><item><title><![CDATA[Reply to Better Cloudron release rollout on Mon, 28 Sep 2026 07:36:14 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/user/necrevistonnezr" aria-label="Profile: necrevistonnezr">@<bdi>necrevistonnezr</bdi></a> what cloudron version are you on ? 9.2.0 ? or 10.0.5 ?</p>
]]></description><link>https://forum.cloudron.io/post/130112</link><guid isPermaLink="true">https://forum.cloudron.io/post/130112</guid><dc:creator><![CDATA[timconsidine]]></dc:creator><pubDate>Mon, 28 Sep 2026 07:36:14 GMT</pubDate></item><item><title><![CDATA[Reply to Better Cloudron release rollout on Mon, 28 Sep 2026 06:16:05 GMT]]></title><description><![CDATA[<p dir="auto">The nginx validation bug/corruption bug is fixed. To fix it, you can simply delete that bad nginx config file and systemctl restart nginx. Then the dashboard will be up and you can proceed to repair the apps. Unfortunate that a bug in code brings down the whole system.</p>
]]></description><link>https://forum.cloudron.io/post/130110</link><guid isPermaLink="true">https://forum.cloudron.io/post/130110</guid><dc:creator><![CDATA[girish]]></dc:creator><pubDate>Mon, 28 Sep 2026 06:16:05 GMT</pubDate></item><item><title><![CDATA[Reply to Better Cloudron release rollout on Mon, 28 Sep 2026 05:46:10 GMT]]></title><description><![CDATA[<blockquote>
<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/user/timconsidine" aria-label="Profile: timconsidine">@<bdi>timconsidine</bdi></a> <a href="/post/130103">said</a>:</p>
<blockquote>
<p dir="auto">But the logs do expose the real suspect. Line 417 (<code>writeAppLocationNginxConfig</code>) shows the generated proxyAuth nginx location:</p>
</blockquote>
<pre><code>"path":"!regexp:^/(public|health)(/|$)"
"location":"~ ^(?!(regexp:\\^/\\(public\\|health\\)\\(/\\|\\$\\)))"
</code></pre>
<blockquote>
<p dir="auto">Decoded, that location is <code>~ ^(?!(regexp:\^/\(public\|health\)\(/\|\$\)))</code> — the literal <code>regexp:</code> prefix leaked into the pattern and every char is backslash-escaped, so it matches a string, not a regexp. That is malformed and is what changed between box versions; it is not a <code>'</code>/<code>'</code> issue.</p>
</blockquote>
</blockquote>
<p dir="auto">That’s literally the bug I reported here: <a href="https://forum.cloudron.io/post/129507">https://forum.cloudron.io/post/129507</a> ?</p>
<p dir="auto">And BTW it’s not an issue for my <em>own</em> app using <code>proxyAuth</code>.</p>
]]></description><link>https://forum.cloudron.io/post/130108</link><guid isPermaLink="true">https://forum.cloudron.io/post/130108</guid><dc:creator><![CDATA[necrevistonnezr]]></dc:creator><pubDate>Mon, 28 Sep 2026 05:46:10 GMT</pubDate></item><item><title><![CDATA[Reply to Better Cloudron release rollout on Sun, 27 Sep 2026 23:33:09 GMT]]></title><description><![CDATA[<p dir="auto">Hello <a class="plugin-mentions-user plugin-mentions-a" href="/user/timconsidine" aria-label="Profile: timconsidine">@<bdi>timconsidine</bdi></a><br />
I just installed your community app on Cloudron 10.0.5 and also got the same issue.<br />
The latest Cloudron box code that is not yet released does not have this problem.<br />
It should be this commit <a href="https://git.cloudron.io/platform/box/-/commit/6705b1c1107ec43ce09eff1945cd84ef9c60ff07" target="_blank" rel="noopener noreferrer nofollow ugc">https://git.cloudron.io/platform/box/-/commit/6705b1c1107ec43ce09eff1945cd84ef9c60ff07</a> that fixes it the problem which is part of Cloudron 10.1 which is not yet released.<br />
But maybe you can hotfix your box with these changes for now.</p>
]]></description><link>https://forum.cloudron.io/post/130105</link><guid isPermaLink="true">https://forum.cloudron.io/post/130105</guid><dc:creator><![CDATA[james]]></dc:creator><pubDate>Sun, 27 Sep 2026 23:33:09 GMT</pubDate></item><item><title><![CDATA[Reply to Better Cloudron release rollout on Sun, 27 Sep 2026 23:08:30 GMT]]></title><description><![CDATA[<p dir="auto">Sorry, Cloudron team, I'm out of my depth here :</p>
<ul>
<li>appsmonitor installed and ran fine on 9.2.0</li>
<li>it won't start on Cloudron 10.0.5</li>
<li>uninstalled it and reinstalled it : same issue.</li>
</ul>
<blockquote>
<p dir="auto"><strong>No</strong> — <code>applogs.txt</code> nginx lines contain no <code>'</code> and no <code>&amp;#39;</code>. So apostrophes/entities are not the problem.</p>
</blockquote>
<blockquote>
<p dir="auto">But the logs do expose the real suspect. Line 417 (<code>writeAppLocationNginxConfig</code>) shows the generated proxyAuth nginx location:</p>
</blockquote>
<pre><code>"path":"!regexp:^/(public|health)(/|$)"
"location":"~ ^(?!(regexp:\\^/\\(public\\|health\\)\\(/\\|\\$\\)))"
</code></pre>
<blockquote>
<p dir="auto">Decoded, that location is <code>~ ^(?!(regexp:\^/\(public\|health\)\(/\|\$\)))</code> — the literal <code>regexp:</code> prefix leaked into the pattern and every char is backslash-escaped, so it matches a string, not a regexp. That is malformed and is what changed between box versions; it is not a <code>'</code>/<code>&amp;#39;</code> issue.</p>
</blockquote>
]]></description><link>https://forum.cloudron.io/post/130103</link><guid isPermaLink="true">https://forum.cloudron.io/post/130103</guid><dc:creator><![CDATA[timconsidine]]></dc:creator><pubDate>Sun, 27 Sep 2026 23:08:30 GMT</pubDate></item><item><title><![CDATA[Reply to Better Cloudron release rollout on Sun, 27 Sep 2026 23:22:54 GMT]]></title><description><![CDATA[<p dir="auto">And :</p>
<p dir="auto"><code>    location ~ &amp;#39;^/($|settings|app|api)(/|$)&amp;#39; {</code></p>
<blockquote>
<p dir="auto">This is not a system limit issue; this is a data corruption/template bug caused by the update.</p>
</blockquote>
<blockquote>
<p dir="auto">Somewhere in the Cloudron Manager's process, a string that should have been plain text (the Nginx regex) was HTML-encoded before being saved to the database. When the Manager writes the config file, it is literally printing HTML code into a plain-text Nginx configuration.</p>
</blockquote>
<blockquote>
<p dir="auto">Nginx sees ' and has no idea what it is. Because the characters are invalid, the Nginx parser gets confused and reports that the location directive was never properly opened with a {, even though the { is right there.</p>
</blockquote>
]]></description><link>https://forum.cloudron.io/post/130101</link><guid isPermaLink="true">https://forum.cloudron.io/post/130101</guid><dc:creator><![CDATA[timconsidine]]></dc:creator><pubDate>Sun, 27 Sep 2026 23:22:54 GMT</pubDate></item></channel></rss>