Cloudron Forum

I've added a new step into my update process for n8n - basically after updating, immediately restart the app. Something weird going on with some updates leaving the running instance in a broken state. Restarting seems to fix it (sometimes).

@marcusquinn really nice work Marcus. We're only allowed to use locally hosted models using OpenAI compatible endpoints. Does AIDevOps support local models? We have 10 or so nodes that can run anything from 24-70GB of RAM + context each on top (so most 30b models are compatible with big context). Any models, system prompts or settings you'd recommend for using this with locally hosted setups? I do like the sound of spooling up a bunch of workers and getting around the network limit by having them point to different nodes.

@andreasdueren said:

https://git.due.ren/andreas/hermes-cloudron/-/raw/main/CloudronVersions.json?ref_type=heads

Strange I can't access this link. Where is it hosted? I'd be accessing it from NZ.

Also, I don't think I can trust their release logs anymore either. "Fix for Project Settings" and "Fix for Gemini" across two releases yet none of my Workflows use Gemini and a UI fix shouldn't impact my workflows - yet somehow, there are changes beyond what is being reported in the change log that are breaking things. Trust level is diminishing with these guys.

@dualoswinwiz are you sure? 2.17.5 was good - however on Friday last week I updated to 2.17.7 and it looked like a similar bug crept in again where things are not waiting or duplicate requests are being fired. Didn't have time to establish the root cause, so I reverted to 2.17.5 and things are "back to normal". I'm getting real tired of these releases from n8n. Now I'm spending 15-20 minutes reviewing the Issues board 2-3 days after each release just to see if it'll be safe to upgrade a version.

This looks cool. What are the chances of getting this officially added to Cloudron?

I'd like to see this added into Cloudron.

Thank you @ethanxrosen @coniunctio and @divemasterza -- this was exactly the solution I was looking for.

@DualOSWinWiz said:

@Package-Updates please do not update yet as 2.16.1 has a bug as wait node does not work

upstream issue was already created by someone https://github.com/n8n-io/n8n/issues/28632

I don't think it has been fixed in 2.16.2 either. Not to mention there is a Websocket Connection issue. Pretty frustrating given there is a security release going out tomorrow and these are two major defects.

Glad to hear it! Thanks @nebulon.

Just a warning to those who follow this forum, make sure you update.

"Action required: update to the latest patch version of n8n

We recently informed you about upcoming patches and security advisories for high- or critical-severity security vulnerabilities in n8n.

These vulnerabilities have been fixed in the following n8n versions:

1.x: Versions < 1.123.27 are patched in 1.123.27
Stable: Versions >= 2.0.0 < 2.13.3 are patched in 2.13.3
Beta: Versions >= 2.14.0 < 2.14.1 are patched in 2.14.1
If you are running a version below the fixed version for your release branch, please upgrade to the applicable fixed version (or later) as soon as possible to protect your instance. For guidance on how to update your self-hosted instance, please refer to our updating documentation.

The related security advisories have been published. You can find links to the advisories below:

Critical | RCE via SQL Mode of Merge Node (CVE-2026-33660)
Critical | Prototype Pollution in GSuiteAdmin node parameters leads to RCE (CVE-2026-33696)
High | Credential Theft via Name-Based Resolution and Permission Checker Bypass in Community Edition (CVE-2026-33663)
High | In-Process Memory Disclosure in Task Runner (CVE-2026-27496)
High | LDAP Email-Based Account Linking Allows Privilege Escalation and Account Takeover (CVE-2026-33665)
High | SQL Injection in Data Table Node via orderByColumn Expression (CVE-2026-33713)
High | External Secrets Authorization Bypass in Credential Saving (CVE-2026-33722)
The information shared here is based on our current knowledge, and we will update you as soon as possible if our guidance changes.

Best regards,
The n8n Security Team"

Looks like this is working. However, some of my disk read/write nodes were broken after this update. I was able to fix it by updating the paths inside the nodes to explicitly include "/app/data/". Otherwise the error I get is something like "Problem in node ‘Write Files to Disk‘ EROFS: read-only file system, open: xxx".
Other than that, it appears the update is working correctly and binary form data is working again!
Thanks for your help!

@girish So that I can test this quickly: Is there any method available for me to directly upgrade to a n8n release in Cloudron or do we have to go through all of the releases to get there?

Here's the logs from when I stopped and started the instance. I truncated the logs so at places where you see 'etc etc', just imagine another 20+ Workflows listed with the same entry or error.

2026-03-12T14:05:01Z Stopping n8n...
2026-03-12T14:05:01Z [runnner:js] Received SIGTERM signal, shutting down...
2026-03-12T14:05:01Z [runnner:js] Task runner stopped
2026-03-12T14:05:02Z 2026-03-12 14:05:02,724 WARN received SIGTERM indicating exit request
2026-03-12T14:05:02Z 2026-03-12 14:05:02,728 INFO waiting for redis, redis-service to die
2026-03-12T14:05:02Z 2026-03-12 14:05:02,733 WARN stopped: redis-service (terminated by SIGTERM)
2026-03-12T14:05:02Z 2026-03-12 14:05:02,840 INFO stopped: redis (exit status 0)
2026-03-12T14:05:02Z 41:M 12 Mar 2026 14:05:02.835 * Saving the final RDB snapshot before exiting.
2026-03-12T14:05:02Z 41:M 12 Mar 2026 14:05:02.835 * User requested shutdown...
2026-03-12T14:05:02Z 41:M 12 Mar 2026 14:05:02.837 * DB saved on disk
2026-03-12T14:05:02Z 41:M 12 Mar 2026 14:05:02.837 * Removing the pid file.
2026-03-12T14:05:02Z 41:M 12 Mar 2026 14:05:02.838 # Redis is now ready to exit, bye bye...
2026-03-12T14:05:02Z 41:signal-handler (1773324302) Received SIGTERM scheduling shutdown...
2026-03-12T14:05:03.402Z box:tasks updating task 1873 with: {"percent":100,"message":"Done"}
2026-03-12T14:05:03.409Z box:tasks setCompleted - 1873: {"result":null,"error":null,"percent":100}
2026-03-12T14:05:03.409Z box:tasks updating task 1873 with: {"completed":true,"result":null,"error":null,"percent":100}
2026-03-12T14:05:03.411Z Exiting with code 0
2026-03-12T14:05:03.411Z box:taskworker Task took 2.087 seconds
2026-03-12T14:05:08.974Z box:taskworker Starting task 1874. Logs are at /home/yellowtent/platformdata/logs/xyz/apptask.log
2026-03-12T14:05:09.017Z box:taskworker Running task of type app
2026-03-12T14:05:09.025Z box:apptask run: startTask installationState: pending_start runState: running
2026-03-12T14:05:09.025Z box:tasks updating task 1874 with: {"percent":10,"message":"Starting app services"}
2026-03-12T14:05:09.226Z box:tasks updating task 1874 with: {"percent":35,"message":"Starting container"}
2026-03-12T14:05:09.423Z box:tasks updating task 1874 with: {"percent":80,"message":"Configuring reverse proxy"}
2026-03-12T14:05:09.430Z box:shell reverseproxy: openssl x509 -noout -subject -issuer
2026-03-12T14:05:09.506Z box:reverseproxy providerMatches: subject=CN = workflow.domain.com domain=workflow.domain.com issuer=C = US, O = Let's Encrypt, CN = E8 wildcard=false/false prod=true/true issuerMismatch=false wildcardMismatch=false match=true
2026-03-12T14:05:09.506Z box:shell reverseproxy: openssl x509 -startdate -enddate -subject -noout
2026-03-12T14:05:09.578Z box:reverseproxy expiryDate: subject=CN = workflow.domain.com notBefore=Feb 23 11:12:14 2026 GMT notAfter=May 24 11:12:13 2026 GMT daysLeft=72.87990071759259
2026-03-12T14:05:09.579Z box:reverseproxy ensureCertificate: workflow.domain.com acme cert exists and is up to date
2026-03-12T14:05:09.579Z box:reverseproxy needsRenewal: false. force: false
2026-03-12T14:05:09.589Z box:reverseproxy writeAppLocationNginxConfig: writing config for "workflow.domain.com" to /home/yellowtent/platformdata/nginx/applications/xyz/workflow.domain.com.conf with options {"sourceDir":"/home/yellowtent/box","vhost":"workflow.domain.com","hasIPv6":true,"ip":"{redacted}","port":5678,"endpoint":"app","redirectTo":null,"certFilePath":"/home/yellowtent/platformdata/nginx/cert/workflow.domain.com.cert","keyFilePath":"/home/yellowtent/platformdata/nginx/cert/workflow.domain.com.key","robotsTxtQuoted":"\"# Disable search engine indexing\\n\\nUser-agent: *\\nDisallow: /\"","cspQuoted":null,"hideHeaders":[],"proxyAuth":{"enabled":false,"id":"xyz","location":"/"},"upstreamUri":"","hstsPreload":false}
2026-03-12T14:05:09.590Z box:shell reverseproxy: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/restartservice.sh nginx
2026-03-12T14:05:09.649Z box:tasks updating task 1874 with: {"percent":100,"message":"Done"}
2026-03-12T14:05:09.676Z box:tasks setCompleted - 1874: {"result":null,"error":null,"percent":100}
2026-03-12T14:05:09.676Z box:tasks updating task 1874 with: {"completed":true,"result":null,"error":null,"percent":100}
2026-03-12T14:05:09.681Z Exiting with code 0
2026-03-12T14:05:09.681Z box:taskworker Task took 0.71 seconds
2026-03-12T14:05:09Z 2026-03-12 14:05:09,504 INFO Included extra file "/etc/supervisor/conf.d/redis-service.conf" during parsing
2026-03-12T14:05:09Z 2026-03-12 14:05:09,504 INFO Included extra file "/etc/supervisor/conf.d/redis.conf" during parsing
2026-03-12T14:05:09Z 2026-03-12 14:05:09,504 INFO Set uid to user 0 succeeded
2026-03-12T14:05:09Z 2026-03-12 14:05:09,514 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2026-03-12T14:05:09Z 2026-03-12 14:05:09,514 INFO RPC interface 'supervisor' initialized
2026-03-12T14:05:09Z 2026-03-12 14:05:09,514 INFO supervisord started with pid 1
2026-03-12T14:05:09Z ==> Ensure directories
2026-03-12T14:05:09Z ==> Setting permissions
2026-03-12T14:05:09Z ==> Starting N8N
2026-03-12T14:05:09Z ==> Starting supervisor
2026-03-12T14:05:10.000Z => Healthcheck error: Error: connect ECONNREFUSED {redacted}:5678
2026-03-12T14:05:10Z 13:C 12 Mar 2026 14:05:10.542 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2026-03-12T14:05:10Z 13:C 12 Mar 2026 14:05:10.542 * Configuration loaded
2026-03-12T14:05:10Z 13:C 12 Mar 2026 14:05:10.542 * Redis version=8.2.2, bits=64, commit=00000000, modified=1, pid=13, just started
2026-03-12T14:05:10Z 13:C 12 Mar 2026 14:05:10.542 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2026-03-12T14:05:10Z 13:M 12 Mar 2026 14:05:10.543 * monotonic clock: POSIX clock_gettime
2026-03-12T14:05:10Z 13:M 12 Mar 2026 14:05:10.544 # Failed to write PID file: Permission denied
2026-03-12T14:05:10Z 13:M 12 Mar 2026 14:05:10.544 * Running mode=standalone, port=6379.
2026-03-12T14:05:10Z 13:M 12 Mar 2026 14:05:10.545 * DB loaded from disk: 0.000 seconds
2026-03-12T14:05:10Z 13:M 12 Mar 2026 14:05:10.545 * Done loading RDB, keys loaded: 0, keys expired: 0.
2026-03-12T14:05:10Z 13:M 12 Mar 2026 14:05:10.545 * Loading RDB produced by version 8.2.2
2026-03-12T14:05:10Z 13:M 12 Mar 2026 14:05:10.545 * RDB age 8 seconds
2026-03-12T14:05:10Z 13:M 12 Mar 2026 14:05:10.545 * RDB memory usage when created 0.71 Mb
2026-03-12T14:05:10Z 13:M 12 Mar 2026 14:05:10.545 * Ready to accept connections tcp
2026-03-12T14:05:10Z 13:M 12 Mar 2026 14:05:10.545 * Server initialized
2026-03-12T14:05:10Z 2026-03-12 14:05:10,518 INFO spawned: 'redis' with pid 13
2026-03-12T14:05:10Z 2026-03-12 14:05:10,522 INFO spawned: 'redis-service' with pid 14
2026-03-12T14:05:10Z Redis service endpoint listening on http://:::3000
2026-03-12T14:05:11Z 2026-03-12 14:05:11,740 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-03-12T14:05:11Z 2026-03-12 14:05:11,741 INFO success: redis-service entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-03-12T14:05:13Z Initializing n8n process
2026-03-12T14:05:13Z n8n Task Broker ready on 127.0.0.1, port 5679
2026-03-12T14:05:13Z n8n ready on ::, port 5678
2026-03-12T14:05:14Z Deprecation warning: The storage directory "/app/data/user/.n8n/binaryData" will be renamed to "/app/data/user/.n8n/storage" in n8n v3. To migrate now, set N8N_MIGRATE_FS_STORAGE_PATH=true. If you have a volume mounted at the old path, update your mount configuration after migration.
2026-03-12T14:05:14Z Failed to start Python task runner in internal mode. because its virtual environment is missing from this system. Launching a Python runner in internal mode is intended only for debugging and is not recommended for production. Users are encouraged to deploy in external mode. See: https://docs.n8n.io/hosting/configuration/task-runners/#setting-up-external-mode
2026-03-12T14:05:14Z [license SDK] Skipping renewal on init: license cert is not due for renewal
2026-03-12T14:05:15Z (Use `node --trace-deprecation ...` to show where the warning was created)
2026-03-12T14:05:15Z (node:1) [DEP0169] DeprecationWarning: `url.parse()` behavior is not standardized and prone to errors that have security implications. Use the WHATWG URL API instead. CVEs are not issued for `url.parse()` vulnerabilities.
2026-03-12T14:05:15Z Registered runner "JS Task Runner" (YbZvK9naPF-koKlaM82Kt)
026-03-12T14:05:18Z - My Workflow 1 (ID: bR91JeCWm5Mx4sFs)
etc etc
2026-03-12T14:05:18Z Building workflow dependency index...
2026-03-12T14:05:18Z Currently active workflows:
2026-03-12T14:05:18Z Found unfinished executions: 1767388, 1767389, 1767392, 1767414, 1767422
2026-03-12T14:05:18Z Start Active Workflows:
2026-03-12T14:05:18Z This could be due to a crash of an active workflow or a restart of n8n
2026-03-12T14:05:18Z Version: 2.11.3
2026-03-12T14:05:18Z [Recovery] Logs available, amended execution
2026-03-12T14:05:18Z [Recovery] Logs available, amended execution
2026-03-12T14:05:18Z [Recovery] Logs available, amended execution
2026-03-12T14:05:18Z [Recovery] Logs available, amended execution
2026-03-12T14:05:18Z [Recovery] Logs available, amended execution
2026-03-12T14:05:19Z (node:1) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
2026-03-12T14:05:19Z Activated workflow "My Workflow 1" (ID: 313JrHCjBD8mSLx8)
2026-03-12T14:05:19Z Issue on initial workflow activation try of "My Workflow 2" (ID: JaltokDBM25yzRQF) (startup)
2026-03-12T14:05:19Z libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory
etc etc
2026-03-12T14:05:21Z Activation of workflow "My Workflow" (TjFAytoNhfUA6QT9) did fail with error: "libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory" | retry in 2 seconds
2026-03-12T14:05:21Z Editor is now accessible via:
2026-03-12T14:05:21Z Press "o" to open in Browser.
2026-03-12T14:05:21Z Try to activate workflow "My Workflow" (TjFAytoNhfUA6QT9)
2026-03-12T14:05:21Z libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory

@girish I can access the application (it comes up like normal), and some workflows are working (for example, error workflow that calls teams), but a large majority the workflows don't come up properly and show that error on boot. Most of those have scheduled starting triggers or web hooks. I didn't have time to try republishing them because I wasn't expecting it to fail this way (which is a much bigger failure than the form binary upload issue that only affects 5-6 of our workflows). I've got the logs, I'll sanitize and post here shortly.

@girish Sorry to report, but that update didn't fix the issue, in fact, it completely broke the n8n install.

Some sample errors from the logs before I had to restore from backup:

2026-03-12T14:05:51Z Activation of workflow "My Workflow 1" (FMTzz3R99WCvdr2P) did fail with error: "libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory" | retry in 32 seconds
2026-03-12T14:05:51Z libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory

That failure happened x every active/published workflow (40+). So I got a tonne of error notifications sent to Teams after the container started.

Is there a way to choose which update we want to jump to? Every release from 9.4.2 is broken so we are updating multiple times 0>1>2>3>4>Current, then reverting to 0 when it doesn't work.

@James @brutalbirdie @Joseph

Much appreciated team.

@girish @Joseph n8n team are claiming the issue could be with changes made by Cloudron between 2.9.4 and above. Can you guys look into this?

There are multiple people claiming that sending binaries via multi-part-forms breaks after upgrading from 2.9.4 - but I don't know if they're using Cloudron.

Here's an example of someone getting the same error sending a multi-part form using the Open AI file upload:
https://github.com/n8n-io/n8n/issues/26777

Other references to connection issues when uploading files:
https://github.com/n8n-io/n8n/issues/26746
https://github.com/n8n-io/n8n/issues/26606
https://github.com/n8n-io/n8n/issues/25567

Please note this error persists into the latest version of n8n.

Specifically, I have a sub workflow that is executed that uses an HTTP request to upload files to a third party service. There are a lot of changes between 2.9.4 (working) and 2.10.2 (breaks). I'll continue to investigate and report back if I find anything.

https://github.com/n8n-io/n8n/issues/26748