Warning: Connection Reset Issue in [4.9.0] which is n8n 2.10.2

[umnz]

@James @brutalbirdie @Joseph

Much appreciated team.

[girish]

@umnz @paramedicspecialist I pushed a new package that does not use npm anymore and uses the upstream docker image. Can you please check?

[umnz]

@girish Sorry to report, but that update didn't fix the issue, in fact, it completely broke the n8n install.

Some sample errors from the logs before I had to restore from backup:

2026-03-12T14:05:51Z Activation of workflow "My Workflow 1" (FMTzz3R99WCvdr2P) did fail with error: "libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory" | retry in 32 seconds
2026-03-12T14:05:51Z libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory

That failure happened x every active/published workflow (40+). So I got a tonne of error notifications sent to Teams after the container started.

Is there a way to choose which update we want to jump to? Every release from 9.4.2 is broken so we are updating multiple times 0>1>2>3>4>Current, then reverting to 0 when it doesn't work.

[girish]

@umnz oh, that's not good. I will revoke the release for now.

[Lars.Emil]

Hi @staff,

I am experiencing a systemic failure in n8n version 2.11.3 (Cloudron package 4.9.x) where most workflows fail to activate or execute with the following error:

libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory

This appears to be a library mismatch introduced in a recent update. The error triggers specifically on nodes that utilize the JS sandbox (Merge nodes, Code nodes, etc.).

Technical findings:
I ran ldd on the isolated-vm binary inside the container, and it confirms that the module is compiled against musl (Alpine), while the Cloudron environment is glibc based (Ubuntu/Debian):

root@...:/app/code# ldd /app/code/node_modules/isolated-vm/out/isolated_vm.node
    linux-vdso.so.1 (0x00007ffca68bd000)
    libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x000071f4e608e000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x000071f4e6060000)
    libc.musl-x86_64.so.1 => not found
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000071f4e5f77000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000071f4e5d65000)
    /lib64/ld-linux-x86-64.so.2 (0x000071f4e6421000)

Observed behavior:

Workflow activation fails on startup for any workflow containing a Merge or Code node.

The Python task runner also fails to start in internal mode (virtual environment missing).

Setting N8N_BLOCK_JS_SANDBOX=true does not resolve the activation errors.

It seems the latest package might have been built/bundled in an Alpine environment by mistake, or a dependency has leaked a musl requirement into the glibc image.

Could you please look into recompiling the dependencies for the current base image?

[girish]

Thing is the Dockerfile is installing musl, so I am not sure what is going wrong here. Hmm

[girish]

@umnz so it doesn't even start up in your case or do you see this during a specific workflow? It installs and runs fine here

[umnz]

@girish I can access the application (it comes up like normal), and some workflows are working (for example, error workflow that calls teams), but a large majority the workflows don't come up properly and show that error on boot. Most of those have scheduled starting triggers or web hooks. I didn't have time to try republishing them because I wasn't expecting it to fail this way (which is a much bigger failure than the form binary upload issue that only affects 5-6 of our workflows). I've got the logs, I'll sanitize and post here shortly.

[girish]

@umnz I found the problem. The location of musl is different in alpine and ubuntu. Making a new package.

Our tests are not comprehensive enough to detect workflow errors. They only test if n8n comes up and one can login etc. Have to improve this.

[umnz]

Here's the logs from when I stopped and started the instance. I truncated the logs so at places where you see 'etc etc', just imagine another 20+ Workflows listed with the same entry or error.

2026-03-12T14:05:01Z Stopping n8n...
2026-03-12T14:05:01Z [runnner:js] Received SIGTERM signal, shutting down...
2026-03-12T14:05:01Z [runnner:js] Task runner stopped
2026-03-12T14:05:02Z 2026-03-12 14:05:02,724 WARN received SIGTERM indicating exit request
2026-03-12T14:05:02Z 2026-03-12 14:05:02,728 INFO waiting for redis, redis-service to die
2026-03-12T14:05:02Z 2026-03-12 14:05:02,733 WARN stopped: redis-service (terminated by SIGTERM)
2026-03-12T14:05:02Z 2026-03-12 14:05:02,840 INFO stopped: redis (exit status 0)
2026-03-12T14:05:02Z 41:M 12 Mar 2026 14:05:02.835 * Saving the final RDB snapshot before exiting.
2026-03-12T14:05:02Z 41:M 12 Mar 2026 14:05:02.835 * User requested shutdown...
2026-03-12T14:05:02Z 41:M 12 Mar 2026 14:05:02.837 * DB saved on disk
2026-03-12T14:05:02Z 41:M 12 Mar 2026 14:05:02.837 * Removing the pid file.
2026-03-12T14:05:02Z 41:M 12 Mar 2026 14:05:02.838 # Redis is now ready to exit, bye bye...
2026-03-12T14:05:02Z 41:signal-handler (1773324302) Received SIGTERM scheduling shutdown...
2026-03-12T14:05:03.402Z box:tasks updating task 1873 with: {"percent":100,"message":"Done"}
2026-03-12T14:05:03.409Z box:tasks setCompleted - 1873: {"result":null,"error":null,"percent":100}
2026-03-12T14:05:03.409Z box:tasks updating task 1873 with: {"completed":true,"result":null,"error":null,"percent":100}
2026-03-12T14:05:03.411Z Exiting with code 0
2026-03-12T14:05:03.411Z box:taskworker Task took 2.087 seconds
2026-03-12T14:05:08.974Z box:taskworker Starting task 1874. Logs are at /home/yellowtent/platformdata/logs/xyz/apptask.log
2026-03-12T14:05:09.017Z box:taskworker Running task of type app
2026-03-12T14:05:09.025Z box:apptask run: startTask installationState: pending_start runState: running
2026-03-12T14:05:09.025Z box:tasks updating task 1874 with: {"percent":10,"message":"Starting app services"}
2026-03-12T14:05:09.226Z box:tasks updating task 1874 with: {"percent":35,"message":"Starting container"}
2026-03-12T14:05:09.423Z box:tasks updating task 1874 with: {"percent":80,"message":"Configuring reverse proxy"}
2026-03-12T14:05:09.430Z box:shell reverseproxy: openssl x509 -noout -subject -issuer
2026-03-12T14:05:09.506Z box:reverseproxy providerMatches: subject=CN = workflow.domain.com domain=workflow.domain.com issuer=C = US, O = Let's Encrypt, CN = E8 wildcard=false/false prod=true/true issuerMismatch=false wildcardMismatch=false match=true
2026-03-12T14:05:09.506Z box:shell reverseproxy: openssl x509 -startdate -enddate -subject -noout
2026-03-12T14:05:09.578Z box:reverseproxy expiryDate: subject=CN = workflow.domain.com notBefore=Feb 23 11:12:14 2026 GMT notAfter=May 24 11:12:13 2026 GMT daysLeft=72.87990071759259
2026-03-12T14:05:09.579Z box:reverseproxy ensureCertificate: workflow.domain.com acme cert exists and is up to date
2026-03-12T14:05:09.579Z box:reverseproxy needsRenewal: false. force: false
2026-03-12T14:05:09.589Z box:reverseproxy writeAppLocationNginxConfig: writing config for "workflow.domain.com" to /home/yellowtent/platformdata/nginx/applications/xyz/workflow.domain.com.conf with options {"sourceDir":"/home/yellowtent/box","vhost":"workflow.domain.com","hasIPv6":true,"ip":"{redacted}","port":5678,"endpoint":"app","redirectTo":null,"certFilePath":"/home/yellowtent/platformdata/nginx/cert/workflow.domain.com.cert","keyFilePath":"/home/yellowtent/platformdata/nginx/cert/workflow.domain.com.key","robotsTxtQuoted":"\"# Disable search engine indexing\\n\\nUser-agent: *\\nDisallow: /\"","cspQuoted":null,"hideHeaders":[],"proxyAuth":{"enabled":false,"id":"xyz","location":"/"},"upstreamUri":"","hstsPreload":false}
2026-03-12T14:05:09.590Z box:shell reverseproxy: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/restartservice.sh nginx
2026-03-12T14:05:09.649Z box:tasks updating task 1874 with: {"percent":100,"message":"Done"}
2026-03-12T14:05:09.676Z box:tasks setCompleted - 1874: {"result":null,"error":null,"percent":100}
2026-03-12T14:05:09.676Z box:tasks updating task 1874 with: {"completed":true,"result":null,"error":null,"percent":100}
2026-03-12T14:05:09.681Z Exiting with code 0
2026-03-12T14:05:09.681Z box:taskworker Task took 0.71 seconds
2026-03-12T14:05:09Z 2026-03-12 14:05:09,504 INFO Included extra file "/etc/supervisor/conf.d/redis-service.conf" during parsing
2026-03-12T14:05:09Z 2026-03-12 14:05:09,504 INFO Included extra file "/etc/supervisor/conf.d/redis.conf" during parsing
2026-03-12T14:05:09Z 2026-03-12 14:05:09,504 INFO Set uid to user 0 succeeded
2026-03-12T14:05:09Z 2026-03-12 14:05:09,514 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2026-03-12T14:05:09Z 2026-03-12 14:05:09,514 INFO RPC interface 'supervisor' initialized
2026-03-12T14:05:09Z 2026-03-12 14:05:09,514 INFO supervisord started with pid 1
2026-03-12T14:05:09Z ==> Ensure directories
2026-03-12T14:05:09Z ==> Setting permissions
2026-03-12T14:05:09Z ==> Starting N8N
2026-03-12T14:05:09Z ==> Starting supervisor
2026-03-12T14:05:10.000Z => Healthcheck error: Error: connect ECONNREFUSED {redacted}:5678
2026-03-12T14:05:10Z 13:C 12 Mar 2026 14:05:10.542 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2026-03-12T14:05:10Z 13:C 12 Mar 2026 14:05:10.542 * Configuration loaded
2026-03-12T14:05:10Z 13:C 12 Mar 2026 14:05:10.542 * Redis version=8.2.2, bits=64, commit=00000000, modified=1, pid=13, just started
2026-03-12T14:05:10Z 13:C 12 Mar 2026 14:05:10.542 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2026-03-12T14:05:10Z 13:M 12 Mar 2026 14:05:10.543 * monotonic clock: POSIX clock_gettime
2026-03-12T14:05:10Z 13:M 12 Mar 2026 14:05:10.544 # Failed to write PID file: Permission denied
2026-03-12T14:05:10Z 13:M 12 Mar 2026 14:05:10.544 * Running mode=standalone, port=6379.
2026-03-12T14:05:10Z 13:M 12 Mar 2026 14:05:10.545 * DB loaded from disk: 0.000 seconds
2026-03-12T14:05:10Z 13:M 12 Mar 2026 14:05:10.545 * Done loading RDB, keys loaded: 0, keys expired: 0.
2026-03-12T14:05:10Z 13:M 12 Mar 2026 14:05:10.545 * Loading RDB produced by version 8.2.2
2026-03-12T14:05:10Z 13:M 12 Mar 2026 14:05:10.545 * RDB age 8 seconds
2026-03-12T14:05:10Z 13:M 12 Mar 2026 14:05:10.545 * RDB memory usage when created 0.71 Mb
2026-03-12T14:05:10Z 13:M 12 Mar 2026 14:05:10.545 * Ready to accept connections tcp
2026-03-12T14:05:10Z 13:M 12 Mar 2026 14:05:10.545 * Server initialized
2026-03-12T14:05:10Z 2026-03-12 14:05:10,518 INFO spawned: 'redis' with pid 13
2026-03-12T14:05:10Z 2026-03-12 14:05:10,522 INFO spawned: 'redis-service' with pid 14
2026-03-12T14:05:10Z Redis service endpoint listening on http://:::3000
2026-03-12T14:05:11Z 2026-03-12 14:05:11,740 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-03-12T14:05:11Z 2026-03-12 14:05:11,741 INFO success: redis-service entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-03-12T14:05:13Z Initializing n8n process
2026-03-12T14:05:13Z n8n Task Broker ready on 127.0.0.1, port 5679
2026-03-12T14:05:13Z n8n ready on ::, port 5678
2026-03-12T14:05:14Z Deprecation warning: The storage directory "/app/data/user/.n8n/binaryData" will be renamed to "/app/data/user/.n8n/storage" in n8n v3. To migrate now, set N8N_MIGRATE_FS_STORAGE_PATH=true. If you have a volume mounted at the old path, update your mount configuration after migration.
2026-03-12T14:05:14Z Failed to start Python task runner in internal mode. because its virtual environment is missing from this system. Launching a Python runner in internal mode is intended only for debugging and is not recommended for production. Users are encouraged to deploy in external mode. See: https://docs.n8n.io/hosting/configuration/task-runners/#setting-up-external-mode
2026-03-12T14:05:14Z [license SDK] Skipping renewal on init: license cert is not due for renewal
2026-03-12T14:05:15Z (Use `node --trace-deprecation ...` to show where the warning was created)
2026-03-12T14:05:15Z (node:1) [DEP0169] DeprecationWarning: `url.parse()` behavior is not standardized and prone to errors that have security implications. Use the WHATWG URL API instead. CVEs are not issued for `url.parse()` vulnerabilities.
2026-03-12T14:05:15Z Registered runner "JS Task Runner" (YbZvK9naPF-koKlaM82Kt)
026-03-12T14:05:18Z - My Workflow 1 (ID: bR91JeCWm5Mx4sFs)
etc etc
2026-03-12T14:05:18Z Building workflow dependency index...
2026-03-12T14:05:18Z Currently active workflows:
2026-03-12T14:05:18Z Found unfinished executions: 1767388, 1767389, 1767392, 1767414, 1767422
2026-03-12T14:05:18Z Start Active Workflows:
2026-03-12T14:05:18Z This could be due to a crash of an active workflow or a restart of n8n
2026-03-12T14:05:18Z Version: 2.11.3
2026-03-12T14:05:18Z [Recovery] Logs available, amended execution
2026-03-12T14:05:18Z [Recovery] Logs available, amended execution
2026-03-12T14:05:18Z [Recovery] Logs available, amended execution
2026-03-12T14:05:18Z [Recovery] Logs available, amended execution
2026-03-12T14:05:18Z [Recovery] Logs available, amended execution
2026-03-12T14:05:19Z (node:1) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
2026-03-12T14:05:19Z Activated workflow "My Workflow 1" (ID: 313JrHCjBD8mSLx8)
2026-03-12T14:05:19Z Issue on initial workflow activation try of "My Workflow 2" (ID: JaltokDBM25yzRQF) (startup)
2026-03-12T14:05:19Z libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory
etc etc
2026-03-12T14:05:21Z Activation of workflow "My Workflow" (TjFAytoNhfUA6QT9) did fail with error: "libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory" | retry in 2 seconds
2026-03-12T14:05:21Z Editor is now accessible via:
2026-03-12T14:05:21Z Press "o" to open in Browser.
2026-03-12T14:05:21Z Try to activate workflow "My Workflow" (TjFAytoNhfUA6QT9)
2026-03-12T14:05:21Z libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory

[girish]

@umnz https://git.cloudron.io/packages/n8n-app/-/merge_requests/158

[umnz]

@girish So that I can test this quickly: Is there any method available for me to directly upgrade to a n8n release in Cloudron or do we have to go through all of the releases to get there?

[james]

Hello @lars.emil and welcome to the Cloudron forum

Sorry for the disruption.
We are currently deploying a new app update that should fix the issue.

---

Hello @umnz

@umnz said:

Is there any method available for me to directly upgrade to a n8n release in Cloudron or do we have to go through all of the releases to get there?

No. You will have to update each sequential update.

[girish]

@umnz @lars.emil Can you please try the latest package 4.13.0? @james helped me test and it should work now.