Process hanging when starting n8n - Starting migration AddConstraintToExecutionMetadata1720101653148

[AmbroiseUnly]

Restarting the app didn't work.

There was an awaiting Ubuntu update, I restarted the whole server and n8n booted properly this time.
So, it seems to be related to Ubuntu update, which got stuck the whole startup process.
I'm not sure what can be done to avoid this kind of situation.

[joseph]

Upstream thread - https://community.n8n.io/t/process-hanging-when-starting-n8n-starting-migration-1720101653148/50380/4

@AmbroiseUnly If the issue comes up again, we should investigate in more detail. Thanks for reporting

[zonzonzon]

I encountered a similar issue. After updating to the latest Cloudron version, I restarted Cloudron as prompted by the system. After that, N8N was unable to connect.

Feb 06 21:09:50 => Healthcheck error: Error: connect ECONNREFUSED 172.18.18.255:5678

[joseph]

@zonzonzon can you paste more logs? that line only shows that n8n is not responding

[zonzonzon]

...
Feb 06 21:09:20 box:reverseproxy providerMatches: subject=CN = webapp.abc.com domain=webapp.abc.com issuer=C = US, O = Let's Encrypt, CN = E6 wildcard=false/false prod=true/true issuerMismatch=false wildcardMismatch=false match=true
...
box:reverseproxy writeAppLocationNginxConfig: writing config for "webapp.abc.com" to /home/yellowtent/platformdata/nginx/applications/e83dd39f-2eb7-449a-bbd4-1e9186013b24/webapp.abc.com.conf with options {"sourceDir":"/home/yellowtent/box","vhost":"webapp.abc.com","hasIPv6":true,"ip":"172.18.0.255","port":5678,"endpoint":"app"...}
...
nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "e5.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/app1.abc.com.cert"
nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "e5.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/app2.abc.com.cert"
nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "e6.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/app3.abc.com.cert"
nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "e6.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/webapp.abc.com.cert"
...
=> Healtheck error: Error: connect ECONNREFUSED 172.18.0.255:5678

[zonzonzon]

The process is as follows:

I saw a notification to restart after updating to the latest Cloudron version and followed that notification, restarting in Cloudron.
N8n was inaccessible and there was a log message as I sent above.
Other apps run normally, only n8n has an error.
N8n status Starting...

[nebulon]

Is n8n itself starting up within the container according to the app logs? Maybe look at the logs while restarting the n8n app instance to get a better picture.

[zonzonzon]

I have tried restarting all other apps, and they are working except for n8n, which is having issues. Below is the log after restarting n8n.

Feb 06 21:32:01 box:taskworker Running task of type app
Feb 06 21:32:01 box:apptask run: startTask installationState: pending_restart runState: running 
Feb 06 21:32:01 box:tasks update 4449: {"percent":10,"message":"Starting app services"}
Feb 06 21:32:01 box:tasks update 4449: {"percent":20,"message":"Restarting container"}
Feb 06 21:32:11 => Ensure directories
Feb 06 21:32:11 => Loading configuration
Feb 06 21:32:11 box:tasks update 4449: {"percent":80,"message":"Configuring reverse proxy"}
Feb 06 21:32:11 box:shell reverseproxy: openssl x509 -noout -subject -issuer
Feb 06 21:32:11 box:reverseproxy providerMatches: subject=CN = webapp.domain.com domain=webapp.domain.com issuer=C = US, O = Let's Encrypt, CN = E6 wildcard=false/false prod=true/true issuerMismatch=false wildcardMismatch=false match=true
Feb 06 21:32:11 box:shell reverseproxy: openssl x509 -startdate -enddate -subject -noout  
Feb 06 21:32:11 box:reverseproxy expiryDate: subject=CN = webapp.domain.com notBefore=Jan 9 09:12:15 2025 GMT notAfter=Apr 9 09:12:14 2025 GMT daysLeft=61.777801840277775
Feb 06 21:32:11 box:reverseproxy needsRenewal: false. force: false
Feb 06 21:32:11 box:reverseproxy ensureCertificate: webapp.domain.com acme cert exists and is up to date
Feb 06 21:32:11 box:shell reverseproxy: openssl x509 -in /home/yellowtent/platformdata/nginx/cert/webapp.domain.com.cert -noout -ocsp_uri
Feb 06 21:32:11 box:reverseproxy writeAppLocationNginxConfig: writing config for "webapp.domain.com" to /home/yellowtent/platformdata/nginx/applications/e83dd39f-2eb7-449a-bbd4-1e9186013b24/webapp.domain.com.conf with options {"sourceDir":"/home/yellowtent/box","vhost":"webapp.domain.com","hasIPv6":true,"ip":"172.18.18.255","port":5678,"endpoint":"app","redirectTo":null,"certFilePath":"/home/yellowtent/platformdata/nginx/cert/webapp.domain.com.cert","keyFilePath":"/home/yellowtent/platformdata/nginx/cert/webapp.domain.com.key","robotsTxtQuoted":"\"# Disable search engine indexing\\n\\nUser-agent: *\\nDisallow: /\"","cspQuoted":null,"hideHeaders":[],"proxyAuth":{"enabled":false,"id":"e83dd39f-2eb7-449a-bbd4-1e9186013b24","location":"/"},"upstreamUri":"","ocsp":true,"hstsPreload":false}
Feb 06 21:32:11 box:shell reverseproxy /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
Feb 06 21:32:11 => Setting permissions
Feb 06 21:32:11 box:shell nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "e5.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/app1.domain.com.cert"
Feb 06 21:32:11
Feb 06 21:32:11 box:shell nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "e5.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/app2.domain.com.cert"
Feb 06 21:32:11
Feb 06 21:32:11 box:shell nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "e6.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/app3.domain.com.cert"
Feb 06 21:32:11
Feb 06 21:32:11 box:shell nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "e6.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/webapp.domain.com.cert"
Feb 06 21:32:11
Feb 06 21:32:12 box:tasks update 4449: {"percent":100,"message":"Done"}
Feb 06 21:32:12 box:taskworker Task took 10.636 seconds
Feb 06 21:32:12 box:tasks setCompleted - 4449: {"result":null,"error":null}
Feb 06 21:32:12 box:tasks update 4449: {"percent":100,"result":null,"error":null}
Feb 06 21:32:12 => Installing handlebars@4.7.7 jsonata@2.0.2 marked@4.3.0 bwip-js@3.3.0 ajv-formats@2.1.1 odoo-xmlrpc@1.0.8 firebase-admin@11.10.1
Feb 06 21:32:20 => Healtheck error: Error: connect ECONNREFUSED 172.18.18.255:5678
Feb 06 21:32:30 => Healtheck error: Error: connect ECONNREFUSED 172.18.18.255:5678

[zonzonzon]

Get notify on cloudron and click reboot

After restarting, n8n cannot connect and there is a log as I sent above

@nebulon

[nebulon]

You can ignore the reboot required notification if you have already rebooted.

Since the app startup logs basically end with

...
=> Installing handlebars@4.7.7 jsonata@....

This means the n8n server itself hasn't started yet. Does the app automatically restart? If so maybe it runs out of memory.

Otherwise does it ever print anything more than healthecheck failures after that?
Depending on the plugins you have installed, it may just take a very long time or some plugins are just not compatible anymore.

[joseph]

Good catch @nebulon .

@zonzonzon https://docs.cloudron.io/apps/n8n/#custom-node-modules . Just edit /app/data/env.sh and set EXTRA_NODE_MODULES to empty string and then restart the app, see if that helps.