Process hanging when starting n8n - Starting migration AddConstraintToExecutionMetadata1720101653148

AmbroiseUnly

Apparently, my n8n app is down with "n8n is starting up. Please wait" since yesterday and I'm noticing it just now.

The issue seems to be related to an update. I restarted the app and now it's stuck at Starting migration AddConstraintToExecutionMetadata1720101653148 for 15mn without anything happening in the logs and web app is still down.

Is that a known issue?

Jul 18 23:01:44 box:docker pullImage: {"status":"Extracting","progressDetail":{"current":1372,"total":1372},"progress":"[==================================================>] 1.372kB/1.372kB","id":"3f0342ae64ba"}
Jul 18 23:01:44 box:docker pullImage: {"status":"Extracting","progressDetail":{"current":1372,"total":1372},"progress":"[==================================================>] 1.372kB/1.372kB","id":"3f0342ae64ba"}
Jul 18 23:01:44 box:docker pullImage: {"status":"Pull complete","progressDetail":{},"id":"3f0342ae64ba"}
Jul 18 23:01:44 box:docker pullImage: {"status":"Digest: sha256:b7f6e86bb634b0fc68c2020ec7760e3de437db744ac474a70137b16590a496a4"}
Jul 18 23:01:44 box:docker pullImage: {"status":"Status: Downloaded newer image for cloudron/io.n8n.cloudronapp:20240718-050754-90025e1e0"}
Jul 18 23:01:44 box:docker downloaded image cloudron/io.n8n.cloudronapp:20240718-050754-90025e1e0 . error: false
Jul 18 23:01:44 box:tasks update 1511: {"percent":35,"message":"Cleaning up old install"}
Jul 18 23:01:44 box:apptask deleteContainer: deleting app containers (app, scheduler)
Jul 18 23:01:44 box:shell removeLogrotateConfig /usr/bin/sudo -S /home/yellowtent/box/src/scripts/configurelogrotate.sh remove 693caa42-bcf9-44c2-8f44-78f8676fe44c
Jul 18 23:02:18 box:services teardownAddons: Tearing down []
Jul 18 23:02:18 box:tasks update 1511: {"percent":45,"message":"Downloading icon"}
Jul 18 23:02:18 box:apptask downloadIcon: Downloading icon of io.n8n.cloudronapp@3.41.0
Jul 18 23:02:19 box:tasks update 1511: {"percent":60,"message":"Updating addons"}
Jul 18 23:02:19 box:services setupAddons: Setting up ["localstorage","postgresql","sendmail"]
Jul 18 23:02:19 box:services setupAddons: setting up addon localstorage with options {}
Jul 18 23:02:19 box:services setupLocalStorage
Jul 18 23:02:19 box:shell createVolume /usr/bin/sudo -S /home/yellowtent/box/src/scripts/mkdirvolume.sh /home/yellowtent/appsdata/693caa42-bcf9-44c2-8f44-78f8676fe44c/data
Jul 18 23:02:19 box:services setupAddons: setting up addon postgresql with options {}
Jul 18 23:02:19 box:services Setting up postgresql
Jul 18 23:02:19 box:services Setting postgresql addon config to [{"name":"CLOUDRON_POSTGRESQL_URL","value":"postgres://user693caa42bcf944c28f4478f8676fe44c:e86932a5f7a3594662665ca9471e8369daecdaf0047796c38a928558e2678cd16a39086932e427b94f104606d5d9534487d9d9020683d8e13bd3f2280a3385f5@postgresql/db693caa42bcf944c28f4478f8676fe44c"},{"name":"CLOUDRON_POSTGRESQL_USERNAME","value":"user693caa42bcf944c28f4478f8676fe44c"},{"name":"CLOUDRON_POSTGRESQL_PASSWORD","value":"e86932a5f7a3594662665ca9471e8369daecdaf0047796c38a928558e2678cd16a39086932e427b94f104606d5d9534487d9d9020683d8e13bd3f2280a3385f5"},{"name":"CLOUDRON_POSTGRESQL_HOST","value":"postgresql"},{"name":"CLOUDRON_POSTGRESQL_PORT","value":"5432"},{"name":"CLOUDRON_POSTGRESQL_DATABASE","value":"db693caa42bcf944c28f4478f8676fe44c"}]
Jul 18 23:02:19 box:services setupAddons: setting up addon sendmail with options {"supportsDisplayName":true}
Jul 18 23:02:19 box:services Setting up SendMail
Jul 18 23:02:19 box:services Setting sendmail addon config to [{"name":"CLOUDRON_MAIL_SMTP_SERVER","value":"mail"},{"name":"CLOUDRON_MAIL_SMTP_PORT","value":"2525"},{"name":"CLOUDRON_MAIL_SMTPS_PORT","value":"2465"},{"name":"CLOUDRON_MAIL_STARTTLS_PORT","value":"2587"},{"name":"CLOUDRON_MAIL_SMTP_USERNAME","value":"n8n@unly.org"},{"name":"CLOUDRON_MAIL_SMTP_PASSWORD","value":"a36d29838cf0ff6943b73e69475aa61a6bfda56caff91d43"},{"name":"CLOUDRON_MAIL_FROM","value":"n8n@unly.org"},{"name":"CLOUDRON_MAIL_DOMAIN","value":"unly.org"},{"name":"CLOUDRON_MAIL_FROM_DISPLAY_NAME","value":"my.n8n.unly.org"}]
Jul 18 23:02:19 box:tasks update 1511: {"percent":70,"message":"Creating container"}
Jul 18 23:02:19 box:apptask createContainer: creating container
Jul 18 23:02:19 box:shell getSwaps execArgs: swapon ["--noheadings","--raw","--bytes","--show=type,size,used,name"]
Jul 18 23:02:57 box:shell addLogrotateConfig /usr/bin/sudo -S /home/yellowtent/box/src/scripts/configurelogrotate.sh add 693caa42-bcf9-44c2-8f44-78f8676fe44c /tmp/693caa42-bcf9-44c2-8f44-78f8676fe44c.logrotate
Jul 18 23:02:57 box:apptask startApp: starting container
Jul 18 23:02:59 box:tasks update 1511: {"percent":90,"message":"Configuring reverse proxy"}
Jul 18 23:02:59 box:tasks update 1511: {"percent":100,"message":"Done"}
Jul 18 23:02:59 box:tasks setCompleted - 1511: {"result":null,"error":null}
Jul 18 23:02:59 box:tasks update 1511: {"percent":100,"result":null,"error":null}
Jul 18 23:02:59 box:taskworker Task took 177.89 seconds
Jul 19 09:10:35 Stopping n8n... "{ file: 'start.js', function: 'stopProcess' }"
Jul 19 09:10:35 2024-07-19T07:10:35.205Z | info | Received SIGTERM. Shutting down... "{ file: 'BaseCommand.js' }"
Jul 19 09:10:35 2024-07-19T07:10:35.211Z | debug | Shutting down component "Queue.pause()" "{ file: 'Shutdown.service.js', function: 'shutdownComponent' }"
Jul 19 09:10:35 2024-07-19T07:10:35.212Z | info |
Jul 19 09:10:35 2024-07-19T07:10:35.214Z | error | Error: There was an error shutting down n8n. "{ file: 'LoggerProxy.js', function: 'exports.error' }"
Jul 19 09:10:35 2024-07-19T07:10:35.214Z | error | TypeError: Cannot read properties of undefined (reading 'removeAllQueuedWorkflowActivations') "{ file: 'LoggerProxy.js', function: 'exports.error' }"
Jul 19 09:10:35 2024-07-19T07:10:35.216Z | debug | Shutting down component "License.shutdown()" "{ file: 'Shutdown.service.js', function: 'shutdownComponent' }"
Jul 19 09:10:35 2024-07-19T07:10:35.216Z | debug | Shutting down component "Push.onShutdown()" "{ file: 'Shutdown.service.js', function: 'shutdownComponent' }"
Jul 19 09:10:35 2024-07-19T07:10:35.217Z | debug | Shutting down component "ActiveWorkflowManager.removeAllTriggerAndPollerBasedWorkflows()" "{ file: 'Shutdown.service.js', function: 'shutdownComponent' }"
Jul 19 09:10:35 2024-07-19T07:10:35.217Z | debug | Shutting down component "ExecutionRecoveryService.shutdown()" "{ file: 'Shutdown.service.js', function: 'shutdownComponent' }"
Jul 19 09:10:35 2024-07-19T07:10:35.218Z | debug | Removing all trigger- and poller-based workflows "{\n file: 'ActiveWorkflowManager.js',\n function: 'removeAllTriggerAndPollerBasedWorkflows'\n}"
Jul 19 09:10:35 2024-07-19T07:10:35.219Z | debug | Shutting down component "AbstractServer.onShutdown()" "{ file: 'Shutdown.service.js', function: 'shutdownComponent' }"
Jul 19 09:10:35 2024-07-19T07:10:35.219Z | debug | Shutting down component "PruningService.shutdown()" "{ file: 'Shutdown.service.js', function: 'shutdownComponent' }"
Jul 19 09:10:35 2024-07-19T07:10:35.221Z | debug | Shutting down component "RedisClientService.disconnectClients()" "{ file: 'Shutdown.service.js', function: 'shutdownComponent' }"
Jul 19 09:10:35 2024-07-19T07:10:35.221Z | debug | [Pruning] Removing soft-deletion and hard-deletion timers "{ file: 'pruning.service.js', function: 'stopPruning' }"
Jul 19 09:10:35 box:taskworker Starting task 1518. Logs are at /home/yellowtent/platformdata/logs/693caa42-bcf9-44c2-8f44-78f8676fe44c/apptask.log
Jul 19 09:10:35 box:apptask run: startTask installationState: pending_restart runState: running
Jul 19 09:10:35 box:tasks update 1518: {"percent":10,"message":"Starting app services"}
Jul 19 09:10:35 box:tasks update 1518: {"percent":20,"message":"Restarting container"}
Jul 19 09:10:38 => Ensure directories
Jul 19 09:10:38 => Loading configuration
Jul 19 09:10:38 => Setting permissions
Jul 19 09:10:38 box:tasks update 1518: {"percent":80,"message":"Configuring reverse proxy"}
Jul 19 09:10:38 box:shell providerMatches execArgs: openssl ["x509","-noout","-subject","-issuer"]
Jul 19 09:10:38 box:reverseproxy providerMatches: subject=CN = *.unly.org domain=*.unly.org issuer=C = US, O = Let's Encrypt, CN = E5 wildcard=true/true prod=true/true issuerMismatch=false wildcardMismatch=false match=true
Jul 19 09:10:38 box:shell getCertificateDates execArgs: openssl ["x509","-startdate","-enddate","-subject","-noout"]
Jul 19 09:10:38 box:reverseproxy expiryDate: subject=CN = *.unly.org notBefore=Jun 13 02:10:26 2024 GMT notAfter=Sep 11 02:10:25 2024 GMT daysLeft=53.791505925925925
Jul 19 09:10:38 box:reverseproxy ensureCertificate: n8n.unly.org acme cert exists and is up to date
Jul 19 09:10:38 box:reverseproxy needsRenewal: false. force: false
Jul 19 09:10:38 box:shell isOscpEnabled execArgs: openssl ["x509","-in","/home/yellowtent/platformdata/nginx/cert/_.unly.org.cert","-noout","-ocsp_uri"]
Jul 19 09:10:38 box:reverseproxy writeAppLocationNginxConfig: writing config for "n8n.unly.org" to /home/yellowtent/platformdata/nginx/applications/693caa42-bcf9-44c2-8f44-78f8676fe44c/n8n.unly.org.conf with options {"sourceDir":"/home/yellowtent/box","vhost":"n8n.unly.org","hasIPv6":true,"ip":"172.18.19.176","port":5678,"endpoint":"app","redirectTo":null,"certFilePath":"/home/yellowtent/platformdata/nginx/cert/_.unly.org.cert","keyFilePath":"/home/yellowtent/platformdata/nginx/cert/_.unly.org.key","robotsTxtQuoted":"\"User-agent: *\\nDisallow: /\"","cspQuoted":null,"hideHeaders":[],"proxyAuth":{"enabled":false,"id":"693caa42-bcf9-44c2-8f44-78f8676fe44c","location":"/"},"upstreamUri":"","ocsp":true,"hstsPreload":false}
Jul 19 09:10:38 box:shell reload /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
Jul 19 09:10:39 => Installing cloudinary@2.2.0 node-fetch@3.1.1 form-data@4.0.0
Jul 19 09:10:39 box:tasks update 1518: {"percent":100,"message":"Done"}
Jul 19 09:10:39 box:taskworker Task took 4.249 seconds
Jul 19 09:10:39 box:tasks setCompleted - 1518: {"result":null,"error":null}
Jul 19 09:10:39 box:tasks update 1518: {"percent":100,"result":null,"error":null}
Jul 19 09:10:40 => Healtheck error: Error: connect ECONNREFUSED 172.18.19.176:5678
Jul 19 09:10:50 => Healtheck error: Error: connect ECONNREFUSED 172.18.19.176:5678
Jul 19 09:10:54 [..................] - idealTree:code: sill idealTree buildDeps [..................] \ idealTree:code: sill idealTree buildDeps [..................] \ idealTree:code: sill idealTree buildDeps [..................] \ idealTree:code: sill idealTree buildDeps [..................] \ idealTree:code: sill idealTree buildDeps [..................] \ idealTree:code: sill idealTree buildDeps [..................] | idealTree:code: sill idealTree buildDeps [..................] | idealTree:code: sill idealTree buildDeps [..................] - idealTree:code: sill idealTree buildDeps [..................] | idealTree:code: sill idealTree buildDeps [#########.........] \ idealTree: timing idealTree Completed in 4204ms [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [#####
Jul 19 09:10:54 #############] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t <30>1 2024-07-19T07:10:54Z unly-n8n 693caa42-bcf9-44c2-8f44-78f8676fe44c 1050 693caa42-bcf9-44c2-8f44-78f8676fe44c - up to date, audited 1657 packages in 15s <30>1 2024-07-19T07:10:54Z unly-n8n 693caa42-bcf9-44c2-8f44-78f8676fe44c 1050 693caa42-bcf9-44c2-8f44-78f8676fe44c - <30>1 2024-07-19T07:10:54Z unly-n8n 693caa42-bcf9-44c2-8f44-78f8676fe44c 1050 693caa42-bcf9-44c2-8f44-78f8676fe44c - 185 packages are looking for funding <30>1 2024-07-19T07:10:54Z unly-n8n 693caa42-bcf9-44c2-8f44-78f8676fe44c 1050 693caa42-bcf9-44c2-8f44-78f8676fe44c - run `npm fund` for details
Jul 19 09:10:54 <30>1 2024-07-19T07:10:54Z unly-n8n 693caa42-bcf9-44c2-8f44-78f8676fe44c 1050 693caa42-bcf9-44c2-8f44-78f8676fe44c - 19 vulnerabilities (9 moderate, 3 high, 7 critical) <30>1 2024-07-19T07:10:54Z unly-n8n 693caa42-bcf9-44c2-8f44-78f8676fe44c 1050 693caa42-bcf9-44c2-8f44-78f8676fe44c - <30>1 2024-07-19T07:10:54Z unly-n8n 693caa42-bcf9-44c2-8f44-78f8676fe44c 1050 693caa42-bcf9-44c2-8f44-78f8676fe44c - To address issues that do not require attention, run: <30>1 2024-07-19T07:10:54Z unly-n8n 693caa42-bcf9-44c2-8f44-78f8676fe44c 1050 693caa42-bcf9-44c2-8f44-78f8676fe44c - npm audit fix <30>1 2024-07-19T07:10:54Z unly-n8n 693caa42-bcf9-44c2-8f44-78f8676fe44c 1050 693caa42-bcf9-44c2-8f44-78f8676fe44c - <30>1 2024-07-19T07:10:54Z unly-n8n 693caa42-bcf9-44c2-8f44-78f8676fe44c 1050 693caa42-bcf9-44c2-8f44-78f8676fe44c - To address all issues (including breaking changes), run: <30>1 2024-07-19T07:10:54Z unly-n8n 693caa42-bcf9-44c2-8f44-78f8676fe44c 1050 693caa42-bcf9-44c2-8f44-78f8676fe44c - npm audit fix --force
Jul 19 09:10:54 <30>1 2024-07-19T07:10:54Z unly-n8n 693caa42-bcf9-44c2-8f44-78f8676fe44c 1050 693caa42-bcf9-44c2-8f44-78f8676fe44c - Run `npm audit` for details.
Jul 19 09:10:55 => Starting N8N
Jul 19 09:10:57 User settings loaded from: /app/data/user/.n8n/config
Jul 19 09:10:57 Loaded config overwrites from /app/data/configs/default.json
Jul 19 09:11:00 => Healtheck error: Error: connect ECONNREFUSED 172.18.19.176:5678
Jul 19 09:11:00 2024-07-19T07:11:00.894Z | info | Initializing n8n process "{ file: 'start.js', function: 'init' }"
Jul 19 09:11:01 2024-07-19T07:11:01.209Z | debug | Lazy Loading credentials and nodes from n8n-nodes-base "{\n credentials: 356,\n nodes: 455,\n file: 'LoggerProxy.js',\n function: 'exports.debug'\n}"
Jul 19 09:11:01 2024-07-19T07:11:01.231Z | debug | Lazy Loading credentials and nodes from @n8n/n8n-nodes-langchain "{\n credentials: 15,\n nodes: 75,\n file: 'LoggerProxy.js',\n function: 'exports.debug'\n}"
Jul 19 09:11:01 2024-07-19T07:11:01.514Z | debug | [Concurrency Control] Service disabled "{ file: 'concurrency-control.service.js', function: 'log' }"
Jul 19 09:11:01 2024-07-19T07:11:01.528Z | info | n8n ready on 0.0.0.0, port 5678 "{ file: 'AbstractServer.js', function: 'init' }"
Jul 19 09:11:01 2024-07-19T07:11:01.594Z | warn | Migrations in progress, please do NOT stop the process. "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
Jul 19 09:11:01 2024-07-19T07:11:01.595Z | info | Starting migration AddConstraintToExecutionMetadata1720101653148 "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"

Here are the full logs:
https://gist.github.com/Vadorequest/7f879c614d8d6a4e40597bdad1c6b9bc

AmbroiseUnly

Restarting the app didn't work.

There was an awaiting Ubuntu update, I restarted the whole server and n8n booted properly this time.
So, it seems to be related to Ubuntu update, which got stuck the whole startup process.
I'm not sure what can be done to avoid this kind of situation.

joseph

Upstream thread - https://community.n8n.io/t/process-hanging-when-starting-n8n-starting-migration-1720101653148/50380/4

@AmbroiseUnly If the issue comes up again, we should investigate in more detail. Thanks for reporting

zonzonzon

I encountered a similar issue. After updating to the latest Cloudron version, I restarted Cloudron as prompted by the system. After that, N8N was unable to connect.

Feb 06 21:09:50 => Healthcheck error: Error: connect ECONNREFUSED 172.18.18.255:5678

joseph

@zonzonzon can you paste more logs? that line only shows that n8n is not responding

zonzonzon

...
Feb 06 21:09:20 box:reverseproxy providerMatches: subject=CN = webapp.abc.com domain=webapp.abc.com issuer=C = US, O = Let's Encrypt, CN = E6 wildcard=false/false prod=true/true issuerMismatch=false wildcardMismatch=false match=true
...
box:reverseproxy writeAppLocationNginxConfig: writing config for "webapp.abc.com" to /home/yellowtent/platformdata/nginx/applications/e83dd39f-2eb7-449a-bbd4-1e9186013b24/webapp.abc.com.conf with options {"sourceDir":"/home/yellowtent/box","vhost":"webapp.abc.com","hasIPv6":true,"ip":"172.18.0.255","port":5678,"endpoint":"app"...}
...
nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "e5.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/app1.abc.com.cert"
nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "e5.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/app2.abc.com.cert"
nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "e6.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/app3.abc.com.cert"
nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "e6.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/webapp.abc.com.cert"
...
=> Healtheck error: Error: connect ECONNREFUSED 172.18.0.255:5678

zonzonzon

The process is as follows:

I saw a notification to restart after updating to the latest Cloudron version and followed that notification, restarting in Cloudron.
N8n was inaccessible and there was a log message as I sent above.
Other apps run normally, only n8n has an error.
N8n status Starting...

nebulon

Is n8n itself starting up within the container according to the app logs? Maybe look at the logs while restarting the n8n app instance to get a better picture.

zonzonzon

I have tried restarting all other apps, and they are working except for n8n, which is having issues. Below is the log after restarting n8n.

Feb 06 21:32:01 box:taskworker Running task of type app
Feb 06 21:32:01 box:apptask run: startTask installationState: pending_restart runState: running 
Feb 06 21:32:01 box:tasks update 4449: {"percent":10,"message":"Starting app services"}
Feb 06 21:32:01 box:tasks update 4449: {"percent":20,"message":"Restarting container"}
Feb 06 21:32:11 => Ensure directories
Feb 06 21:32:11 => Loading configuration
Feb 06 21:32:11 box:tasks update 4449: {"percent":80,"message":"Configuring reverse proxy"}
Feb 06 21:32:11 box:shell reverseproxy: openssl x509 -noout -subject -issuer
Feb 06 21:32:11 box:reverseproxy providerMatches: subject=CN = webapp.domain.com domain=webapp.domain.com issuer=C = US, O = Let's Encrypt, CN = E6 wildcard=false/false prod=true/true issuerMismatch=false wildcardMismatch=false match=true
Feb 06 21:32:11 box:shell reverseproxy: openssl x509 -startdate -enddate -subject -noout  
Feb 06 21:32:11 box:reverseproxy expiryDate: subject=CN = webapp.domain.com notBefore=Jan 9 09:12:15 2025 GMT notAfter=Apr 9 09:12:14 2025 GMT daysLeft=61.777801840277775
Feb 06 21:32:11 box:reverseproxy needsRenewal: false. force: false
Feb 06 21:32:11 box:reverseproxy ensureCertificate: webapp.domain.com acme cert exists and is up to date
Feb 06 21:32:11 box:shell reverseproxy: openssl x509 -in /home/yellowtent/platformdata/nginx/cert/webapp.domain.com.cert -noout -ocsp_uri
Feb 06 21:32:11 box:reverseproxy writeAppLocationNginxConfig: writing config for "webapp.domain.com" to /home/yellowtent/platformdata/nginx/applications/e83dd39f-2eb7-449a-bbd4-1e9186013b24/webapp.domain.com.conf with options {"sourceDir":"/home/yellowtent/box","vhost":"webapp.domain.com","hasIPv6":true,"ip":"172.18.18.255","port":5678,"endpoint":"app","redirectTo":null,"certFilePath":"/home/yellowtent/platformdata/nginx/cert/webapp.domain.com.cert","keyFilePath":"/home/yellowtent/platformdata/nginx/cert/webapp.domain.com.key","robotsTxtQuoted":"\"# Disable search engine indexing\\n\\nUser-agent: *\\nDisallow: /\"","cspQuoted":null,"hideHeaders":[],"proxyAuth":{"enabled":false,"id":"e83dd39f-2eb7-449a-bbd4-1e9186013b24","location":"/"},"upstreamUri":"","ocsp":true,"hstsPreload":false}
Feb 06 21:32:11 box:shell reverseproxy /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
Feb 06 21:32:11 => Setting permissions
Feb 06 21:32:11 box:shell nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "e5.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/app1.domain.com.cert"
Feb 06 21:32:11
Feb 06 21:32:11 box:shell nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "e5.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/app2.domain.com.cert"
Feb 06 21:32:11
Feb 06 21:32:11 box:shell nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "e6.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/app3.domain.com.cert"
Feb 06 21:32:11
Feb 06 21:32:11 box:shell nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "e6.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/webapp.domain.com.cert"
Feb 06 21:32:11
Feb 06 21:32:12 box:tasks update 4449: {"percent":100,"message":"Done"}
Feb 06 21:32:12 box:taskworker Task took 10.636 seconds
Feb 06 21:32:12 box:tasks setCompleted - 4449: {"result":null,"error":null}
Feb 06 21:32:12 box:tasks update 4449: {"percent":100,"result":null,"error":null}
Feb 06 21:32:12 => Installing handlebars@4.7.7 jsonata@2.0.2 marked@4.3.0 bwip-js@3.3.0 ajv-formats@2.1.1 odoo-xmlrpc@1.0.8 firebase-admin@11.10.1
Feb 06 21:32:20 => Healtheck error: Error: connect ECONNREFUSED 172.18.18.255:5678
Feb 06 21:32:30 => Healtheck error: Error: connect ECONNREFUSED 172.18.18.255:5678

zonzonzon

Get notify on cloudron and click reboot

Screenshot 2025-02-06 214317.png

After restarting, n8n cannot connect and there is a log as I sent above

@nebulon

nebulon

You can ignore the reboot required notification if you have already rebooted.

Since the app startup logs basically end with

...
=> Installing handlebars@4.7.7 jsonata@....

This means the n8n server itself hasn't started yet. Does the app automatically restart? If so maybe it runs out of memory.

Otherwise does it ever print anything more than healthecheck failures after that?
Depending on the plugins you have installed, it may just take a very long time or some plugins are just not compatible anymore.

joseph

Good catch @nebulon .

@zonzonzon https://docs.cloudron.io/apps/n8n/#custom-node-modules . Just edit /app/data/env.sh and set EXTRA_NODE_MODULES to empty string and then restart the app, see if that helps.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

Process hanging when starting n8n - Starting migration AddConstraintToExecutionMetadata1720101653148