n8n / Platform Services not starting since v7.7.1 upgrade

[rosano]

I'm using an older version of n8n (0.234.0) and it seems to be the only app that gets stuck on 'Restarting'. It stays on "Restarting - Waiting for Platform Services to start. See the Services view", and when I look at /services it seems like mail is the only one with a red light (stopped); I think I use Mailgun so not sure if it was ever running; clicking the Restart button on that service flashes this error (tokens replaced with XXXXX):

Cloudron Error
configureMail errored with code 125 message Command failed: docker run --restart=always -d --name=mail --net cloudron --net-alias mail --log-driver syslog --log-opt syslog-address=udp://127.0.0.1:2514 --log-opt syslog-format=rfc5424 --log-opt tag=mail -m NaN --memory-swap 536870912 --dns 172.18.0.1 --dns-search=. -e CLOUDRON_MAIL_TOKEN=XXXXX -e CLOUDRON_RELAY_TOKEN=XXXXX -e LOGLEVEL=info -v /home/yellowtent/boxdata/mail:/app/data -v /home/yellowtent/platformdata/addons/mail:/etc/mail:ro --label isCloudronManaged=true --read-only -v /run -v /tmp registry.docker.com/cloudron/mail:3.12.1@sha256:f539bea6c7360d3c0aa604323847172359593f109b304bb2d2c5152ca56be05c invalid argument "NaN" for "-m, --memory" flag: invalid size: 'NaN' See 'docker run --help'. 

The light stays yellow after the error disappears, but when refreshing the page it goes back to red.

Logs for n8n show:

Mar 21 01:04:49 => Ensure directories
Mar 21 01:04:49 => Loading configuration
Mar 21 01:04:54 => Setting permissions
Mar 21 01:04:56 => Starting N8N
Mar 21 01:05:12 Loading config overwrites [ '/app/data/configs/default.json' ]

I also restarted Cloudron but it results in the same state.

[rosano]

@nebulon seems also that the swap file was commented out in /etc/fstab (again not sure why/how) so it doesn't persist across restarts, and I guess the script you posted wouldn't affect this.

# /apps.swap  none  swap  sw  0 0

After uncommenting and reboot now, there seems to be a swap file via swapon:

/apps.swap file 3.7G 1.8M   -2

And all apps are running / stopped as expected, nothing hanging thanks for your patience and help .

I wonder what about this could have to do with the latest Cloudron update as I didn't touch anything in the last days and everything seemed to be fine.

[girish]

@rosano for some reason the memory is computed as NaN on your server! not good. Do you have swap disabled on your server by any chance?

[rosano]

@girish I think not, but maybe? What can I type in to verify / enable?

[girish]

@rosano if you run swapon on the server, it should tell you if it has any swap at all.

[rosano]

@girish ssh times out:

ssh: connect to host XXXXX port 22: Operation timed out

Does this also need platform services to start first? I feel a bit stuck here. Can I downgrade somehow?

[girish]

@rosano The error now is different. So, you cannot ssh into the server? This is more at the VPS/server level. Maybe you moved SSH to a different port or have some block list ?

[rosano]

@girish I did it a few days ago, not sure what changed other than the upgrade? I have another server for CapRover with the same provider (Hetzner) and I can ssh there without issue.

[girish]

@rosano the upgrade shouldn't make the server unreachable. Can you think of a way to ssh into your server? Maybe hard reboot the server? Is this a home server or a VPS? Usually, there is a way to access the console and you can check the boot logs in the console.

[rosano]

@girish seems like hard reboot has the same result. Hetzner has a web console, but I will need to figure out what the password is, let me see…

Also tried stopping some apps I installed recently (cloudron build, docker registry, one custom packaged app) but it seems that process also stalls 'waiting for platform services to start'.

[nebulon]

@rosano is it possible that you had changed the ssh port away from 22 to maybe 202 for example?

[rosano]

@girish got in via the web console. swapon finishes with no output.

[nebulon]

Ok so no swap file exists, probably causing the issue. Can you run /home/yellowtent/box/setup/start/cloudron-resize-fs.sh with sudo via SSH? This should ensure a swap file

[rosano]

@nebulon it says:

Swap file already exists at /apps.swap . Skipping

[rosano]

@nebulon I checked via sudo grep Port /etc/ssh/sshd_config and it says Port 22.

[nebulon]

so something went wrong with that swap file then. Try to remove that file and run the script again, that should rebuild swap

[rosano]

swapoff seemed to not do anything, but sudo rm /apps.swap did and I was able to run the script which seemed to be successful. After reboot via the Cloudron console and also power cycle on Hetzner, I'm still in the same place: 8 apps Running, 1 app stuck on Restarting, 3 apps stuck on Stopping, timeout when connecting via ssh.

[girish]

@rosano under Services, is everything running? Or is mail container stuck in the same way as before?

[rosano]

@girish mail is stopped, clicking restart shows the same error as my initial post. might have to do with argument "NaN" for "-m, --memory" flag: invalid size: 'NaN'?

[girish]

@rosano ok, I think NaN is some cloudron bug that we have to debug further. But before we can debug that, we have to get SSH access somehow. Did you figure out the root password ? It seems you ran sudo grep Port /etc/ssh/sshd_config , so I guess you had root password, somehow?

[rosano]

@girish I was able to reset via Hetzner console yes, but I ran everything through their 'web console' not SSH. Would you able to use this root password from your end?