n8n / Platform Services not starting since v7.7.1 upgrade

[girish]

@rosano the upgrade shouldn't make the server unreachable. Can you think of a way to ssh into your server? Maybe hard reboot the server? Is this a home server or a VPS? Usually, there is a way to access the console and you can check the boot logs in the console.

[rosano]

@girish seems like hard reboot has the same result. Hetzner has a web console, but I will need to figure out what the password is, let me see…

Also tried stopping some apps I installed recently (cloudron build, docker registry, one custom packaged app) but it seems that process also stalls 'waiting for platform services to start'.

[nebulon]

@rosano is it possible that you had changed the ssh port away from 22 to maybe 202 for example?

[rosano]

@girish got in via the web console. swapon finishes with no output.

[nebulon]

Ok so no swap file exists, probably causing the issue. Can you run /home/yellowtent/box/setup/start/cloudron-resize-fs.sh with sudo via SSH? This should ensure a swap file

[rosano]

@nebulon it says:

Swap file already exists at /apps.swap . Skipping

[rosano]

@nebulon I checked via sudo grep Port /etc/ssh/sshd_config and it says Port 22.

[nebulon]

so something went wrong with that swap file then. Try to remove that file and run the script again, that should rebuild swap

[rosano]

swapoff seemed to not do anything, but sudo rm /apps.swap did and I was able to run the script which seemed to be successful. After reboot via the Cloudron console and also power cycle on Hetzner, I'm still in the same place: 8 apps Running, 1 app stuck on Restarting, 3 apps stuck on Stopping, timeout when connecting via ssh.

[girish]

@rosano under Services, is everything running? Or is mail container stuck in the same way as before?

[rosano]

@girish mail is stopped, clicking restart shows the same error as my initial post. might have to do with argument "NaN" for "-m, --memory" flag: invalid size: 'NaN'?

[girish]

@rosano ok, I think NaN is some cloudron bug that we have to debug further. But before we can debug that, we have to get SSH access somehow. Did you figure out the root password ? It seems you ran sudo grep Port /etc/ssh/sshd_config , so I guess you had root password, somehow?

[rosano]

@girish I was able to reset via Hetzner console yes, but I ran everything through their 'web console' not SSH. Would you able to use this root password from your end?

[girish]

@rosano yes, please can you drop a mail to support@cloudron.io , I can debug why this is happenning now.

[rosano]

@girish sent, number 18901.

[rosano]

still working via support, but just want to note that ssh doesn't time out when replacing the IP address with the cloudron domain:

ssh root@my.example.com

[rosano]

@rosano I might have misread #Port 22 as Port 22 as it seems to have been commented out (not sure how/why). Uncommenting and rebooting enables ssh login via ip address again. Computers are hard

[rosano]

@nebulon seems also that the swap file was commented out in /etc/fstab (again not sure why/how) so it doesn't persist across restarts, and I guess the script you posted wouldn't affect this.

# /apps.swap  none  swap  sw  0 0

After uncommenting and reboot now, there seems to be a swap file via swapon:

/apps.swap file 3.7G 1.8M   -2

And all apps are running / stopped as expected, nothing hanging thanks for your patience and help .

I wonder what about this could have to do with the latest Cloudron update as I didn't touch anything in the last days and everything seemed to be fine.

[girish]

There is a bug in Cloudron that it parses swap info incorrectly when there are no swaps in the system. This is fixed in https://git.cloudron.io/cloudron/box/-/commit/b66e77a2d8930381c95153b041d7c3edd331364a