Solved Cloudron 7.0.1: GitLab stuck after update

[msbt]

Just realized that my gitlab machine updated Cloudron last night (so >10 hours ago) and GitLab didn't come back up. The dashboard said

and I've tried a systemctl restart box which didn't help.

The app just came back online after issuing that cmd, just wanted to let you know that there was an issue, but I don't know what it was

[girish]

@msbt is there any error in the box logs - /home/yellowtent/platformdata/logs/box.log (you can just search for "Error") ? It seems like the platform code did not start up for some reason (until you did the restart).

Apps using postgres are automatically restarted after the update because we updated postgres addon. That's why it's showing a pending task for GitLab.

[msbt]

@girish I think this is it:

...
2021-10-26T06:05:40.818Z box:shell removePostgresql (stderr): null
2021-10-26T06:05:40.818Z box:shell startPostgresql exec: docker run --restart=always -d --name="postgresql"                 --hostname postgresql                 --net cloudron                 --net-al>
2021-10-26T06:05:41.479Z box:shell startPostgresql (stdout): e14886de0e8f7fed78d8696cf094ba54bf5b498d56f1ce85b2218cfa94d87e11

2021-10-26T06:05:41.479Z box:shell startPostgresql (stderr): null
2021-10-26T06:05:41.479Z box:services Waiting for postgresql
2021-10-26T06:05:56.567Z box:database Connection 176 error: Connection lost: The server closed the connection. PROTOCOL_CONNECTION_LOST
2021-10-26T06:05:56.576Z box:database Connection 177 error: Connection lost: The server closed the connection. PROTOCOL_CONNECTION_LOST
2021-10-26T06:05:56.892Z box:cloudron Startup task at index 3 failed: connect ECONNREFUSED 127.0.0.1:3306

2021-10-26T06:10:24.215Z box:locker Lock unreleased platform_start
2021-10-26T06:15:24.215Z box:locker Lock unreleased platform_start
2021-10-26T06:20:24.216Z box:locker Lock unreleased platform_start
2021-10-26T06:25:24.215Z box:locker Lock unreleased platform_start
...

Not sure if this is helpful, but after doing the box restart it was coming up fine

[girish]

@msbt Yup, that's helpful. The issue is that MySQL "restarts" for some reason. This happens immediately after an update, I am not sure why but probably some systemd dependency makes systemd restart MySQL. I tried to fix this a while ago - https://git.cloudron.io/cloudron/box/-/commit/aa71a734b93abfb9ef6a2b16cd1d20653daa4cb0 . Maybe I will fix this by just making our database layer more resilient to mysql going down.

[msbt]

@girish sounds good! that was a contabo machine which isn't the fastest, maybe it's some race condition issue?

[girish]

I have added a retry now for the startup tasks on db failures in 7.0.2