Apps Not Starting After Upgrade to 7.4.1 - but postgresql instead of mysql

[holm]

Hi,

after upgrading to 7.4.1 there is the "Starting services, this can take a while" Message and all Database based Apps stucks.

systemctl restart box

brings in the log:

2023-04-17T15:45:30.750Z box:services pipeRequestToFile: connected with status code 404
2023-04-17T15:45:30.750Z box:services exportDatabase: Error exporting postgresql of app b1f26524-5ea6-43a0-bcd7-da77b20fb136. BoxError: Unexpected response code or HTTP error when piping http://172.18.30.2:3000/databases/dbb1f265245ea643a0bcd7da77b20fb136/backup?access_token=VERYLONGTOKEN to /home/yellowtent/appsdata/b1f26524-5ea6-43a0-bcd7-da77b20fb136/postgresqldump: status 404
    at ClientRequest.<anonymous> (/home/yellowtent/box/src/services.js:1224:60)
    at ClientRequest.emit (node:events:513:28)
    at HTTPParser.parserOnIncomingClient (node:_http_client:693:27)
    at HTTPParser.parserOnHeadersComplete (node:_http_common:128:17)
    at Socket.socketOnData (node:_http_client:534:22)
    at Socket.emit (node:events:513:28)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at Socket.Readable.push (node:internal/streams/readable:228:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23) {
  reason: 'Addons Error',
  details: {}
}
2023-04-17T15:45:30.751Z box:platform Failed to start services. retry=false (attempt 0): Unexpected response code or HTTP error when piping http://172.18.30.2:3000/databases/dbb1f265245ea643a0bcd7da77b20fb136/backup?access_token=VERYLONGTOKEN to /home/yellowtent/appsdata/b1f26524-5ea6-43a0-bcd7-da77b20fb136/postgresqldump: status 404
2023-04-17T15:45:30.751Z box:cloudron Startup task at index 2 failed: Unexpected response code or HTTP error when piping http://172.18.30.2:3000/databases/dbb1f265245ea643a0bcd7da77b20fb136/backup?access_token=VERYLONGTOKEN to /home/yellowtent/appsdata/b1f26524-5ea6-43a0-bcd7-da77b20fb136/postgresqldump: status 404 BoxError: Unexpected response code or HTTP error when piping http://172.18.30.2:3000/databases/dbb1f265245ea643a0bcd7da77b20fb136/backup?access_token=VERYLONGTOKEN to /home/yellowtent/appsdata/b1f26524-5ea6-43a0-bcd7-da77b20fb136/postgresqldump: status 404
    at ClientRequest.<anonymous> (/home/yellowtent/box/src/services.js:1224:60)
    at ClientRequest.emit (node:events:513:28)
    at HTTPParser.parserOnIncomingClient (node:_http_client:693:27)
    at HTTPParser.parserOnHeadersComplete (node:_http_common:128:17)
    at Socket.socketOnData (node:_http_client:534:22)
    at Socket.emit (node:events:513:28)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at Socket.Readable.push (node:internal/streams/readable:228:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23)

thats all i can see what ist curious. what to do next?

i think, that i have done all steps from https://forum.cloudron.io/topic/8169/starting-services-this-can-take-a-while-on-v7-3-4-update and https://forum.cloudron.io/topic/9042/apps-not-starting-after-upgrade-to-7-4-1 but t seams for me, that the problem is in postgresql / postgresql startup tasks.

[girish]

@holm I replied to you on support. There is a bug in the update process. It hits a race when the update process encounters an app which has no database (like gitlab above).

[holm]

@holm

the named app b1f26524-5ea6-43a0-bcd7-da77b20fb136 is a disabled gitlab instance.

[girish]

@holm I replied to you on support. There is a bug in the update process. It hits a race when the update process encounters an app which has no database (like gitlab above).

[holm]

@girish

thank you for the fast help! the system is up and running again!

[potemkin_ai]

@girish thanks for sharing the solution and I understand things has to keep moving, but this update was quite a disaster...

Could you possibly introduce a configuration flag to automatically update the platform on minor updates, but instead send an e-mail requesting manual update with a major versions?

I got my company paralyzed for a few hours... =(

[holm]

@potemkin_ai
full ack!

its like the thing a few years ago, when the etherpad major upgrade kills all my pluginfull etherpad instances.

it's a tricky situation. to apply no updates is bad. to apply all updates is risky and more or less in schedule not synced with my worktimes or holidays or other ressources.

all or nothing or something between. hard to decide...

[potemkin_ai]

@holm you shouldn't

That's why SW developers have major and minor version separations - the minor is usually updated automatically, while the major is only handled manually, with all precautions required

[avatar1024]

I am hitting the same issue. Update to 7.4.1 from 7.4.0, the system was running fine and had a notification asking to reboot and after reboot, database services are stuck and apps are not responding.

How did you solve it??

[avatar1024]

@potemkin_ai said in Apps Not Starting After Upgrade to 7.4.1 - but postgresql instead of mysql:

the minor is usually updated automatically, while the major is only handled manually, with all precautions required

a different update approach for the platform along those lines is what I proposed here and here, but I'm not sure it's getting much traction.

[avatar1024]

@avatar1024 said in Apps Not Starting After Upgrade to 7.4.1 - but postgresql instead of mysql:

I am hitting the same issue. Update to 7.4.1 from 7.4.0, the system was running fine and had a notification asking to reboot and after reboot, database services are stuck and apps are not responding.

Seems like I just wasn't patient enough...it's taken a while but it seems to now be working.

[potemkin_ai]

@avatar1024 hopefully now it gets a good kick

What you say, @girish ?