MongoDB does not start
-
Sorry I have a similar problem on a cloudron 7.4.3 on ubuntu 20, a pending task is blocking everything (backup, update, uninstall, ...) on my secondary cloudron instance:
- I have a blue notice saying: Platform status / Starting services, this can take a while
- I have the only app active in: Configuring (Queued) - Queued
- the cancel task redX button is not responding
- Crash Recovery buttons are not clickable
https://my.cloudron/api/v1/tasks/12960?access_token=XXXXXXmytoken
{ "id": "12960", "type": "app", "percent": 1, "message": "Waiting for Cloudron to initialize", "error": { "message": "Task was stopped because the server was restarted or crashed", "code": "crashed" }, "active": false, "pending": true, "creationTime": "2023-12-17T21:39:50.000Z", "result": null, "ts": "2023-12-17T21:39:50.000Z", "success": false },
- I've tried a systemctl check as BrutalBirdie
root@my:~# systemctl status box-task-12960.service Unit box-task-12960.service could not be found.
Cannot GET /api/v1/tasks/12960/stop
But no way to kill the pending task. Do you have any suggestions?
tail -f /home/yellowtent/platformdata/logs/box.log Box POST /api/v1/tasks/12967/stop 409 Conflict task is not active 91.551 ms - 61 2024-01-04T11:30:41.308Z box:locker Lock unreleased platform_start
Thank you and best regards.
-
@josephcosta In one terminal, keep
tail -f /home/yellowtent/platformdata/logs/box.log
running and then in another terminal can you runsystemctl restart box
?The box code is not starting up in your case (different from the original report). It's getting stuck when starting one of the addons. You will see where it's getting stuck in the logs, if you do above.
-
Hello girish and thank you for your availability.
Please find the fulltail -f /home/yellowtent/platformdata/logs/box.log
here:https://private.josephcosta.org/?a3c283d20a04b958#7svM5Dns9GPGBwgMkK1DBQjh471Sa2Q3AcFoyV36RbJy
There seems to be a problem on mongodb, but on services the mongodb row has a greenlight (while graphite and ftp are red and become green clicking on restart for both rows).
-
@nebulon hello!
here is the mongodb log:
https://private.josephcosta.org/?3243caee470a6dea#E6yHbRRNpBkgWBy2gsfh6HKsXL7mLS7gt2efBQDxCMKx -
This one is a full log after systemctl restart box:
https://private.josephcosta.org/?cce43f2e454b0896#9zFEx4jitpAqz16MSFQtSy3wuqRUYqbXPiH9iLvzN8ky
Thank you for your assistance @nebulon -
This looks actually fine, which may then indicate that the system is just too slow for the 20 retries it waits for the service. Maybe you can increase the retry attempt count at https://git.cloudron.io/cloudron/box/-/blob/master/src/services.js?ref_type=heads#L581 (you will find that file in
/home/yellowtent/box
and after edit runsystemctl restart box
) -
Hello, unfortunately this does not work, the services mongodb circle is green and still the log says:
2024-01-05T15:13:25.978Z box:services Attempt 8 failed. Will retry: Network error waiting for mongodb: Timeout of 5000ms exceeded.
Graphite and ftp were red so I restarted those services using UI and the log said:
2024-01-05T15:15:59.638Z box:shell restartcollectd spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh collectd 2024-01-05T15:16:00.988Z box:services Attempt 9 failed. Will retry: Network error waiting for mongodb: Timeout of 5000ms exceeded
All is green but still
2024-01-05T15:18:35.999Z box:services Attempt 10 failed. Will retry: Network error waiting for mongodb: Timeout of 5000ms exceeded 2024-01-05T15:19:44.790Z box:locker Lock unreleased platform_start
-
@josephcosta does it not continue after attempt 10? Per the code, it should attempt 20 times.
You can also reach out on support@cloudron.io and we can debug this further.
p.s.: i moved this to a separate thread.
-
-
@girish thank you as always, I reached you out on support@cloudron.io.
Best regards -
@josephcosta the issue here is just the timeout for services. This was already fixed a while ago in https://git.cloudron.io/cloudron/box/-/commit/7511339656a0866d733d6577e5cab8fa05e20791 (part of v7.5.0). I applied the patch locally and it's working now.
2024-01-09T12:12:59.557Z box:cloudron Startup task at index 2 failed: Network error waiting for mongodb: Timeout of 5000ms exceeded BoxError: Network error waiting for mongodb: Timeout of 5000ms exceeded at /home/yellowtent/box/src/services.js:585:33 at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async promiseRetry (/home/yellowtent/box/src/promise-retry.js:17:20) at async waitForContainer (/home/yellowtent/box/src/services.js:580:5) at async startMongodb (/home/yellowtent/box/src/services.js:1519:9) at async Object.startServices (/home/yellowtent/box/src/services.js:803:9) at async Object.start (/home/yellowtent/box/src/platform.js:68:13) at async onActivated (/home/yellowtent/box/src/cloudron.js:82:5) at async Array.<anonymous> (/home/yellowtent/box/src/cloudron.js:134:9)
-
-
Thank you very much @girish, you rock!