MongoDB does not start

[josephcosta]

Sorry I have a similar problem on a cloudron 7.4.3 on ubuntu 20, a pending task is blocking everything (backup, update, uninstall, ...) on my secondary cloudron instance:

• I have a blue notice saying: Platform status / Starting services, this can take a while
• I have the only app active in: Configuring (Queued) - Queued
  • the cancel task redX button is not responding
  • Crash Recovery buttons are not clickable

https://my.cloudron/api/v1/tasks/12960?access_token=XXXXXXmytoken

    {
      "id": "12960",
      "type": "app",
      "percent": 1,
      "message": "Waiting for Cloudron to initialize",
      "error": {
        "message": "Task was stopped because the server was restarted or crashed",
        "code": "crashed"
      },
      "active": false,
      "pending": true,
      "creationTime": "2023-12-17T21:39:50.000Z",
      "result": null,
      "ts": "2023-12-17T21:39:50.000Z",
      "success": false
    },

• I've tried a systemctl check as BrutalBirdie

root@my:~# systemctl status box-task-12960.service
Unit box-task-12960.service could not be found.

• https://mycloudron/api/v1/tasks/12960/stop?access_token=XXXXXXmytoken

Cannot GET /api/v1/tasks/12960/stop

But no way to kill the pending task. Do you have any suggestions?

tail -f /home/yellowtent/platformdata/logs/box.log
Box POST /api/v1/tasks/12967/stop 409 Conflict task is not active 91.551 ms - 61
2024-01-04T11:30:41.308Z box:locker Lock unreleased platform_start

Thank you and best regards.

[girish]

@josephcosta In one terminal, keep tail -f /home/yellowtent/platformdata/logs/box.log running and then in another terminal can you run systemctl restart box ?

The box code is not starting up in your case (different from the original report). It's getting stuck when starting one of the addons. You will see where it's getting stuck in the logs, if you do above.

[josephcosta]

Hello girish and thank you for your availability.
Please find the full tail -f /home/yellowtent/platformdata/logs/box.log here:

https://private.josephcosta.org/?a3c283d20a04b958#7svM5Dns9GPGBwgMkK1DBQjh471Sa2Q3AcFoyV36RbJy

There seems to be a problem on mongodb, but on services the mongodb row has a greenlight (while graphite and ftp are red and become green clicking on restart for both rows).

[nebulon]

@josephcosta can you also then check /home/yellowtent/platformdata/logs/mongodb/app.log to see why mongodb isn't coming up?

[josephcosta]

@nebulon hello!
here is the mongodb log:
https://private.josephcosta.org/?3243caee470a6dea#E6yHbRRNpBkgWBy2gsfh6HKsXL7mLS7gt2efBQDxCMKx

[josephcosta]

This one is a full log after systemctl restart box:
https://private.josephcosta.org/?cce43f2e454b0896#9zFEx4jitpAqz16MSFQtSy3wuqRUYqbXPiH9iLvzN8ky
Thank you for your assistance @nebulon

[nebulon]

This looks actually fine, which may then indicate that the system is just too slow for the 20 retries it waits for the service. Maybe you can increase the retry attempt count at https://git.cloudron.io/cloudron/box/-/blob/master/src/services.js?ref_type=heads#L581 (you will find that file in /home/yellowtent/box and after edit run systemctl restart box)

[josephcosta]

Hello, unfortunately this does not work, the services mongodb circle is green and still the log says:

2024-01-05T15:13:25.978Z box:services Attempt 8 failed. Will retry: Network error waiting for mongodb: Timeout of 5000ms exceeded.

Graphite and ftp were red so I restarted those services using UI and the log said:

2024-01-05T15:15:59.638Z box:shell restartcollectd spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh collectd
2024-01-05T15:16:00.988Z box:services Attempt 9 failed. Will retry: Network error waiting for mongodb: Timeout of 5000ms exceeded

All is green but still

2024-01-05T15:18:35.999Z box:services Attempt 10 failed. Will retry: Network error waiting for mongodb: Timeout of 5000ms exceeded
2024-01-05T15:19:44.790Z box:locker Lock unreleased platform_start

[girish]

@josephcosta does it not continue after attempt 10? Per the code, it should attempt 20 times.

You can also reach out on support@cloudron.io and we can debug this further.

p.s.: i moved this to a separate thread.

[josephcosta]

@girish thank you as always, I reached you out on support@cloudron.io.
Best regards

[girish]

@josephcosta the issue here is just the timeout for services. This was already fixed a while ago in https://git.cloudron.io/cloudron/box/-/commit/7511339656a0866d733d6577e5cab8fa05e20791 (part of v7.5.0). I applied the patch locally and it's working now.

2024-01-09T12:12:59.557Z box:cloudron Startup task at index 2 failed: Network error waiting for mongodb: Timeout of 5000ms exceeded BoxError: Network error waiting for mongodb: Timeout of 5000ms exceeded
    at /home/yellowtent/box/src/services.js:585:33
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async promiseRetry (/home/yellowtent/box/src/promise-retry.js:17:20)
    at async waitForContainer (/home/yellowtent/box/src/services.js:580:5)
    at async startMongodb (/home/yellowtent/box/src/services.js:1519:9)
    at async Object.startServices (/home/yellowtent/box/src/services.js:803:9)
    at async Object.start (/home/yellowtent/box/src/platform.js:68:13)
    at async onActivated (/home/yellowtent/box/src/cloudron.js:82:5)
    at async Array.<anonymous> (/home/yellowtent/box/src/cloudron.js:134:9)

[josephcosta]

Thank you very much @girish, you rock!