New Problem: Recovery after disk full, all services runing, still unreachable dashboard re: ....the DNS record of...is set to this server's IP but cloudron has no apps configured for this domain

[ZeZaung]

This is a follow-up thread based on our own resolution of issues with nginx, certs, and configs, which we removed based on a recommendation found with a very similar error encountered by other cloudron users. That thread can be found here: https://forum.cloudron.io/topic/7722/nginx-problems-running-after-diskfull-recovery-walkthrough/3

But now we have a new error: unreachable dashboard, this is what we get:

You are seeing this page because the DNS record of my.redacted.domain is set to this server's IP but Cloudron has no app configured for this domain.

We dont' know where to go now. All our services are in a running state, but we fear something is misconfigured and do not know how to debug any further without potentially causing even more issues.

[nebulon]

Do you see anything in the box logs at /home/yellowtent/platformdata/logs/box.log on the server? You may also try to run the /home/yellowtent/box/setup/start.sh as root to ensure required service are running.

[ZeZaung]

@nebulon

Hi. It seems the matter has solved itself. We did run start.sh and still weren't able to access dashboard (everythign was down except all services appeared active & running) Waited approx 18 hours and now it's all back up again. I guess the certs got replaced eventually, not sure.

Here's some snippets of the start.sh action:

2022-09-26T09:00:00.295Z box:apps Error initiating autoupdate of ce6e40e0-b6d5-48c6-beb1-5ba775d0137f. Not allowed in error state
2022-09-26T09:00:00.305Z box:apps Error initiating autoupdate of d028e2b5-bcd1-40f8-9b12-1139588eb328. Not allowed in error state
2022-09-26T09:00:10.085Z box:apphealthmonitor app health: 21 alive / 25 dead.
2022-09-26T10:30:00.534Z box:disks checkDiskSpace: disk space checked. out of space: no
2022-09-26T12:00:00.015Z box:janitor Cleaning up expired tokens
2022-09-26T12:00:00.019Z box:janitor Cleaning up docker volumes
2022-09-26T12:00:00.022Z box:janitor Cleaned up 0 expired tokens
2022-09-26T12:00:00.030Z box:tasks startTask - starting task 5607 with options {}. logs at /home/yellowtent/platformdata/logs/tasks/5607.log
2022-09-26T12:00:00.031Z box:shell startTask spawn: /usr/bin/sudo -S -E /home/yellowtent/box/src/scripts/starttask.sh 5607 /home/yellowtent/platformdata/logs/tasks/5607.log 0 400
2022-09-26T12:00:00.151Z box:janitor cleanupTmpVolume ["/26b359fd-4fc2-4e40-99f4-d93f78915e8f"]
2022-09-26T12:00:00.207Z box:apphealthmonitor app health: 21 alive / 25 dead.
2022-09-26T12:00:00.312Z box:shell startTask (stdout): Running as unit: box-task-5607.service
2022-09-26T12:00:01.310Z box:shell startTask (stdout): 10633 (process group ID) old priority 0, new priority 0
2022-09-26T12:00:01.660Z box:janitor cleanupTmpVolume ["/8533d5c4-a3c1-40e5-9546-bf3bc9295001"]
2022-09-26T12:00:01.754Z box:janitor cleanupTmpVolume ["/43633466-4a91-4e35-a10e-f80fcde30ff1"]
2022-09-26T12:00:01.852Z box:janitor cleanupTmpVolume ["/c68e6d0e-dba3-4bef-8542-c2fd13a46072"]
2022-09-26T12:00:01.969Z box:janitor cleanupTmpVolume ["/5195f5c9-4e62-4f3c-a74b-1908ef057d99"]
2022-09-26T12:00:02.072Z box:janitor cleanupTmpVolume ["/b8932afc-5c61-4b32-ac43-c3433d5c8d9e"]
2022-09-26T12:00:02.074Z box:janitor BoxError: Failed to exec container: (HTTP code 409) container stopped/paused - Container dd659c1b56c55000b45e4598178af05a7f5b8d4d1dbb0d10171922d30cf113ff is restarting, wait until the container is running
    at cleanupTmpVolume (/home/yellowtent/box/src/janitor.js:34:22)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
2022-09-26T12:00:02.075Z box:janitor cleanupTmpVolume ["/6614e9bf-7cde-4a4d-81cb-83f24e19d5b7"]
2022-09-26T12:00:02.182Z box:janitor cleanupTmpVolume ["/dc855c65-ad9e-4c58-ace7-37735a857a0d"]
2022-09-26T12:00:02.275Z box:janitor cleanupTmpVolume ["/redis-dc855c65-ad9e-4c58-ace7-37735a857a0d"]
2022-09-26T12:00:02.356Z box:janitor cleanupTmpVolume ["/9e486f53-493d-4c33-861e-5ea3bcf0120e"]
2022-09-26T12:00:02.435Z box:janitor cleanupTmpVolume ["/d49ecfd4-5798-49d7-a92f-1a60a3eedff7"]
2022-09-26T12:00:02.523Z box:janitor cleanupTmpVolume ["/71a74797-240c-4224-8431-dcf0ca8b31d8"]
2022-09-26T12:00:02.598Z box:janitor cleanupTmpVolume ["/52226e9a-5e77-4e8d-9c8f-d8f3aca985dc"]
2022-09-26T12:00:02.687Z box:janitor cleanupTmpVolume ["/5faaa596-1239-4ea8-9e2f-78a310227e12"]
2022-09-26T12:00:02.763Z box:janitor cleanupTmpVolume ["/redis-52226e9a-5e77-4e8d-9c8f-d8f3aca985dc"]
2022-09-26T12:00:02.851Z box:janitor cleanupTmpVolume ["/56fcfa77-bab6-490e-81ba-59ad7a446f4d"]
2022-09-26T12:00:02.927Z box:janitor cleanupTmpVolume ["/eaab0051-f08d-4443-b7cc-e2a8dd0f6df0"]
2022-09-26T12:00:03.004Z box:janitor cleanupTmpVolume ["/3da8624a-96aa-4a04-bb3f-dd1602ea3133"]
2022-09-26T12:00:03.083Z box:janitor cleanupTmpVolume ["/7fef7038-0f05-44cc-85ae-5ab5bfe2cedd"]
2022-09-26T12:00:03.168Z box:janitor cleanupTmpVolume ["/1f4a4797-f865-41cd-b33c-253c35b5014a"]
2022-09-26T12:00:03.248Z box:janitor cleanupTmpVolume ["/798c7dab-463b-4f0b-a31b-aa4138ff7ba8"]
2022-09-26T12:00:03.332Z box:janitor cleanupTmpVolume ["/redis-71a74797-240c-4224-8431-dcf0ca8b31d8"]
2022-09-26T12:00:03.424Z box:janitor cleanupTmpVolume ["/mail"]
2022-09-26T12:00:03.504Z box:janitor cleanupTmpVolume ["/redis-5faaa596-1239-4ea8-9e2f-78a310227e12"]
2022-09-26T12:00:03.584Z box:janitor cleanupTmpVolume ["/0d41c0b8-79f4-4f4a-9918-3210ae6a5f6e"]
2022-09-26T12:00:03.660Z box:janitor cleanupTmpVolume ["/c72baee9-ec9b-4bbb-8a9b-a4e72e416817"]
2022-09-26T12:00:03.743Z box:janitor cleanupTmpVolume ["/11fcd0ab-ff4c-4dac-a58f-4c564fa59036"]
2022-09-26T12:00:03.828Z box:janitor cleanupTmpVolume ["/175049fb-c70d-4cff-b724-54cbf9561762"]
2022-09-26T12:00:03.912Z box:janitor cleanupTmpVolume ["/c960e9ec-b551-424a-a1a2-8bd04a1f311f"]
2022-09-26T12:00:03.989Z box:janitor cleanupTmpVolume ["/920daad0-7f7c-4610-acc3-348fdfe5ef03"]
2022-09-26T12:00:03.991Z box:janitor BoxError: Failed to exec container: (HTTP code 409) container stopped/paused - Container 61200336d6ea05fbfe7ff3060712aa28cdb3b41af622aa36919d54cd4f582709 is restarting, wait until the container is running
    at cleanupTmpVolume (/home/yellowtent/box/src/janitor.js:34:22)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
2022-09-26T12:00:03.991Z box:janitor cleanupTmpVolume ["/6e7ad12f-d27b-4978-877f-13867e41340e"]
2022-09-26T12:00:03.992Z box:janitor BoxError: Failed to exec container: (HTTP code 409) container stopped/paused - Container 2b9c86c1ec8f0a098294b1b3a0b381d7470835376ea7ec640cafb0eea03dd59a is restarting, wait until the container is running
    at cleanupTmpVolume (/home/yellowtent/box/src/janitor.js:34:22)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
2022-09-26T12:00:03.992Z box:janitor cleanupTmpVolume ["/ce717ac0-91cf-467d-b457-1dc110003537"]
2022-09-26T12:00:04.088Z box:janitor cleanupTmpVolume ["/sftp"]
2022-09-26T12:00:04.169Z box:janitor cleanupTmpVolume ["/redis-d49ecfd4-5798-49d7-a92f-1a60a3eedff7"]
2022-09-26T12:00:04.262Z box:janitor cleanupTmpVolume ["/redis-d028e2b5-bcd1-40f8-9b12-1139588eb328"]
2022-09-26T12:00:04.351Z box:janitor cleanupTmpVolume ["/redis-c107e741-82e7-4670-b970-1ffded0e4557"]
2022-09-26T12:00:04.434Z box:janitor cleanupTmpVolume ["/redis-aedf9b10-4fa8-4c8f-80d8-28faacd1cd86"]
2022-09-26T12:00:04.520Z box:janitor cleanupTmpVolume ["/redis-9e486f53-493d-4c33-861e-5ea3bcf0120e"]
2022-09-26T12:00:04.602Z box:janitor cleanupTmpVolume ["/redis-70560788-b7f7-42b5-b722-ad557d1821e4"]
2022-09-26T12:00:04.683Z box:janitor cleanupTmpVolume ["/redis-4f4daf11-09e2-4396-bb6b-84748c017e89"]
2022-09-26T12:00:04.767Z box:janitor cleanupTmpVolume ["/redis-35f25c44-7707-43b8-ab2a-611327d5ca20"]
2022-09-26T12:00:04.850Z box:janitor cleanupTmpVolume ["/redis-2df3d2af-3983-4403-aa08-a02f68e39e8c"]
2022-09-26T12:00:04.926Z box:janitor cleanupTmpVolume ["/redis-20a23213-ed2e-45d0-a83d-e6856f128c20"]
2022-09-26T12:00:05.005Z box:janitor cleanupTmpVolume ["/redis-1f4a4797-f865-41cd-b33c-253c35b5014a"]
2022-09-26T12:00:05.088Z box:janitor cleanupTmpVolume ["/redis-1319b8cd-e506-412b-a317-38c94b88a4e7"]
2022-09-26T12:00:05.174Z box:janitor cleanupTmpVolume ["/redis-06579a1c-cd5f-48aa-852c-db2a147eca16"]
2022-09-26T12:00:05.263Z box:janitor cleanupTmpVolume ["/mongodb"]
2022-09-26T12:00:05.352Z box:janitor cleanupTmpVolume ["/postgresql"]
2022-09-26T12:00:05.426Z box:janitor cleanupTmpVolume ["/0b3d023e-be95-435f-98d9-956726387db3"]
2022-09-26T12:00:05.502Z box:janitor cleanupTmpVolume ["/d028e2b5-bcd1-40f8-9b12-1139588eb328"]
2022-09-26T12:00:05.587Z box:janitor cleanupTmpVolume ["/c107e741-82e7-4670-b970-1ffded0e4557"]
2022-09-26T12:00:05.662Z box:janitor cleanupTmpVolume ["/4f4daf11-09e2-4396-bb6b-84748c017e89"]
2022-09-26T12:00:05.753Z box:janitor cleanupTmpVolume ["/2df3d2af-3983-4403-aa08-a02f68e39e8c"]
2022-09-26T12:00:05.847Z box:janitor cleanupTmpVolume ["/20a23213-ed2e-45d0-a83d-e6856f128c20"]
2022-09-26T12:00:05.923Z box:janitor cleanupTmpVolume ["/06579a1c-cd5f-48aa-852c-db2a147eca16"]
2022-09-26T12:00:06.000Z box:janitor cleanupTmpVolume ["/graphite"]
2022-09-26T12:00:06.084Z box:janitor cleanupTmpVolume ["/mysql"]
2022-09-26T12:00:06.167Z box:janitor cleanupTmpVolume ["/turn"]
2022-09-26T12:00:06.257Z box:janitor cleanupTmpVolume ["/gracious_jackson"]
2022-09-26T12:00:10.104Z box:apphealthmonitor app health: 21 alive / 25 dead.
2022-09-26T12:00:20.111Z box:apphealthmonitor app health: 21 alive / 25 dead.
2022-09-26T12:00:30.092Z box:apphealthmonitor app health: 21 alive / 25 dead.
2022-09-26T12:00:40.086Z box:apphealthmonitor app health: 21 alive / 25 dead.
2022-09-26T12:00:50.086Z box:apphealthmonitor app health: 21 alive / 25 dead.
2022-09-26T12:01:00.127Z box:apphealthmonitor app health: 21 alive / 25 dead.
2022-09-26T12:01:00.548Z box:shell startTask (stdout): Finished with result: success
Main processes terminated with: code=exited/status=0
Service runtime: 1min 240ms
2022-09-26T12:01:00.548Z box:shell startTask (stdout): Service box-task-5607 finished with exit code 0
2022-09-26T12:01:00.552Z box:tasks startTask: 5607 completed with code 0 and signal 0
2022-09-26T12:01:00.553Z box:tasks startTask: 5607 done. error: null
2022-09-26T12:01:10.091Z box:apphealthmonitor app health: 21 alive / 25 dead.
2022-09-26T12:01:20.080Z box:apphealthmonitor app health: 21 alive / 25 dead.
2022-09-26T12:01:30.185Z box:apphealthmonitor app health: 21 alive / 25 dead.
2022-09-26T12:01:40.142Z box:apphealthmonitor app health: 21 alive / 25 dead.
2022-09-26T12:01:50.434Z box:apphealthmonitor app health: 21 alive / 25 dead.
2022-09-26T12:02:00.097Z box:apphealthmonitor app health: 21 alive / 25 dead.
2022-09-26T12:02:10.108Z box:apphealthmonitor app health: 21 alive / 25 dead.
2022-09-26T12:02:13.538Z box:locker Acquired : apptask
2022-09-26T12:02:13.539Z box:scheduler suspendJobs: 47c707f0-e8c2-41f6-b66f-4c62c43ef165
2022-09-26T12:02:13.539Z box:tasks startTask - starting task 5608 with options {"timeout":72000000,"nice":15,"memoryLimit":400,"logFile":"/home/yellowtent/platformdata/logs/47c707f0-e8c2-41f6-b66f-4c62c43ef165/apptask.log"}. logs at /home/yellowtent/platformdata/logs/47c707f0-e8c2-41f6-b66f-4c62c43ef165/apptask.log
2022-09-26T12:02:13.539Z box:shell startTask spawn: /usr/bin/sudo -S -E /home/yellowtent/box/src/scripts/starttask.sh 5608 /home/yellowtent/platformdata/logs/47c707f0-e8c2-41f6-b66f-4c62c43ef165/apptask.log 15 400
2022-09-26T12:02:13.697Z box:shell startTask (stdout): Running as unit: box-task-5608.service
2022-09-26T12:02:14.682Z box:shell startTask (stdout): 13398 (process group ID) old priority 0, new priority 15
2022-09-26T12:02:17.299Z box:shell startTask (stdout): Finished with result: exit-code
Main processes terminated with: code=exited/status=50
Service runtime: 3.618s
2022-09-26T12:02:17.300Z box:shell startTask code: 50, signal: null
2022-09-26T12:02:17.301Z box:tasks startTask: 5608 completed with code 50 and signal null
2022-09-26T12:02:17.303Z box:apps scheduleTask: task 5608 of 47c707f0-e8c2-41f6-b66f-4c62c43ef165 completed
2022-09-26T12:02:17.303Z box:locker Released : apptask
2022-09-26T12:02:17.304Z box:scheduler resumeJobs: 47c707f0-e8c2-41f6-b66f-4c62c43ef165
2022-09-26T12:02:17.304Z box:tasks startTask: 5608 done. error: [object Object]
2022-09-26T12:02:20.112Z box:apphealthmonitor app health: 21 alive / 25 dead.

[nebulon]

@ZeZaung maybe due to the disk issue, some certs ended up being corrupted and nginx then failed to restart, which got solved once new certs were fetched.

[ZeZaung]

@nebulon

That sounds like exactly what happened, what kind of monitoring solution would you recommend for cloudron (interested in space and RAM monitoring)? We've been hesitant about using any sort of apt get for fear of interfering with cloudron operation