my.cloudron.tld is not accessible after Cloudron update to 8.0.2

martinkbs

Hi guys,

This morning I was updating my Cloudron instance to the latest version (8.0.2) and in the reboot process it got stuck and now I cannot access my Cloudron dashboard.

I have tried restarting the server, and it still won't boot. I have accessed via SSH and I see all the containers stopped, but they do not start.

Thank you very much in advance.

girish

What do you see in the logs /home/yellowtent/platformdata/logs/box.log ?

martinkbs

Hi @girish,

These are the last lines from the last healthmonitor:

2024-07-29T11:30:53.167Z box:apphealthmonitor app health: 21 running / 5 stopped / 0 unresponsive
Finished with result: success
Main processes terminated with: code=exited/status=0
Service runtime: 6min 14.757s
CPU time consumed: 27.475s
box-task-14938 stopped
Service box-task-14938 finished with exit code 0
2024-07-29T11:30:53.283Z box:tasks startTask: 14938 completed with code 0
2024-07-29T11:30:53.285Z box:tasks setCompleted - 14938: {"error":{"message":"Task 14938 stopped","code":"stopped"}}
2024-07-29T11:30:53.285Z box:tasks update 14938: {"percent":100,"error":{"message":"Task 14938 stopped","code":"stopped"}}
2024-07-29T11:30:53.481Z box:locker Released : box_update
2024-07-29T11:30:53.482Z box:updater Update failed with error. { message: 'Task 14938 stopped', code: 'stopped' }
2024-07-29T11:30:53.483Z box:tasks startTask: 14938 done. error: { message: 'Task 14938 stopped', code: 'stopped' }
[object Object]
2024-07-29T11:31:03.507Z box:apphealthmonitor app health: 21 running / 5 stopped / 0 unresponsive
2024-07-29T11:31:10.737Z box:locker Acquired : box_update
2024-07-29T11:31:10.933Z box:tasks startTask - starting task 14939 with options {"timeout":72000000,"nice":15,"memoryLimit":1024}. logs at /home/yellowtent/platformdata/logs/tasks/14939.log
2024-07-29T11:31:10.934Z box:shell startTask /usr/bin/sudo -S -E /home/yellowtent/box/src/scripts/starttask.sh 14939 /home/yellowtent/platformdata/logs/tasks/14939.log 15 1024
Running as unit: box-task-14939.service
2024-07-29T11:31:14.236Z box:apphealthmonitor app health: 21 running / 5 stopped / 0 unresponsive
2024-07-29T11:31:22.563Z box:apphealthmonitor app health: 21 running / 5 stopped / 0 unresponsive
2024-07-29T11:31:30.906Z box:apphealthmonitor app health: 21 running / 5 stopped / 0 unresponsive
2024-07-29T11:31:36.621Z box:box Received SIGTERM. Shutting down.
2024-07-29T11:31:36.622Z box:platform uninitializing platform
2024-07-29T11:31:36.623Z box:shell startTask: /usr/bin/sudo -S -E /home/yellowtent/box/src/scripts/starttask.sh 14939 /home/yellowtent/platformdata/logs/tasks/14939.log 15 1024 errored BoxError: startTask exited with code null signal SIGTERM
    at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:118:19)
    at ChildProcess.emit (node:events:518:28)
    at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
  reason: 'Shell Error',
  details: {},
  code: null,
  signal: 'SIGTERM'
}
2024-07-29T11:31:36.623Z box:tasks startTask: 14939 completed with code null
2024-07-29T11:31:36.624Z box:platform onDeactivated: stopping post activation services
2024-07-29T11:31:36.625Z box:tasks stopAllTasks: stopping all tasks
2024-07-29T11:31:36.625Z box:shell stopTask /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
2024-07-29T11:31:36.631Z box:tasks setCompleted - 14939: {"error":{"message":"Task 14939 crashed with code null","code":"crashed"}}
2024-07-29T11:31:36.631Z box:tasks update 14939: {"percent":100,"error":{"message":"Task 14939 crashed with code null","code":"crashed"}}
2024-07-29T11:31:36.634Z box:locker Released : box_update
2024-07-29T11:31:36.634Z box:updater Update failed with error. { message: 'Task 14939 crashed with code null', code: 'crashed' }
2024-07-29T11:31:36.634Z box:tasks startTask: 14939 done. error: { message: 'Task 14939 crashed with code null', code: 'crashed' }
All tasks stopped

girish

Look like the update failed. Can you check /home/yellowtent/platformdata/logs/tasks/14939.log ?

martinkbs

Yes @girish, this is the content of /home/yellowtent/platformdata/logs/tasks/14939.log:

2024-07-29T11:31:16.050Z box:taskworker Starting task 14939. Logs are at /home/yellowtent/platformdata/logs/tasks/14939.log
2024-07-29T11:31:16.255Z box:tasks update 14939: {"percent":1,"message":"Checking disk space"}
2024-07-29T11:31:16.257Z box:shell file execArgs: df ["-B1","--output=source,fstype,size,used,avail,pcent,target","/"]
2024-07-29T11:31:16.346Z box:tasks update 14939: {"percent":5,"message":"Downloading and verifying release"}
2024-07-29T11:31:16.348Z box:shell cleanupOldArtifacts exec: rm -rf /tmp/box-*
2024-07-29T11:31:20.263Z box:shell cleanupOldArtifacts: rm -rf /tmp/box-* errored Error: Command failed: rm -rf /tmp/box-*
rm: cannot remove '/tmp/box-2971389862/node_modules/ssh2/lib/protocol/crypto/build/config.gypi': Permission denied

    at genericNodeError (node:internal/errors:984:15)
    at wrappedFn (node:internal/errors:538:14)
    at ChildProcess.exithandler (node:child_process:422:12)
    at ChildProcess.emit (node:events:518:28)
    at maybeClose (node:internal/child_process:1105:16)
    at ChildProcess._handle.onexit (node:internal/child_process:305:5) {
  code: 1,
  killed: false,
  signal: null,
  cmd: 'rm -rf /tmp/box-*'
}
2024-07-29T11:31:20.266Z box:updater BoxError: cleanupOldArtifacts errored with code 1 message Command failed: rm -rf /tmp/box-*
rm: cannot remove '/tmp/box-2971389862/node_modules/ssh2/lib/protocol/crypto/build/config.gypi': Permission denied

    at /home/yellowtent/box/src/shell.js:71:23
    at ChildProcess.exithandler (node:child_process:430:5)
    at ChildProcess.emit (node:events:518:28)
    at maybeClose (node:internal/child_process:1105:16)
    at ChildProcess._handle.onexit (node:internal/child_process:305:5)
2024-07-29T11:31:20.266Z box:updater downloadUrl: downloading https://releases.cloudron.io/versions.json to /home/yellowtent/platformdata/update/versions.json
2024-07-29T11:31:20.266Z box:shell downloadUrl execArgs: curl ["-s","--fail","https://releases.cloudron.io/versions.json","-o","/home/yellowtent/platformdata/update/versions.json"]
2024-07-29T11:31:21.766Z box:updater downloadUrl: done
2024-07-29T11:31:21.766Z box:updater downloadUrl: downloading https://releases.cloudron.io/versions.json.sig to /home/yellowtent/platformdata/update/versions.json.sig
2024-07-29T11:31:21.766Z box:shell downloadUrl execArgs: curl ["-s","--fail","https://releases.cloudron.io/versions.json.sig","-o","/home/yellowtent/platformdata/update/versions.json.sig"]
2024-07-29T11:31:22.568Z box:updater downloadUrl: done
2024-07-29T11:31:22.569Z box:updater gpgVerify: /usr/bin/gpg --status-fd 1 --no-default-keyring --keyring /home/yellowtent/box/src/releases.gpg --verify /home/yellowtent/platformdata/update/versions.json.sig /home/yellowtent/platformdata/update/versions.json
2024-07-29T11:31:22.569Z box:shell gpgVerify execArgs: /usr/bin/gpg ["--status-fd","1","--no-default-keyring","--keyring","/home/yellowtent/box/src/releases.gpg","--verify","/home/yellowtent/platformdata/update/versions.json.sig","/home/yellowtent/platformdata/update/versions.json"]
2024-07-29T11:31:22.671Z box:updater downloadUrl: downloading https://releases.cloudron.io/box-8d9043e590-8d9043e590-8.0.2.tar.gz to /home/yellowtent/platformdata/update/box.tar.gz
2024-07-29T11:31:22.672Z box:shell downloadUrl execArgs: curl ["-s","--fail","https://releases.cloudron.io/box-8d9043e590-8d9043e590-8.0.2.tar.gz","-o","/home/yellowtent/platformdata/update/box.tar.gz"]

nebulon

So for some reason the user does not have permission to delete that folder from /tmp. This is probably a side-effect of some other problem. Can you maybe send a mail to support@cloudron.io and enable remote SSH support for us https://docs.cloudron.io/support/#ssh-keys to take a look at this?

martinkbs

Hi @nebulon,

I think @girish has already resolved the issue, because I can access my Cloudron Dashboard and all the applications are working correctly. In addition, it is already updated to version 8.0.3.

Anyway, I just enabled remote SSH support so you can take a look and know what could have happened.

Please keep me informed of what you find and when I can disable remote SSH support.

girish

Looks like some tmpfs corruption caused files to have incorrect permissions. The reboot cleared up tmp and the previous state was not there. Just redownloading 8.0.1 and updating again made it work.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

my.cloudron.tld is not accessible after Cloudron update to 8.0.2