SOLVED Cloudron Dashboard Offline, Cannot start manually

[nj]

My Cloudron dashboard is offline, and returns an app not responding message. It seems the Cloudron app wasn't running. I tried to restart it manually, but the script got stuck. Here's the log, please assist.
PS: All other apps are running fine; only the dashboard is inaccessible.

Now all apps show an INVALID CERTIFICATE (Self Signed) error after a reboot;

root@myserver:~# /home/yellowtent/box/setup/start.sh
2021-07-09T05:17:02 ==> start: Cloudron Start
media:x:500:
2021-07-09T05:17:02 ==> start: Configuring docker
Synchronizing state of apparmor.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable apparmor
Error response from daemon: network with name cloudron already exists
2021-07-09T05:17:03 ==> start: Ensuring directories
2021-07-09T05:17:03 ==> start: Configuring journald
2021-07-09T05:17:03 ==> start: Setting up unbound
2021-07-09T05:17:04 ==> start: Adding systemd services
Synchronizing state of unbound.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable unbound
Synchronizing state of cron.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable cron
2021-07-09T05:17:07 ==> start: Configuring sudoers
2021-07-09T05:17:07 ==> start: Configuring collectd
2021-07-09T05:17:07 ==> start: Configuring logrotate
2021-07-09T05:17:07 ==> start: Adding motd message for admins
2021-07-09T05:17:07 ==> start: Configuring nginx
2021-07-09T05:17:08 ==> start: Starting mysql
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
Warning: Since password will be sent to server in plain text, use ssl connection to ensure password safety.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
2021-07-09T05:17:08 ==> start: Migrating data
cp: -r not specified; omitting directory '/home/yellowtent/boxdata/certs/tmp'
[ERROR] Error: Command failed: cp /home/yellowtent/boxdata/certs/* /home/yellowtent/platformdata/nginx/cert
cp: -r not specified; omitting directory '/home/yellowtent/boxdata/certs/tmp'

    at checkExecSyncError (child_process.js:616:11)
    at Object.execSync (child_process.js:652:15)
    at /home/yellowtent/box/migrations/20210505223829-blobs-migrate-certs.js:43:27
    at wrapper (/home/yellowtent/box/node_modules/async/dist/async.js:268:20)
    at replenish (/home/yellowtent/box/node_modules/async/dist/async.js:435:29)
    at iterateeCallback (/home/yellowtent/box/node_modules/async/dist/async.js:424:21)
    at /home/yellowtent/box/node_modules/async/dist/async.js:321:20
    at /home/yellowtent/box/node_modules/async/dist/async.js:2955:19
    at wrapper (/home/yellowtent/box/node_modules/async/dist/async.js:268:20)
    at replenish (/home/yellowtent/box/node_modules/async/dist/async.js:435:29)
    at iterateeCallback (/home/yellowtent/box/node_modules/async/dist/async.js:424:21)
    at /home/yellowtent/box/node_modules/async/dist/async.js:321:20
    at /home/yellowtent/box/node_modules/async/dist/async.js:2953:17
    at tryCatcher (/home/yellowtent/box/node_modules/bluebird/js/release/util.js:16:23)
    at Promise.successAdapter [as _fulfillmentHandler0] (/home/yellowtent/box/node_modules/bluebird/js/release/nodeify.js:23:30)
    at Promise._settlePromise (/home/yellowtent/box/node_modules/bluebird/js/release/promise.js:601:21)
2021-07-09T05:17:08 ==> start: DB migration failed

[nj]

It seems the latest update 6.3.4 completely broke my system. This is the log that shows that the update script was broken. Currently, no user can log in (says password incorrect, even if it's correct because I use a password manager)

None of the installed apps have had a valid certificate; all apps have Self-Signed certificates. I cannot do a password reset because the email isn't working.

[nj]

I cannot restore from backup because automatic backups were failing for the last few days, (no apps were backed up) so I will lose the last three days' worth of data. Surprisingly, all apps were working till now, but when I restarted the system the apps lost their certificates.

[nj]

I am 100% sure the culprit is this file: 20210505223829-blobs-migrate-certs.js because looking at the logs, it failed to run the cp command to copy certs from /home/yellowtent/boxdata/certs/* to /home/yellowtent/platformdata/nginx/cert and failed because of the missing -r flag.

[nj]

Finally, I fixed the issue by modifying the migration file. Adding -r to the cp command, and then manually restarting Cloudron /home/yellowtent/box/setup/start.sh fixed the issue for me. I'm surprised that no other Cloudron instances had this issue.

File: /home/yellowtent/box/migrations/20210505223829-blobs-migrate-certs.js Line 43:

child_process.execSync(`cp -r ${OLD_CERTS_DIR}/* ${NEW_CERTS_DIR}`); // this way we copy the non-migrated ones like .host, .user etc as well

This thread is a copy of the emails I sent to the support statff. I posted here so it might help someone else in this situation.

[nebulon]

@nj great that you figured this by your own then. Strange that you had a tmp folder there causing the issue, I guess you had created that in the past. Generally it is not recommended to create custom folder structures in /home/yellowtent/boxdata/ or /home/yellowtent/platformdata/ since we cannot really test such cases for updates.

Is there any specific use-case to have that folder there, which we should consider?

[girish]

I am also interested in knowing why the -r fixed it. Do you have custom nginx configs?

[nj]

@girish I don't remember creating any folder myself, so I can't tell how it got created there. None of my other Cloudron instances had this issue so it might have happened somehow. I don't have any custom nginx config.

Maybe removing the tmp folder could have solved the issue without the -r flag. The error cp: -r not specified; omitting directory '/home/yellowtent/boxdata/certs/tmp' meant it couldn't copy the tmp folder because -r flag was missing. Also, I added -r because the error message hinted me to do so and it fixed the issue.

[nebulon]

@nj yes with the folder in place there adding the -r to solve this makes sense, however the initial issue is that this folder should not be there in the first place. As the name already suggests, I guess this was just some intermediate manual action to stash certs. Essentially if you don't actively use files in that folder then just delete it to solve this for future releases.