Restore process tested - what happens if ownership of backup files was wrong?

[warg]

Hello,

I just tested the backup and restore procedure of Cloudron v7.5.0 due to @scooke recent forum thread.

For that I downloaded the full backup in /var/backups of this date, took a copy of the backup configuration, reinstalled the VPS with an Ubuntu 22 LTS image and executed the Cloudron setup there + uploaded the backup files. IPs + domain were the same before and afterwards (no change). That went fine and I was able to execute the restore button. Then the dashboard was down for 2-3 minutes and I noticed I forgot the warning in the orange box under https://docs.cloudron.io/backups/#restore-cloudron. Then I changed the permissions of the directory + files to yellowtent:yellowtent. Dashboard was available again and the apps are running. I have three questions about this:

1. Does the restore process just try to restore in an endless loop until the files are readable? Otherwise I'm not sure why the restore worked although the permissions were wrong.
2. In the Cloudron log, I see some errors now. Are they related to the wrong backup procedure (no permissions) or is this a bug in the restore process?
   2.1 Aug 16 18:26:08 box:shell support (stderr): grep: /home/cloudron-support/.ssh/authorized_keys: No such file or directory
   2.2.

"2023-08-16T16:14:54.440Z box:shell startTask (stderr): Running as unit: box-task-908.service
2023-08-16T16:15:00.041Z box:scheduler sync: adding jobs of 8856cefd-14a0-4142-a62a-e3edea36dbae (nextcloud.<removed>)
2023-08-16T16:15:00.047Z box:scheduler sync: adding jobs of 8856cefd-14a0-4142-a62a-e3edea36dbae (nextcloud.<removed>)
2023-08-16T16:15:00.111Z box:scheduler BoxError: (HTTP code 400) unexpected - invalid mount config for type "bind": bind source path does not exist: /home/yellowtent/appsdata/8856cefd-14a0-4142-a62a-e3edea36dbae/data
    at Object.createSubcontainer (/home/yellowtent/box/src/docker.js:413:28)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-08-16T16:15:00.112Z box:scheduler BoxError: (HTTP code 400) unexpected - invalid mount config for type "bind": bind source path does not exist: /home/yellowtent/appsdata/8856cefd-14a0-4142-a62a-e3edea36dbae/data
    at Object.createSubcontainer (/home/yellowtent/box/src/docker.js:413:28)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-08-16T16:15:00.120Z box:apphealthmonitor app health: 1 running / 0 stopped / 3 unresponsive
2023-08-16T16:15:00.122Z box:apphealthmonitor app health: 1 running / 0 stopped / 3 unresponsive
2023-08-16T16:15:02.258Z box:oidc Creating OpenID storage adapter for Session
2023-08-16T16:15:02.258Z box:oidc load: model Session based on /home/yellowtent/platformdata/oidc/Session.json.
2023-08-16T16:15:02.259Z box:oidc load: failed to read /home/yellowtent/platformdata/oidc/Session.json, start with new one.
2023-08-16T16:15:02.262Z box:oidc Creating OpenID storage adapter for Client
2023-08-16T16:15:02.281Z box:oidc Creating OpenID storage adapter for Interaction
2023-08-16T16:15:02.281Z box:oidc load: model Interaction based on /home/yellowtent/platformdata/oidc/Interaction.json.
2023-08-16T16:15:02.281Z box:oidc load: failed to read /home/yellowtent/platformdata/oidc/Interaction.json, start with new one.
2023-08-16T16:15:02.282Z box:oidc save: model Interaction to /home/yellowtent/platformdata/oidc/Interaction.json.
2023-08-16T16:15:02.289Z box:oidc save: model Session to /home/yellowtent/platformdata/oidc/Session.json.
2023-08-16T16:15:04.333Z box:shell startTask (stderr): Finished with result: success
Main processes terminated with: code=exited/status=0

2.3 2023-08-16T16:17:00.065Z box:scheduler BoxError: (HTTP code 409) unexpected - Conflict. The container name "/8856cefd-14a0-4142-a62a-e3edea36dbae-housekeeping" is already in use by container "5a97c1b276294c53f398bf13435e10423074ddba6c1d77653f874fb526f515a0". You have to remove (or rename) that container to be able > at Object.createSubcontainer (/home/yellowtent/box/src/docker.js:412:62) at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
3. In the backups view, I see the old backups that I didn't upload nor save. They are lost in data nirvana now (as expected from my side). Will the non-existing backups getting deleted from the list in the next task/job run?

Should this be investigated (I see some other errors as well with similar wording as e. g. no. 2.3 or where it complains that mongodb, postgresql, sftp were not up yet although it was expected)? Beside of that, it was a straightforward process. Just a tiny UI improvement would be great: A way to download the full backup + backup config with 1 click (no need for sftp to download the /var/backups/2023-08-16[...]/* files).

Best Regards,

[michaelpope]

Hey @warg ,
For 2.1, Not with the Cloudron Team, but (2) sounds like the ssh keys that cloudron uses for support aren't there. I'm guessing you authorized them supporting you in the past, but on the new box, have not authorized them, or something like that.
For 2.2, did you add a secondary volume in your NextCloud?

[warg]

Hey @michaelpope,

thanks for your reply! Appreciated!

@michaelpope said in Restore process tested - what happens if ownership of backup files was wrong?:

For 2.1, Not with the Cloudron Team, but (2) sounds like the ssh keys that cloudron uses for support aren't there. I'm guessing you authorized them supporting you in the past, but on the new box, have not authorized them, or something like that.

Yes, I think so too. It looks like this error is thrown everytime I call the support page of Cloudron so should be fine as it is like you suspected.

For 2.2: I never added a volume to the previous or current instance. When I check the page, it is just empty (no entries for the table at /#/volumes).

Maybe it is related to the OS updates I applied on the previous instance (trying to fix a turn service issue) but from the error messages it sounds unlikely as it's just Cloudron-specific messages without direct OS/docker context.

Just noticed that in the backup overview there's aso a red error: Task was stopped because the server was restarted or crashed. Could be from the reboot after Cloudron install?

[girish]

@warg said in Restore process tested - what happens if ownership of backup files was wrong?:

Does the restore process just try to restore in an endless loop until the files are readable? Otherwise I'm not sure why the restore worked although the permissions were wrong.

Not in endless loop, but the download is tried a few times. The backup code has a generic retry for download requests to accomodate network errors. This backup code/logic does a retry even if the backups are just local. I guess that's why it started working. But after some retries, it will fail and go back to the restore UI.

2.1 - this is only an error message from the shell script. This can be ignored. When you visit Support page, it tries to check if they keys are there or not. I guess grep says that the file is not there.

2.2, 2.3 - these can be ignored too. Those errors are from the scheduler trying to create sidecar containers for cron jobs. When restoring, the app containers are created slowly (3 at a time), so the scheduler is saying they don't exist. Eventually scheduler will pick up the app containers
when they get created.

[girish]

@warg said in Restore process tested - what happens if ownership of backup files was wrong?:

3. In the backups view, I see the old backups that I didn't upload nor save. They are lost in data nirvana now (as expected from my side). Will the non-existing backups getting deleted from the list in the next task/job run?

Yes, if you cleanup backups, those should go away.

[girish]

@warg said in Restore process tested - what happens if ownership of backup files was wrong?:

Just a tiny UI improvement would be great: A way to download the full backup + backup config with 1 click (no need for sftp to download the /var/backups/2023-08-16[...]/* files).

Thanks for testing The above is planned at some point - to allow uploading a backup from the UI.

One related issue is we are not sure how one can download or upload a "rsync" backup. The download backup UI currently only works for tgz backup.

[warg]

Backups are fine now. it created a backup in the night and it succeeded. Only the backup from yesterday is broken according to the log and I still see the old (non existing) backup logs in the dropdown. So just a tiny bug and the backup system itself works fine.

Thanks for confirming this and explaining how it works in the background!