How would you migrate a Cloudron with ~1TB of data?

[msbt]

Do we have some suggestions or best practices for a Cloudron this size? Now that v6 with Ubuntu 20.04 support is out, I'm looking forward to ditch my last 16.04 (dedicated) server and migrate it to a different provider (from Hetzner to Contabo, because the cloud servers are too expensive with that space usage).

The biggest part is a Nextcloud instance with ~800GB (which isn't part of the current backup due to its size). The biggest tar.gz I had so far was ~50GB and it worked fine, but what about 800GB, that wouldn't fly as a tar backup, right?

If I change to rsync, wouldn't the backup and restore take much longer? I mean, I calculated with 50MB/sec and 1TB of data, that would be >5 hours, but what about rsync, I reckon that would be even slower, right? Has anyone ever done a migration like this?

Cheers, M

[murgero]

@msbt Can you temporarily upgrade both the old and new server to gigabit internet? Then transfer the data using something like torrent for max speed?

[robi]

Is there a way to exclude an app (NC) from migration? Then migrate all other apps and users, and keep the old NC up until data gets slowly migrated.

Can you set the new cloudron as a backup target for the old one so the restore is local & faster?

.tgz is still the way to go, but it would be really nice if we had incremental tar backups.

There may be a way to use a storage gateway device that can front all those requests via a mount (file or network); caching all recent data requests allowing older data to slowly sync.

[msbt]

@murgero they're both gbit links, but I doubt that the process will max out the lines, that doesn't even happen on local hetzner storagebox backups.

@robi good thinking with the backup targets. I figured I could get a second box at contabo, use it for remote backups from the hetzner server and restore from there, this should be faster than provider -> provider. I'll do some testing and keep this updated.

[mehdi]

@msbt Honestly, I would just not use the Cloudron backup/restore mecanism for such an app. Last time I had to change servers for ~2gb of data, I just rsynced the data manually from one server to the other, then re-created the app on the new server, and put the data back there.
On the other hand, I did not have other apps that I did want to restore, so ..

[girish]

@msbt You can disable automatic backups of nextcloud. Then do a full box backup and then you can restore all the apps on the other server using the box backup. Nextcloud will get installed but won't have any data (as expected) because there is no backup for it. So, far so good.

For nextcloud migration, one first needs a "snapshot" of the app on the old server i.e the database dump etc. For this, there is a CLI command called cloudron export --snapshot. This will simply create the database dumps etc. You can then wholesale copy /home/yellowtent/appsdata/{appid} from your old server into the new server using rsync/scp or whatever. Once copied, on the new server, you do cloudron import --in-place. This will then restore nextcloud from the dump files etc (thus, the in place).

The catch is that I implemented cloudron export only after I read your post So, it will only be in the next release. But you can always apply https://git.cloudron.io/cloudron/box/-/commit/78752fde7a402821b40eeb75091089470933b23f on the old server. cloudron import has been there for a long time, it's a hidden thing that we use to import large backups into Cloudron. So, I guess you can wait for 6.0.1, which should be out shortly since we have to release it soonish as it had some regressions.