Help with migrating Cloudron to a new server

[davejgreen]

Hi, we are looking to migrate our Cloudron to a new server and want to minimise down time for our staff. We have about 30 users, 130GB of emails, and recent full backups are around 160GB. We use Cloudron for our business website, emails, and internal chat among other things.

I have reviewed the Docs at https://docs.cloudron.io/backups#move-cloudron-to-another-server, but I'm still a bit unsure of a few things:

• Is migration of users and email boxes only possible by using the "looking to restore" button for restoring full Cloudron backups? And is that button only available when you first install Cloudron? We have a second server up and running with a newer version of Ubuntu and Cloudron and were thinking about importing each app individually (after backing it up from the old Cloudron onto a backup site hosted on the new Cloudron), but I'm not sure if there's a way to import the users and email boxes? If not, I guess this approach will not work for us, we'd have to use a third site for a full backup.

• We have a premium monthly subscription on the old Cloudron instance. Do we need to purchase another one for the new Cloudron instance? (We have more than 2 apps.) And then cancel the old one once the migration is done? Or is there a way of using the same subscription throughout the migration?

[james]

Hello @davejgreen

@davejgreen said:

Is migration of users and email boxes only possible by using the "looking to restore" button for restoring full Cloudron backups?

Yes.

@davejgreen said:

And is that button only available when you first install Cloudron?

Yes.

@davejgreen said:

We have a second server up and running with a newer version of Ubuntu and Cloudron and were thinking about importing each app individually (after backing it up from the old Cloudron onto a backup site hosted on the new Cloudron), but I'm not sure if there's a way to import the users and email boxes? If not, I guess this approach will not work for us, we'd have to use a third site for a full backup.

This is not recommended.
The new server should be installed with the exact Cloudron version you are currently using.
Otherwise you wont even be able to restore the system backup.

@davejgreen said:

Do we need to purchase another one for the new Cloudron instance?

No.
Once migrated simply write a mail to support@cloudron.io and inform us and we will switch the licence to the new server.

---

To minimize downtime use the dry-run backup restore option: https://docs.cloudron.io/backups#dry-run-restore
This way, you can restore the whole system, without changing DNS records and can validate each app / step manually.
The old server stays up and running, serving all the apps, mail, chat etc.
After a review of the migration, when ready to switch permanently, go to Domains view and click Sync DNS.

[davejgreen]

Thank you for the clear responses - much appreciated. I am now attempting a dry run of restoring to the new server (a remote VPS with Mythic Beasts) from a backup from the old server (a remote Linode server), but I am hitting some problems.

I have reinstalled Ubuntu 24.04 on our new server (giving me a clean slate), and added it to our organisation's tailscale network and checked I can ssh between all the relevant devices. I have added several entries to the /etc/hosts file on the new server to point all our Cloudron domains to its IPv4 and IPv6 addresses. I have then installed on our new server the same Cloudron version as our existing server (9.0.17).

On the existing server, I have set up a backup site using NFS (tarball) which stores complete backups on an office device that is also on our tailscale network. I have configured this office device to allow both Cloudron servers to see its NFS export. I have then downloaded the config file for the most recent backup.

I have gone to the IPv4 of the new server in a browser, got to the Cloudron install page and clicked the "Looking to restore?" link. I have then uploaded the backup config file and it has populated the text fields for me. I ticked the "Dry Run" box and clicked "Restore". This gave me the error message:
Access denied. Create /mnt/managedbackups/cloudron-restore-validation/nfs-tarballs/snapshot and run "chown yellowtent:yellowtent /mnt/managedbackups/cloudron-restore-validation/nfs-tarballs" on the server

So, on the new server, I did mkdir -p /mnt/managedbackups/cloudron-restore-validation/nfs-tarballs/snapshot. I could then see the backups from our office device at /mnt/managedbackups/cloudron-restore-validation/nfs-tarballs on the new server. (How did they get there if all I did was make a directory?) I then did the chown as instructed and tried clicking "Restore" in the browser page again. This time I got the error:
Failed to unmount existing mount
Is this referring to a mount that was somehow made when I created the directory? Why does it need to unmount it, and why can't it do that? Do I need to unmount something?

Prior to this attempt, I tried the dry run restore by rsyncing the folder of tarballs from a recent backup onto the new server, and then using "Filesystem" as the storage provider (as mentioned in the docs). This successfully restored Cloudron with all users, but it failed to restore all the apps - they were all in an error state. It seemed to be looking for their backup files in the location they would have been at on the old server (a location whose path included a folder named with a big hash). Is this the better approach? And do I have to work out where the backups should be and try to recreate that location on the new server?

Any further guidance would be much appreciated!

[james]

Hello @davejgreen
This is rather complex backup set up which is out of scope of the nominal Cloudron use-case.

@davejgreen said:

I have added several entries to the /etc/hosts file on the new server to point all our Cloudron domains to its IPv4 and IPv6 addresses.

This should be done on your local device so you can access the domain names without the DNS lookup but with a fixed entry overwriting DNS lookup.

@davejgreen said:

So, on the new server, I did mkdir -p /mnt/managedbackups/cloudron-restore-validation/nfs-tarballs/snapshot. I could then see the backups from our office device at /mnt/managedbackups/cloudron-restore-validation/nfs-tarballs on the new server. (How did they get there if all I did was make a directory?) I then did the chown as instructed and tried clicking "Restore" in the browser page again. This time I got the error:
Failed to unmount existing mount
Is this referring to a mount that was somehow made when I created the directory? Why does it need to unmount it, and why can't it do that? Do I need to unmount something?

This is indeed strange.
Did you configure the NFS mount on the new system manually?
Can you run mount -l and see if this NFS mount is there?
Also, if mounted, there might be a systemd service for that mnt-managedbackups.*.

@davejgreen said:

I tried the dry run restore by rsyncing the folder of tarballs from a recent backup onto the new server, and then using "Filesystem" as the storage provider (as mentioned in the docs).

This means that the location was already filled with data?
I would advise resetting the new server again and only attempting the dry run restore from NFS.
The rsync of files to the same location might have caused issues with the mounting and access of files.

[davejgreen]

Hi @james, thanks for such a prompt response.

I'm surprised that trying to migrate from one server to another using an NFS tarball backup on a local device is considered complex - although I am certainly finding it very complex! But I figured that was down to my inexperience in these things. Is it the NFS tarball backup that is unusual?

Our DNS records are manually configured. I think I understand I probably don't need the new server to have the /etc/hosts entries until we do the actual migration. Our thinking was to keep the existing server live for as long as possible, so we could do the migration with these /etc/host entries on the new server, check it is working, then turn off the live server, switch the DNS records, and remove the /etc/hosts entries on the new server. (Though, maybe that's a bad idea and we should just turn the old server off first.) For other parts of my job I still need access to the existing live Cloudron instance, so I hadn't added anything to my local device's /etc/hosts yet - but was planning to do this once the install had completed so that I could check it. I did this with the earlier rsync-Filesystem attempt.

Just to clarify, I did do a complete reinstall of the server before each attempt - installing an Ubuntu image from our provider and erasing all existing data, so I was starting with a clean slate each time. So the backups I rsync over on the earlier attempt were not still there when I tried again today.

The mounting stuff I am a bit confused by (I'm relatively new to the idea of mounting file systems). Doing mount -l on the new server, I can see there is a mount: our-office-device:/export/nfs-cloudron-backups on /mnt/managedbackups/cloudron-restore-validation type nfs4 (with,various,config). I did not create this (or any other mount) manually. Our office device runs on NixOS, and I added a line for the new Cloudron server to its configuration:

  services.nfs.server = {
    enable = true;
    exports = ''
      /export/nfs-cloudron-backups <old-server-tailscale-IP>(rw,sync,no_subtree_check,no_root_squash)
      /export/nfs-cloudron-backups <new-server-tailscale-IP>(rw,sync,no_subtree_check,no_root_squash)
    '';

My understanding was that this would only allow a device with that IP address to mount the folder at /export/nfs-cloudron-backups, but maybe it 'pushed' the mount to the device it could see at that IP address? I'd be surprised if that was the case though, as I haven't set the mount location on the new server anywhere here, and with the old server, I still had to go and create the backup site which included it mounting to this location. The new server has a systemd service mnt-managedbackups-cloudron\x2drestore\x2dvalidation.mount which is active and mounted. I assume that must have been created when I first clicked the "Restore" button and then got the "Access denied. Create..." message? Is that something Cloudron does? Are there some directory permissions I need to check or change? Should I have manually created a mount so the new server could see the office device backup folder before clicking "Restore"?

[james]

Hello @davejgreen

@davejgreen said:

I'm surprised that trying to migrate from one server to another using an NFS tarball backup on a local device is considered complex - although I am certainly finding it very complex! But I figured that was down to my inexperience in these things. Is it the NFS tarball backup that is unusual?

Sorry, what I referred to was the network isolation setup including the NFS on a device in your office.

@davejgreen said:

Our thinking was to keep the existing server live for as long as possible, so we could do the migration with these /etc/host entries on the new server, check it is working, then turn off the live server, switch the DNS records, and remove the /etc/hosts entries on the new server.

Yes. That is the flow.
But not ON the new server the /etc/hosts change is needed, but on your local device so you device resolves the names to your new server for the dry run restore.

@davejgreen said:

Doing mount -l on the new server, I can see there is a mount: our-office-device:/export/nfs-cloudron-backups on /mnt/managedbackups/cloudron-restore-validation type nfs4 (with,various,config).

This was created by the attempted restore, so that is good.
But since there was some folders and files before from your rsync attempt (if I understood this correctly), that might interfere with the mounting and restore process.

@davejgreen said:

but maybe it 'pushed' the mount to the device it could see at that IP address?

That should not be the case unless your NixOS system has access to the new server via. ssh and runs some provisioning style set up for NFS.

@davejgreen said:

Should I have manually created a mount so the new server could see the office device backup folder before clicking "Restore"?

No.
This should be done by Cloudron.

[davejgreen]

Hi @james, thanks again for a quick response.

No Left Over Files
I think you're not understanding what I said about a complete reinstall. I tried the rsync attempt last week, then I erased everything on the new server to try again. Wiped it clean, all files gone, new install of Ubuntu 24.04, no left over files. Then I did this morning's attempt. There were not any files or folders left over from the rsync attempt when I tried the restore today, so that cannot be the reason it didn't work.

DNS
I don't think I understand your point about the /etc/hosts change. I understand that to view the newly restored Cloudron instance in a browser on my local device, I will need to change my local device's /etc/hosts. But I don't need to do this to see the "Restore Cloudron" page, I can just enter the new server's IPv4 in the browser address bar. Then when I start the restore from there, doesn't this happen on the new server? I didn't think my local device was doing anything other than showing me what was happening on the new server through the browser window? Once the restore has finished, then I would need to adjust my local /etc/hosts to view the newly restored Cloudron instead of the old existing one. Are you saying I need to adjust /etc/hosts on my local device before I do the restore? (If so, why?, given that I'm doing a dry run and that we manage all DNS manually.)

Next Steps?
I'm unsure what to try next. I still have the error "Failed to unmount existing mount". I tried doing sudo umount /mnt/managedbackups/cloudron-restore-validation which appeared to work, and then I clicked the restore button again, but got the same error message. The first error message started with "Access denied." which makes me think the problem might be related to file and folder ownership and/or permissions, as I am often confused by these.

How is this mounting meant to work? What should I try next?