Help with migrating Cloudron to a new server

[davejgreen]

Hi @james, thanks again for a quick response.

No Left Over Files
I think you're not understanding what I said about a complete reinstall. I tried the rsync attempt last week, then I erased everything on the new server to try again. Wiped it clean, all files gone, new install of Ubuntu 24.04, no left over files. Then I did this morning's attempt. There were not any files or folders left over from the rsync attempt when I tried the restore today, so that cannot be the reason it didn't work.

DNS
I don't think I understand your point about the /etc/hosts change. I understand that to view the newly restored Cloudron instance in a browser on my local device, I will need to change my local device's /etc/hosts. But I don't need to do this to see the "Restore Cloudron" page, I can just enter the new server's IPv4 in the browser address bar. Then when I start the restore from there, doesn't this happen on the new server? I didn't think my local device was doing anything other than showing me what was happening on the new server through the browser window? Once the restore has finished, then I would need to adjust my local /etc/hosts to view the newly restored Cloudron instead of the old existing one. Are you saying I need to adjust /etc/hosts on my local device before I do the restore? (If so, why?, given that I'm doing a dry run and that we manage all DNS manually.)

Next Steps?
I'm unsure what to try next. I still have the error "Failed to unmount existing mount". I tried doing sudo umount /mnt/managedbackups/cloudron-restore-validation which appeared to work, and then I clicked the restore button again, but got the same error message. The first error message started with "Access denied." which makes me think the problem might be related to file and folder ownership and/or permissions, as I am often confused by these.

How is this mounting meant to work? What should I try next?

[james]

Hello @davejgreen

@davejgreen said:

I tried the rsync attempt last week, then I erased everything on the new server to try again.

Ah! Thanks for the clarification. I indeed did not understand you correctly.

@davejgreen said:

Then when I start the restore from there, doesn't this happen on the new server?

https://docs.cloudron.io/backups/#dry-run-restore

Dry run skips DNS updates. The new server won't be publicly accessible - you access it using /etc/hosts entries on your local machine.

Dry run will not change any DNS records.
This also means, as soon as you hit the restore with dry run, when a redirect happens to your restored domain, you DNS will resolve to your production server.
This is why you need to update your local device /etc/hosts to prevent this.
If one does not do that, you can have a mixed view of the dashboard of cached content, the old server serving stuff and the new system serving stuff.

@davejgreen said:

The first error message started with "Access denied." which makes me think the problem might be related to file and folder ownership and/or permissions, as I am often confused by these.

This could be an issue.
With this partial mount that is failing, can you review the access to that mounted NFS?

[davejgreen]

When I click Restore, I get the message: "Access denied. Create /mnt/managedbackups/cloudron-restore-validation/nfs-tarballs/snapshot and run "chown yellowtent:yellowtent /mnt/managedbackups/cloudron-restore-validation/nfs-tarballs" on the server".

Then, the /mnt/managedbackups/cloudron-restore-validation folder looks like it is mounted correctly. It has permissions "drwxrwxrwx root root", and I can see the "nfs-tarballs" folder from our office device inside it.

The "nfs-tarballs" folder, and all its contents, have permissions "drwxr-xr-x djg djg" (djg is my user, which happens to have the UID 1000, on the office device there is no user assigned to 1000 and these folders show "drwxr-xr-x 1000 1000"). The "nfs-tarballs" folder contains several folders named with date-times, as well as one called "snapshot". Inside these are the .tar.gz and .backupinfo files. Is the problem that there is already a "snapshot" folder here? (The message is asking me to create it, but it already exists.)

The error message also says to change the ownership of the "nfs-tarballs" folder to yellowtent:yellowtent. If I do that, it changes the ownership to "808:808" on the office device (because that is where the files actually are) - is that what is intended? If I try this and click the Restore button again, I get the message "Failed to unmount existing mount". If I then unmount cloudron-restore-validation, and click the Restore button again, I get the error message: "Unable to create test file as 'yellowtent' user in /mnt/managedbackups/cloudron-restore-validation/nfs-tarballs: EACCES: permission denied, open '/mnt/managedbackups/cloudron-restore-validation/nfs-tarballs/snapshot/cloudron-testfile'. Check dir/mount permissions". Inside the "snapshot" folder there is a full set of .tar.gz and .backupinfo files, but no "cloudron-testfile". What should the "dir/mount" permissions be?

[james]

Hello @davejgreen

@davejgreen said:

The "nfs-tarballs" folder, and all its contents, have permissions "drwxr-xr-x djg djg"

This sounds like the issue.
Since the Cloudron user yellowtent can't access that, it fails.

How does this set up work on the live/production system?
What permissions are set there so it can access the NFS?

@davejgreen said:

If I do that, it changes the ownership to "808:808" on the office device (because that is where the files actually are) - is that what is intended?

So what are the permissions for the live/production NFS?
Can we do some comparison here?

So we have to figure out why the live/production system has no issues with permissions but the new one does.
I assume, that would solve it all.

[davejgreen]

Ah, I think I understand the issue. On our existing server, yellowtent has UID 1000. This is why the office device that is receiving the NFS Tarball backups has everything owned by "1000:1000". But on the new server, I created my user before installing cloudron, and it happened to be assigned UID 1000. Then I installed Cloudron and it must have assigned yellowtent to 808 because 1000 was taken. So, then the ownership does not match.

Thank you, I believe I can sort this out now!

[jdaviescoates]

@davejgreen said:

on the new server, I created my user before installing cloudron

FYI, afaict all the issues you've faced were caused by you doing something that isn't needed

e.g. the above creating a user, plus editing hosts files stuff mentioned previously.

I've migrated my Cloudron server probably 3 or 4 times without any issues

[davejgreen]

Just to clarify, the editing of /etc/hosts was irrelevant and did not cause any of the issues I was having. This is a business set up, and I have been instructed to create at least 2 users on the server, to give them admin rights, and then prevent ssh-ing in as root, so the user accounts on the server are needed. I needed to add the server to our tailscale network before doing the restore, as that is how the server will access the backups. It seemed sensible to add the users and check ssh-ing between devices when I did that.

I wondered if it would be worth adding a note to the migration docs about making sure the UID of the "yellowtent" user on the old server is available on the new server when Cloudron is installed?

[james]

Hello @davejgreen

@davejgreen said:

I wondered if it would be worth adding a note to the migration docs about making sure the UID of the "yellowtent" user on the old server is available on the new server when Cloudron is installed?

That might be something good to add.
But also stems from the issue that the installation instruction was not followed to the letter.
If one does run only the cloudron setup and nothing else, this issue would not arise.
So maybe even a check in the Cloudron installation script to check if a user with UID 1000 is already present.

[james]

For future readers.

I was also just thinking, this issue could be resolved by the NFS host to allow other users access as well.
With setfacl you can allow multiple users to access files/folders.

One could run the following commands on the NFS host to ensure all existing folders and files and future files can be accessed by UID 808

# as user root or run with sudo

# Apply ACLs to existing content
setfacl -R -m u:808:rwx /mnt/nfs-backup

# Apply default ACLs for future content:
setfacl -R -d -m u:808:rwx /mnt/nfs-backup

But for this to work the NFS host must use NFSv4 with idmapping configured properly.
Also, root_squash may block expected access, but your post:

  services.nfs.server = {
    enable = true;
    exports = ''
      /export/nfs-cloudron-backups <old-server-tailscale-IP>(rw,sync,no_subtree_check,no_root_squash)
      /export/nfs-cloudron-backups <new-server-tailscale-IP>(rw,sync,no_subtree_check,no_root_squash)
    '';

Has explicit no_root_squash as documented https://docs.cloudron.io/guides/nfs-share#exposing-a-directory

[davejgreen]

I tried again with a fresh install of Ubuntu 24.04 (this wipes all data on the server, so there is nothing left over from previous attempts). I checked UID 1000 was available, installed Cloudron v9.0.17 (following everything to the letter), but I still ran into exactly the same problem. The new installation gave yellowtent the UID 808, which does not match the UID of yellowtent on the existing server (1000), which is the sole cause of the problem.

The existing server was installed long before I worked here, so I don't know why yellowtent has UID 1000 there, while fresh Cloudron installs seem to be giving it 808. I read that UIDs of 1000 and over are for "normal" users, while those below 1000 are for "system" users. Has the yellowtent user been changed from a "normal" to a "system" user at some point? Or is the UID assigned by Ubuntu? Maybe different versions of Ubuntu do it differently? Or maybe there was some anomaly when our existing Cloudron instance was installed that caused yellowtent to get 1000 instead of 808.

I don't really know anything about idmapping - I had a quick go at the setfacl thing, but it spat out hundreds of lines ending in "Operation not permitted". In any case, I figured we probably want to fix things so yellowtent has the same UID so it can continue using the existing backups after the migration. So, I did another fresh install of Ubuntu and started again. This time, once Cloudron had been installed and I had rebooted the server, I did the following on the new server:

# Stop everything yellowtent is involved with:
systemctl stop box
systemctl disable box
systemctl stop cloudron-syslog.service

# Check this returns empty:
ps -u yellowtent

# Switch the UID:
usermod -u 1000 yellowtent
groupmod -g 1000 yellowtent

# Fix ownership on everything owned by 808 (takes a few mins):
find / -xdev -user 808 -exec chown -h 1000 {} \;
find / -xdev -group 808 -exec chgrp -h 1000 {} \;

# Restart stuff:
systemctl start cloudron-syslog.service
systemctl enable box
systemctl start box

After that, I was able to continue and complete a successful dry run restore on the new server.

[robi]

Sounds like the Cloudron installer needs an update @girish

[jdaviescoates]

@davejgreen said:

I tried again with a fresh install of Ubuntu 24.04

@davejgreen said:

Maybe different versions of Ubuntu do it differently?

I think the only time I've had issues with migrations was when the provider of my VPS (I think it was Netcup) began using stripped down versions of Ubuntu that had stuff Cloudron needs missing.

But I'm guessing you're pulling Ubuntu directly from Ubuntu?