Some struggles with Cloudron migration

[SansGuidon]

Hey,

I'm still migrating my server from Contabo to Hetzner, so far the dashboard works but I'm not done yet and I had some struggles on the way:

• I had restored Cloudron using a backup configuration downloaded from a previous instance, but the private key in the backup is somehow filled yet now valid and the error I got was "could not determine mount failure reason", which is not clear at all.

• The restore guide is not super clear about that, but before the first restore operation, I forgot to point my.<domain>.<tld> to the new IP so I would never reach the new dashboard properly, Cloudron would remain in a non ready state. I noticed in the restore guide that forgetting to update the DNS record to the new IP could be disastrous so I just started afresh reinstall again. Until I face the same situation again despite having correctly pointed the DNS record to new IP. By checking the logs, I noticed that likely I had issues with ACME/DNS because my Firewall was blocking port 80. Once I fixed that, after some patience, I was able to reach the new Cloudron dashboard.

• Apps restore are queued. So I know I have to point all apps DNS records to the new IP. As I'm using Hostinger, it's all manual, yet it's quick. But apps restore are still in queue.

• Volumes were not reachable. I had a strong FireWall configuration applied to my new server and somehow Cloudron just does not handle that well, I just had some weird errors like "mount inactive could not determine mount failure reason" similar to https://forum.cloudron.io/topic/12692/backup-failed-backup-endpoint-is-not-active-could-not-determine-mount-failure-reason-failed-to-mount-inactive-could-not-determine-mount-failure-reason/2 when in fact it was just Hetzner preventing traffic through ports 22 and 23.

I'm still stuck with :

• sync dns failures.
• certs renewal is failing fully or partially.
• apps not restored so not available.
  
• most of services are not in ready state.
  
• some issues I didn't expect to have

# cloudron-support --troubleshoot
Vendor: Hetzner Product: vServer
Linux: 6.8.0-57-generic
Ubuntu: noble 24.04
Processor: AMD EPYC-Milan Processor
BIOS NotSpecified  CPU @ 2.0GHz x 4
RAM: 15985236KB
Disk: /dev/sda1       130G
[OK]	node version is correct
[FAIL]	Server has an IPv6 address but api.cloudron.io is unreachable via IPv6 (ping6 -q -c 1 api.cloudron.io)
Instead of disabling IPv6 globally, you can disable it at an interface level.
	sysctl -w net.ipv6.conf.enp7s0.disable_ipv6=1
	sysctl -w net.ipv6.conf.eth0.disable_ipv6=1
For the above configuration to persist across reboots, you have to add below to /etc/sysctl.conf
	net.ipv6.conf.enp7s0.disable_ipv6=1
	net.ipv6.conf.eth0.disable_ipv6=1

I'll keep posting updates, I believe it's important to learn that despite the migration seems easy, there are some caveats that can force to stay hands on.

EDIT:

• After checking the install guide (not the migration not restore guide), I found more information about the recommended Networking/Firewall config: https://docs.cloudron.io/installation/#firewall-setup and https://docs.cloudron.io/installation/#install_1 this info was AFAIK not mentioned during the migration guide?
• I tuned the Firewall config of Hetzner to the recommended settings and restarted the instance. It turned out to be better for Cloudron:
  
  also for apps restore (still in progress)
  
• Ok, after unblocking port 53, it seems now my apps are struggling to restore as DNS records pointing to IPV6 are expected, which i didn't provide. I'll fix the DNS setup to add the AAAA records pointing to the public ipv6.
• Apps start to restore as expected after adding all the AAAA records pointing to IPV6.

That's all, folks

TLDR; A few confusing guide steps, unclear error messages or prerequisites, inconsistent docs have led to more suffering than expected.

I wish however that this inspires some improvements in the platform and documentation.

Thanks for reading

[BrutalBirdie]

Great writeup!
Sorry it was this difficult in your case.

---

Feedback for improving Cloudron

After reading verything I would like to filter the pain points to improve Cloudron.
If you @SansGuidon could confirm what I assume I understood from your post, that be great

Issue 1 - private key pre-filled with junk:

When restoring a Cloudron, using the backup configuration upload function for sshfs.
The private key field is pre-filled with ●●●●●● giving the illusion, that nothing needs to be done.
This will fail the restore since the private key is not included in the backup config

Issue 2 - Missing documentation about manuell DNS steps in restore process

I can't secon this issue @SansGuidon since https://docs.cloudron.io/backups/#restore-cloudron states:

If your domains use Wildcard, Manual or No-op DNS provider, you should manually switch the DNS of the domains to the new server's IP. At the minimum, you have to change the IP of the dashboard domain (i.e my.domain.com). Note that if you do not switch the IP for the app domains, the restore of those apps will fail and you have to trigger the restore of the app from the Backups section when you are ready to switch the IP.

Did you miss this? Should be placed higher? Should it be labeled with a visual warning for better visibilty?

Until I face the same situation again despite having correctly pointed the DNS record to new IP.

Could have been DNS cache either from TTL or Browser/OS.
You can always use the https://docs.cloudron.io/backups/#dry-run instructions for local DNS overrides to circument these issues.
Might be a good idea to add this to warning above.

Apps restore are queued. So I know I have to point all apps DNS records to the new IP. As I'm using Hostinger, it's all manual, yet it's quick. But apps restore are still in queue.

Like the above documentation point out, switching over the DNS should be done before restorring.
Since you had such troubles with manuell DNS, I would recommend you to switch your DNS provider from Hostinger to Hetzner.
If your Server already is with Hetzner, why not also the DNS?
You would have everything in one place.
ps: With switching DNS I do not talk about transferring domain ownership to the hetzner account. Just DNS Zone delegation.

@SansGuidon said in Some struggles with Cloudron migration:

• sync dns failures.
• certs renewal is failing fully or partially.
• apps not restored so not available.

DNS propagation and DNS cache leading to all three of these problems.

Issue 3 - Firewall

I found more information about the recommended Networking/Firewall config

The recommeneded Network/Firewall config is:
https://docs.cloudron.io/installation/#firewall-setup

We do not recommend adding and modifying rules in iptables/ip6tables since Cloudron already does this.
But since you used the Hetzner Firewall, this line did not exactlly apply to you.
In my opinion, the extra Firewall from the provider is only for expert users.

IMHO, potato potahto, same same but different.

You post suggests that only now, after the switch to Hetzner you started using the extra Firewall from a provider.

Not finding this documentation in the restore section makes sense to me personally.
Since, why include it there? If you have done it before, you know what to do.
If you have never done it, why add extra confusion to the reader.

This whole part about the extra firewall is a mixed bag of feelings for me.
I really appriciate when users want to improve security.
But it always comes with the downsight of missing knowledge leading to problems or even worse, security issues.
Cloudron does provide a very good setup with iptables which is solid.

Also, what will cause issues in the future, is this sneaky little part:

Other ports have to be opened up depending on the apps installed. For example, the git+ssh port has to be opened when using GitLab. Port 53 (UDP) is required if you install an app like AdGuard Home.

Yes, you have read it now. At this exact moment you are aware of this task.
Given one month.
Example:
Cloudron release a new awesome app you want to try out, SFTPGo.
Should be simple like always, click click click.
!ERROR! Not working! whyyyyy?!
Well the form when installing had the following ports mentioned:

• SFTPd 2022
• FTPd 2121
• FTPd Passive 40000 (*even more sneaky, it is a port range with 100 ports, so 40000 to 40100)

Did you set up everything in the provider firewall?
No.
This also reduces the user experience of Cloudron.

What if. . . Cloudron would include APIs for provider Firewalls and would autosetup that as well.
Could be possible. Tremendous effort for the Cloudron maintainers.
iptables exist and works, why add this duplication.
Could be an enterprise feature.
I would not press this feature.

--

I hope my rambling made any sense

To everyone reading this topic, please participate in the discussion.
I would love some more feedback, since I believe the easy backup and restore is one of the key features what many users love.

[SansGuidon]

Thanks for the great feedback and I know my post could have been confusing, it was really written late at night

• your remark about the section about wildcard / dns makes sense, that was a pebkac as the first time I overlooked that section and forgot that I was directly concerned by this. However even on the second attempt, when I did point DNS to the right IP, things still got stucked. Because I had to configure my FW to let Cloudron traffic out in order for the dns resolution to work. Obvious you would say, but not at night in a rush and without much information.
• I admit mentioning Firewall in the migration/restore guide might be too much, but you could consider that in context of a migration to a new VPS provider, especially if this migration is not executed frequently by the operator, and in order to improve the overall migration experience, it makes actually sense to refresh the guide reader about the network requirements (open x ports), there could a link to the installation guide that covers the requirements for having specific ports open. When I find a migration guide I don't specifically think about reading the whole install guide as I assume the migration is a shorter version of the install. Also maybe not everyone is willing to rely solely on Cloudron config for the security. Hetzner gives lot more power to users and control, and Cloudron sometimes feels a bit too vague.
• As a power user (DevOps) and busy dad, Cloudron feels both nice or confusing/limited at times depending what I want to do.
• The mounts issues errors were for most related to network. The message must be clearer.
• I could use Hetzner for DNS but we have already a long term subscription with Hostinger for several domains so we are anyway still stuck a bit. Also I could argue that putting all my eggs on same provider is risky. I had an outage with Contabo in the past where both VPS and Backups location were impacted. this kind of situation is a pain in the ass.

I would assume this whole migration story is a mix of pebkac and unclear documentation, confusing UX, confusing error messages and missed automated checks opportunities. And I wanted to contribute this thread in the hope to open a debate about this and improve the overall experience.

BTW all apps are up & running, I have yet to fix a few email settings, but that will have to wait til I'm back home and fresh after a whole day YAMLing sigh

[BrutalBirdie]

@SansGuidon said in Some struggles with Cloudron migration:

pebkac

What?
Googling . . .

PEBKAC stands for "Problem Exists Between Keyboard and Chair". It's a tech support term, often used humorously or dismissively, to indicate that a problem is caused by user error rather than a technical issue. In essence, it's attributing the problem to the user's actions or lack of understanding, rather than a software or hardware malfunction.

AHHHHH! I know the German version "Das Problem sitzt vor dem Bildschirm" litteral translation "The problem is sitting in front of the screen".
Nice one

@SansGuidon said in Some struggles with Cloudron migration:

it makes actually sense to refresh the guide reader about the network requirements

Good point! Just a "remember your Network setup!" with a link to the other doc would be enough. Right?

@SansGuidon said in Some struggles with Cloudron migration:

Also maybe not everyone is willing to rely solely on Cloudron config for the security. Hetzner gives lot more power to users and control, and Cloudron sometimes feels a bit too vague.

Let me be brutally honest with you. German mode enabled.
99% of users that use Cloudron do not have the knowledge and experience to proivde that.
What Cloudron provides is enough for the 99%.
Generally speaking, not targeted towards you.
IMO everything further is "expert mode" and the expert should know what to do.
And if the expert does not know how, he should either learn or pay.

@SansGuidon said in Some struggles with Cloudron migration:

As a power user (DevOps) and busy dad, Cloudron feels both nice or confusing/limited at times depending what I want to do

I get that
That is the gilded cage experience and to some extent necessary to keep support low.
If you are working in DevOps, I've built multiple customer solutions with Cloudron, Cloudron Gitlab, Cloudron DockerRegistry and the Gitlabrunner to fully automate, development, staging and production deployments.
It is possible, but you need a firm grasb on what Cloudron does and how it does things.
Then you can make it do some crazy things.

@SansGuidon said in Some struggles with Cloudron migration:

The mounts issues errors were for most related to network. The message must be clearer.

Error message improvment. Everyone can benifit from that.

Should there be a normal message with a button "more details" for power users like you and me?

@SansGuidon said in Some struggles with Cloudron migration:

I could use Hetzner for DNS but we have already a long term subscription with Hostinger for several domains so we are anyway still stuck a bit. Also I could argue that putting all my eggs on same provider is risky. I had an outage with Contabo in the past where both VPS and Backups location were impacted. this kind of situation is a pain in the ass.

Egg Basket, agreed. If not Hetzner, some other supported DNS provider that fits your needs so you get rid of the manuall labor? => https://docs.cloudron.io/domains/
My top picks in no order are:

• https://desec.io/
• Cloudflare
• Hetzner
• DigitalOcean

@SansGuidon said in Some struggles with Cloudron migration:

And I wanted to contribute this thread in the hope to open a debate about this and improve the overall experience.

Yes please! Always and more of that.
What ever you find that is "meh", report it in the forum. Everyone benefits.