Cloudron Forum

@girish said in Nextcloud backup crashes:

@opensourced I have been testing things on the side and tried copying over such amount of data to a CIFS mounted Hetzner Storage and just cp -r command failed in the middle. rclone failed as well since the Hetzner storage box mount point disconnects/breaks randomly. I am getting farther with rsync but currently cloudron does not have an adapter for rsync. We are still working on this. Fundamentally, backing up 2.5TB of data reliably is a very complicated task.

All this to say, I don't really have a solution but if someone has ideas, I can try something else.

@girish A few years ago you helped me out with a similar issue (not on forums but via a support ticket) with backing up a large Nextcloud instance (2TB+). The issue was with timeouts and you advised how and where to update to extend the timeout window significantly. I was using a Hetzner Storage Box in Germany and a VPS in Sweden using a CIFS mount. Later I moved it to an sshfs, rsync to a local server.

This fixed the issue and allowed me a couple of years of backups of that large Nextcloud instance (with many other apps running concurrently) without problem. I don't have those emails any more or access to that instance but wanted to mention in case a reminder is useful to you here as it seems to be a similar issue.

@girish said in Notes on Cloudron, crowdfunding app packaging:

In any event, securing some sensitive apps (such as Nextcloud / Vaultwarden) via a "advanced VPN" solution might make sense.

It's in our TODO list to allow access to specific apps only via VPN. The ever growing TODO list

@necrevistonnezr Might be useful to know that there is a Nextcloud app, "Restrict Login to IP address" that can be used to restrict access to Nextcloud except via enterprise VPN. I used that successfully for a number of years, though as with everything Nextcloud it can break on updates if the app doesn't keep up!

Having used cloudron in both the business (SME) and home settings, I concur with everything that's been said here about cloudron being completely suitable for home use, albeit with different use cases.

On the install guide it might be useful to explicitly mention the subsections on Home Server and Intranet, in case people don't look at or see the side bar (easy to do outside of viewing full screen). Something as simple as something at the end like "Additional information is provided for those seeking to install Cloudron on a home server or intranet."

@girish I'm pleased to say that it worked. I had to disable docker and box and reboot, even though I didn't get the exact error you mentioned but something else related to creating the network (duplicate network). I also had to reboot after enabling and restarting box because otherwise the upgrade got stuck, I believe for a similar network creation problem. But once I did that everything worked well and I'm now happily sitting with version 7.6.3 after two subsequent updates. Many thanks for the help with this! Hopefully this thread will be useful for anyone else who might have this issue in future.

@girish Thanks. That is reassuring. I just found your response to my ticket in my spam folder for some reason. I'll see if I can work this out and will report back.

@jdaviescoates said in Update to 7.6.1 failing:

@eganonoa said in Update to 7.6.1 failing:

Whenever there is a later cloudron release (e.g. 7.6.2) or release candidate will I be able to update to that and skip 7.6.1 entirely?

There is already 7.6.2 and 7.6.3 but you can't skip versions.

Ouch! So I'm stuck and my system is effectively broken.

@nebulon That sounds horribly drastic! Whenever there is a later cloudron release (e.g. 7.6.2) or release candidate will I be able to update to that and skip 7.6.1 entirely? I'd probably rather wait than effectively going the "rm -Rf /var/lib/docker" route.

@nebulon Unfortunately, that doesn't work. Tried again just now, did it with and without docker restart and system restart, none worked.

From the various things I've found online it seems that when a pull is forcibly interrupted in some way that prevents docker from effectively clearing up the broken pull, docker is unable to find the broken fragments of the prior pull when you are trying to manually prune, but apparently can find it when trying another pull. From what I've seen this can happen for a number of reasons mid-pull, from a power outage, to a temporarily poor network connection, to a force shutdown of the host system.

And I've not seen anyone report being able to fix it other than to rm -Rf /var/lib/docker. The only reference to an actual fix I've been able to find comes from this issue in the Moby Project, which references a patch they've applied.

My assumption is that this hasn't been addressed, even though it looks on its face to be a major (and frankly ridiculous) problem and was raised as far back as 2017, because ultimately the issue "fixes" itself when a new version of whatever is being pulled from docker is released. This then allows whoever is experiencing the problem to simply move on keeping those broken fragments somewhere on their system doing nothing. So while the issue seems big, it isn't in practice.

What I'm worried about is that this will mean I'm stuck manually updating apps and not the cloudron instance until another version of cloudron is available because the whole update fails in perpetuity when a pull of a component app fails unexpectedly. And I can't remember whether cloudron will even show me that new version to update to until I've been able to update to 7.6.1 first.

@girish I think I understand what is happening here, so thought it best to write on this open thread as it may be "helpful" to others (helpful in quotations because there doesn't seem to be a nice solution!).

The problem appears to be that at some point, for whatever reason, the initial attempt to pull the cloudron/mail:3.11.2 image failed unexpectedly mid-pull (a system reboot perhaps). This seems to create a set of "dangling layers" related to that image that are hidden somewhere and cannot be found or removed with "docker system prune".

When I try to pull cloudron/mail:3.11.2 from the CLI, I see the following:

root@cloudron:/var/lib/docker# docker pull registry.docker.com/cloudron/mail:3.11.2
3.11.2: Pulling from cloudron/mail
445a6a12be2b: Already exists 
4cfe0cdc770e: Already exists 
e6a0eb1fa9b7: Already exists 
e995e5b957f9: Already exists 
e6d226089461: Already exists 
b3243df2776e: Already exists 
debd247c1af3: Already exists 
ea1f575bfbef: Already exists 
566e1eaf48e1: Already exists 
68da526a8544: Already exists 
2f3677647d18: Already exists 
90984d402264: Already exists 
802deede2955: Already exists 
1861003a8fe7: Already exists 
524cf22ec2b3: Already exists 
7604fee16283: Already exists 
930850c4bc07: Already exists 
844343c15467: Already exists 
367e21d14918: Already exists 
6880f0889c4f: Already exists 
5a544c4b0196: Already exists 
7fb39aa7d081: Already exists 
c10af7a3bade: Already exists 
30c062ec98da: Already exists 
956058cafb63: Already exists 
e5c1b069b2dc: Already exists 
27d50608d341: Already exists 
d6d99d73528f: Already exists 
35ad04d78685: Already exists 
0fa0098bd9c2: Already exists 
1289a176c743: Extracting [==================================================>]     690B/690B
bc9e5abd8c84: Download complete 
8c8b3e2950c7: Download complete 
025029896e5c: Download complete 
fc053eff195d: Download complete 
failed to register layer: error creating overlay mount to /var/lib/docker/overlay2/af6ccf9d7fbcd56d80cf2b84a6cedadc60e0f1615d06cf0947b140f244fb200d/merged: too many levels of symbolic links

As you can see, most layers are found as already existing, but when it hits layer 1289a176c743 it extracts it and then fails with the symbolic link error. No matter what I can do I cannot find and delete that.

Others appear to have had this problem with docker preventing future pulls where there are problems with the initial pull. See e.g. here, here link text and the current open issue and here.

This docker problem is compounded with Cloudron. For other, more bespoke, systems you could just move to a different version of whatever it is you are pulling. But with Cloudron the whole package is delivered at once and as a complete package, so if one pull fails the entire update fails, leaving you stuck.

The only solution I can find is listed here, specifically

systemctl stop docker; rm -Rf /var/lib/docker; systemctl start docker

I don't like the sound of doing that! What would the consequences of that be for all my in-app configurations, etc. Would this not be a fresh install?

The good news is that I can pull all other cloudron/mail images, including 3.11.3 and 3.11.4. So I imagine I can move around this issue if there were an update available that bundled any other version of cloudron/mail than 3.11.2. Is that possible or will my cloudron never allow me to access that update until version 7.6.1 is applied?

@girish FYI I ran "docker system prune" and "docker image prune" and restarted docker. Sadly this didn't work.

@girish I found the same when trying to work this out. Seems like a totally random error, with the only solution seemingly to run

docker system prune

which I am hesitant to do as I don't know what will happen.

I've raised a ticket via the cloudron dashboard so that you can see my subscription info.

Many thanks.

@girish I can manually update my installed apps. So it seems to be getting hung up only on a system update and specifically on pulling cloudron/mail with this "too many symbolic links" error.

@girish Tons of space, and everything is vanilla.

Hi @girish. Unfortunately I do. Sorry to take so long to respond. The updater logs were empty. It looks like the problem is being caused by a failure to pull from docker hub because of a problem with too many symbolic links. Here is the relevant error message I am getting when it tries to pull cloudron mail:

to register layer: error creating overlay mount to /var/lib/docker/overlay2/51bddb0c4888bfbcb70a111bf5d13a93baea5652c6f52f837bf6153cd2be9fe4/merged: too many levels of symbolic links

Unfortunately it took a long time to get to this point because what I didn't realize is that even when I stop the update from the cloudron dashboard the updater continues to try to pull in the background every five seconds and quickly triggers a rate limit on docker hub. As a result, I have to reboot the entire server each time I try the update to prevent this happening.

Just noticed a series of update failures. My system is currently on v7.5.2 on an Ubuntu 22.04 machine. The cloudron error log merely says:

"errorMessage": "update exited with code 1 signal null",

Is anyone experiencing similar?

@girish said in allowlist.json in Bedrock Server?:

@eganonoa I put some docs at https://docs.cloudron.io/apps/minecraft/#allowlist

Thank you very much!

Are we able to limit the Bedrock server only to white listed playees as we can in Java? I see that from the Docker guide whitelisting needs to be set as an environment variable. Has that been done? If so, where do we place the allowlist.json?

@potemkin_ai said in What's coming in 7.5:

@eganonoa thank you, that makes much sense.

A few questions/proposals if you wouldn't mind:

1. Are you blocking any other access to Cloudron except via Cloudflare? If so - is it a precautious or a mitigation against well understood problem? If the later - could you please, share your experience?

2. I guess it's more to @nebulon and @girish actually - can't nginx proxy TURN/STUN traffic as well, reducing the required ports and system requirements as well?

For 1. Cloudflare proxying, its WAF with quite restricted settings outside of our static IPs. Then various app-level things as necessary. Mostly a precaution as we know our systems (not Cloudron) have been directly targeted by some sophisticated actors in the past.

For 2. there's been a bit of discussion on this (both re access to turn and the difficulty with VOIP services not running on 443) over the last few years here. Also worth checking out discussions outside of Cloudron for things like Nexrcloud Talk, Jitsi Meet, BigBlueButton. Upshot is that one way or another (whether because you run behind a NAT or just have users win the corporate/academic/government spheres with restrictive firewall rules) you really want an external turn, something that listens directly on 443 and can direct traffic. Theoretically there are (apparently) ways around it, but it adds levels of complexity that are just unnecessary given how utterly trivial it is to run an external turn. If interested BigBlueButton have a script that will set you up without any issue (https://github.com/bigbluebutton/bbb-install#install-a-turn-server)

Ultimately, I think we have to recognize that trying to make Cloudron provide all services to all people at all times is unworkable. If it provides a fully functioning base system and then allows flexibility for those needing more "complex" systems, then it is doing its job perfectly. This Redis and Turn change - long requested - is exactly that kind of solution.

@girish said in What's coming in 7.5:

@eganonoa synapse update is now pushed and has optional turn.

Really wonderful. Thank you. Now restarting matrix does not overwrite that section of homeserver.yaml, with the added bonus that if you ever want to revert to the in-built turn you just "flip a switch" as it were and the settings revert to default. That's a very nice implementation.

@potemkin_ai said in What's coming in 7.5:

@eganonoa just for my information - why are you looking for external TURN servers for Synapse/Matrix and NextCloud? What are the benefits?

A couple of reasons that make calls requiring a turn server not function well.

• We run our services behind Cloudflare, and turn servers don't work well (or at all) via reverse proxies like that as the server cannot accurately direct traffic to the correct IP addresses.
• Even if we didn't use Cloudflare proxying, we have many calls with people in academic and government environments with policies limiting what ports they can connect to, usually only allowing 443. Because Cloudron monopolizes that port its turn server has to run on a different port, so those people cannot use the Cloudron turn server even if we turned off Cloudflare proxying (which we don't want to do).

As a result, the ability to use an external turn server with Cloudron is critical and a very welcome development.

Thanks @girish and @nebulon. If you need to prioritize, for the turn server the one that is simply impossible to add an external turnserver is matrix/element as it requires a restart to apply changes, which then of course leads to those changes being overwritten. I believe the others don't have that limitation, but at the same time, Nextcloud would probably have the quickest positive benefit as it is quite trivial to add the external turn server via the Nextcloud admin panel.