Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    Solved Backup feedback over sshfs

    Support
    sshfs backups
    6
    35
    1034
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R
      Robin last edited by girish

      Some feedback on trying to use sshfs as a backup mechanism. Setup was: cloudron node connected to a remote server hosted in a different country using sshfs.

      • Using the rsync method, this was unusably slow, likely due to smaller file sizes (~10 minutes just to process a G or two of data, despite having a decent internet connection on both ends).
      • Using tarballs was a bit of a better experience, the backup did eventually complete, but this is not really a practical possibility given the size of data I was trying to back up at present (as I understand it, basically, each day would involve a full upload of those tarballs).

      Not sure what can really be done about this from the Cloudron side without some pretty serious surgery, but thought that it was worth mentioning.

      1 Reply Last reply Reply Quote 0
      • girish
        girish Staff last edited by

        For the last release, I did some basic performance tests to get the backup speed to be at par with something like rclone. I did not test sshfs specifically but I am curious how long it takes to copy this amount of data using normal tool in your setup? For example, how long does a cp of so small small files take? (Just rough number is fine). Granted, Cloudron being node.js based is not going to be as fast as native tools, but given that both of them are just copying files sequentially, i don't expect that big a difference. (Like maybe Cloudron takes a minute or two more).

        1 Reply Last reply Reply Quote 0
        • girish
          girish Staff last edited by

          In the other thread, I saw that you were using encrypted backups. Is this the case when you tested this? Encryption requires a decent CPU as well, so a simple cp is not a fair comparison. I wonder if you can enable encryption on the remote disk somehow and not rely on Cloudron encryption?

          1 Reply Last reply Reply Quote 0
          • R
            Robin last edited by

            For the last release, I did some basic performance tests to get the backup speed to be at par with something like rclone. I did not test sshfs specifically but I am curious how long it takes to copy this amount of data using normal tool in your setup?

            My guess is that the problem is inherent to sshfs, and is not something specific to Cloudron (sshfs not being fast is pretty widely known/recognized). The only real reason I am reporting this is to perhaps suggest noting that it does not scale well in the docs, or something like that, because I don't think this gives a particularly good impression as-is.

            sshfs is very sensitive to latency, so any cases where you run it to a remote server on a different network, the slowdown will be very noticable.

            Using a tool more suited to this style of operation (like, say, rsync) will give better results, but it will still not scale well for large files (or large datasets, over say a few hundred GB of small files).

            In the other thread, I saw that you were using encrypted backups. Is this the case when you tested this? Encryption requires a decent CPU as well, so a simple cp is not a fair comparison. I wonder if you can enable encryption on the remote disk somehow and not rely on Cloudron encryption?

            Correct, encryption was enabled, but it wasn't really a factor. CPU was more or less idle while this was going on. Remote disk encryption isn't really an option in this case because I don't have physical control over the remote host, so I want to keep the data hidden 🙂

            1 Reply Last reply Reply Quote 1
            • rmdes
              rmdes last edited by rmdes

              We have been using sshfs backup for a few months now, I had same experience with rsync, it takes several days in our case to finish, so I tried tgz backup, it's way faster than rsync but what really bothers me is that each time the sshfs mount breaks (no idea why, sometimes it gets dismounted out of nowhere) then when I remount it all the backups are gone, meaning, they are physically on my storage box but not listed on each apps or on the main backup view.

              I have to make a full new backup for it to be listed until the next mount drop...

              of course If I attempts to clean the backup, since previous backup are not indexed, they don't exist and need to be cleaned up by hand (which is something I would love to avoid)

              Is there anything I could do to improve this situation or perhaps force some kind of scan that would index properly previous backups (before the sshfs mount breakage) ?

              edit : not using encrypted backup at this stage

              girish 1 Reply Last reply Reply Quote 1
              • girish
                girish Staff @rmdes last edited by

                @rmdes said in Backup feedback over sshfs:

                Is there anything I could do to improve this situation or perhaps force some kind of scan that would index properly previous backups (before the sshfs mount breakage) ?

                It's hard to know why this is happening without looking into the server/logs. If you can email me on support@ , when things are a bit broken, I can debug it further.

                rmdes 1 Reply Last reply Reply Quote 0
                • rmdes
                  rmdes @girish last edited by rmdes

                  @girish OK next time it breaks I'll send an email about it, leaving it as it is so that you can have a look, I already tried to find the reasons but beside "time-out" I couldn't find anything particular 😕

                  checking everyday and now that I'm waiting for it to un-mount, it does not 🙂 haha

                  rmdes 1 Reply Last reply Reply Quote 1
                  • rmdes
                    rmdes @rmdes last edited by rmdes

                    @girish here we go :

                    45598875-41c8-46c2-8927-8d7be665bf2f-image.png

                    This is after a reboot of the server.

                    but this time I clicked Remount Storage and instead of "removing" previous backups, everything is mounted and backups are listed as usual..

                    I think the other times this happened, I was going to Volumes, reconnect the mount (maybe this is what makes previous backups, inside the volume, unlisted after a remount?)
                    then go to backup and Remount the storage, but this time I just clicked once, in the backup view, remount storage and all is fine.

                    19e887f5-548e-40df-8fc5-05d591b77b31-image.png

                    nebulon 1 Reply Last reply Reply Quote 1
                    • nebulon
                      nebulon Staff @rmdes last edited by

                      @rmdes thanks for the details. So this is the same issue with the systemd unit dependencies during boot. The mount units are simply skipped it seems due to this circular dependency. We have to fix this for the next release, just not yet sure how 🙂

                      rmdes 2 Replies Last reply Reply Quote 0
                      • rmdes
                        rmdes @nebulon last edited by

                        @nebulon What if the mount was being made persistent at the OS level so then when cloudron reboot it could check those system mounts and if they match existing volumes, mount them properly ?

                        nebulon 1 Reply Last reply Reply Quote 0
                        • nebulon
                          nebulon Staff @rmdes last edited by

                          @rmdes they are setup with systemd, so they are system-level already. The issue is DNS/unbound dependency for the mounting as well as docker for the volumes. Since there is some circular dependency currently, systemd breaks the circle by disabling the mount points during startup, which is causing the issue that they are not mounted after reboot.

                          rmdes 1 Reply Last reply Reply Quote 1
                          • rmdes
                            rmdes @nebulon last edited by

                            @nebulon and what if the systemd mount unit was delayed, to a certain level so to avoid the breakage ?

                            1 Reply Last reply Reply Quote 0
                            • girish
                              girish Staff last edited by girish

                              Does the xx.your-storagebox.de resolve to a static/fixed single IP?

                              rmdes 1 Reply Last reply Reply Quote 1
                              • rmdes
                                rmdes @girish last edited by

                                @girish nope it does not and hetzner storage box do not allow me to get one...

                                1 Reply Last reply Reply Quote 0
                                • rmdes
                                  rmdes @nebulon last edited by

                                  @nebulon @girish Not in a hurry or anything, but the state bug is happening (happens at each reboot basically) I've left it unmounted and will for the weekend if at some point you want to SSH-support and retrieve log traces related to the mounting issue or I can post here if you indicate the commands you want to get more context ?

                                  rmdes 1 Reply Last reply Reply Quote 0
                                  • rmdes
                                    rmdes @rmdes last edited by rmdes

                                    I have remounted the volume now and something that really is problematic with this type of backup is that, even when you remount the volume and storage backup location (that part works just great with two clicks) it seems previous backup at the mounted location are not listed, it's like a brand new freshly added volume/storage.

                                    Meaning, previous backups can only be restored manually, they can be retrieved anymore from cloudron dashboard, the only way is to follow the manual instruction from the doc to retrieve backup ID etc..

                                    manual remount wouldn't be an issue for my use case If I could get the existing backup listed when the volume/storage is remounted.

                                    volume view remounted :
                                    bb279ea9-0b29-4ef4-b8f7-ead6c5511750-image.png
                                    backup view remounted :
                                    7453e2e1-041c-4e37-9d69-49bcc9b9a022-image.png
                                    the backup listing view after everything is remounted :
                                    3bb93157-17c0-42c0-a94c-c7841838e9a1-image.png

                                    I was wondering if it wouldn't be a good idea to have a button to rescan the backup target for backups that match previously made cloudron backups ?

                                    nebulon 1 Reply Last reply Reply Quote 1
                                    • H
                                      hendrikvl last edited by

                                      I use a Hetzner storagebox via sshfs as backup provider and also have the issue with the volume being not mounted after reach reboot. Clicking the remount volume button solves it for me though. All previous backups are correctly listed after remounting.

                                      Having to remount the volume manually is somewhat annoying though and I would appreciate a solution which solves the issue with the circular dependency on boot.

                                      1 Reply Last reply Reply Quote 1
                                      • nebulon
                                        nebulon Staff @rmdes last edited by

                                        @rmdes actually I am not sure if you use that thing correctly. Volumes and the backup storage are not really related. The way you have it configured, the same mountpoint would be mounted twice. Once for the volume (which is supposed to be used within apps) and the backup storage.

                                        @hendrikvl we try to fix this reboot and volume mounting issue in the upcoming release.

                                        rmdes 2 Replies Last reply Reply Quote 0
                                        • rmdes
                                          rmdes @nebulon last edited by rmdes

                                          @nebulon wow, alright, I may have misunderstood the documentation then, will remove on the volume side

                                          • now I remember, I had it on the volume because one of my apps/domains was reading a music folder from the same sshfs mount point, which happened to be the same storage.
                                          rmdes 1 Reply Last reply Reply Quote 0
                                          • rmdes
                                            rmdes @rmdes last edited by

                                            So that must be why after a remount (wihtout removing the volume or backup mount)
                                            the newly mounted volume cannot "see" the pre-existing files ?
                                            a7737c96-190e-4b6f-a5f8-c70932057f77-image.png

                                            1 Reply Last reply Reply Quote 0
                                            • rmdes
                                              rmdes @nebulon last edited by

                                              @nebulon So this is the steps I did :

                                              • removed the volume since no app was using it
                                              • remounted the backup by clicking "remount storage" on the backup view

                                              All seems to be ready for a backup but old backups are gone :
                                              0cf883e3-196b-4bb7-a436-075e5dec18c3-image.png

                                              nebulon 1 Reply Last reply Reply Quote 0
                                              • nebulon
                                                nebulon Staff @rmdes last edited by

                                                @rmdes can you check on the server if /mnt/cloudronbackup is actually mounted or not? seems to be some discrepancy here.

                                                Further, if you change the backup storage configuration, previous backups will be removed from the listing, since a new backup storage may have different mechanisms to access them and Cloudron only ever stores one storage configuration. Note that the backup meta info is stored in the database, so even if the files would be on that disk, Cloudron does not look around the disk and tries to find existing backups...maybe it should, but it doesn't currently.

                                                rmdes 1 Reply Last reply Reply Quote 0
                                                • rmdes
                                                  rmdes @nebulon last edited by rmdes

                                                  @nebulon it is mounted, I can list all the backups just fine, it would be really neat to have cloudron reading some kind of index that could be generated after each backup keeping the backup directory tree index safe, even in between mounts ?

                                                  • it's totally fine and normal that if backup type is changed that it looses the index, but here it's not the case, nothing was changed, the machine was rebooted and lost the backup mount, I manually remounted it, but all the previous backups are not listed, even tho it's the same backup config.
                                                    I created a new backup now and had to remove all the previous backups manually to make sure I have something I can use if I need to restore an app without crippling my backup storage with untied backups that I can only retrieve manually.

                                                  • A bonus I noticed by having my backup storage mounted a second time (not the case anymore) in the Volumes view is that I could via the file explorer manage my backups with the cloudron file explorer, pretty handy to delete unneeded parts or to manage a non backup folder for music etc..

                                                  rmdes 1 Reply Last reply Reply Quote 0
                                                  • rmdes
                                                    rmdes @rmdes last edited by

                                                    I really wish there was a "scan existing backup" feature, because each time I have to remount, previous backups are gone from the UI

                                                    nebulon 1 Reply Last reply Reply Quote 0
                                                    • nebulon
                                                      nebulon Staff @rmdes last edited by

                                                      @rmdes maybe there is something else wrong, since a remount alone should not touch the backup listing as such. Only a reconfiguration of the backup config would invalidate the backup listing.

                                                      rmdes 1 Reply Last reply Reply Quote 0
                                                      • rmdes
                                                        rmdes @nebulon last edited by

                                                        @nebulon Maybe but I have no idea what can it be..
                                                        if I LS /mnt/cloudronbackup from the box everything is there
                                                        I will try to list /mnt/cloudronbackup next time the box reboot, before remounting in the UI and see what's up..

                                                        • one question : does cloudron backup write an index somewhere inside the mounted backup ?
                                                        nebulon 1 Reply Last reply Reply Quote 0
                                                        • nebulon
                                                          nebulon Staff @rmdes last edited by

                                                          @rmdes as mentioned, the existence of the files on the backup storage is not related to the listing of backups in the dashboard. The latter is based on the meta information stored in the database (so that is kind of the index you asked for), Cloudron will not try to scan and identify potential backups. Even if it did, it would be quite hard to automatically ensure those are correct and complete. So I think, since we are talking about backup storage, our current strategy to invalidate the listing, once the storage is adjusted is ok. Anyways the storage should not be changed frequently.

                                                          I think though, since we have solved the reboot and mount issue with upcoming release. The manual remount will not be required. Although I do not understand why the backup listing will be cleared out if you just remount, I can't spot any code which might do that.

                                                          rmdes 1 Reply Last reply Reply Quote 1
                                                          • rmdes
                                                            rmdes @nebulon last edited by

                                                            @nebulon alright, thanks for the explanation, will report feedback with the upcoming new version, indeed, if the mount problem is gone, no need to touch the backup config 🙂

                                                            rmdes 1 Reply Last reply Reply Quote 0
                                                            • rmdes
                                                              rmdes @rmdes last edited by

                                                              Good news ! even tho the reboot-mount bug is still there, since the last upgrade Backups keep being listed even if the mount point is lost or unmounted.

                                                              56761074-bb9d-416d-bc8d-4a5c3ad40146-image.png

                                                              rmdes 1 Reply Last reply Reply Quote 1
                                                              • rmdes
                                                                rmdes @rmdes last edited by

                                                                I got the entire bug all over again after reboot, lost mountpoint and lost index of backups, not sure what I can do to provide more context/log ?

                                                                0c9b0d8a-4310-4e10-aa40-7757ba6a54e4-image.png

                                                                nebulon 1 Reply Last reply Reply Quote 0
                                                                • nebulon
                                                                  nebulon Staff @rmdes last edited by

                                                                  @rmdes did you click the remount storage button or submitted the configure dialog for backup storage, given that the indicator is green on the top?

                                                                  rmdes 1 Reply Last reply Reply Quote 0
                                                                  • rmdes
                                                                    rmdes @nebulon last edited by

                                                                    @nebulon Correct, should have done the screenshot before remounting it,
                                                                    but the state the cloudron rebooted was no backup index and sshfs unmounted.

                                                                    avatar1024 1 Reply Last reply Reply Quote 0
                                                                    • avatar1024
                                                                      avatar1024 @rmdes last edited by

                                                                      @rmdes @nebulon Just to add to this. I was in touch with @girish through support, reporting a similar issue of the backup listing getting cleared if the storage is not mounted while the automated job of cleaning up the listing is run (the listing gets cleaned regularly to get rid of stale backup entries in the listing). I believe they are trying to find a solution to this.

                                                                      I proposed that either the automated cleaning job should first check that the backup location is accessible before cleaning up the entries in the db (preferred solution), or that the listing should not get cleaned automatically but only if triggered manually by pressing the "Cleanup Backup" button, in which case the user can ensure the volume is mounted before doing it (but this is less preferred as there is still a risk of deleting the entries by errors).

                                                                      girish 1 Reply Last reply Reply Quote 1
                                                                      • girish
                                                                        girish Staff @avatar1024 last edited by girish

                                                                        @avatar1024 Yes, thanks for the feedback. I am working though a fix now. For a start, the backup cleaner logic is not checking the mount status before cleaning up. I think if I fix that atleast the backup db entries won't disappear erroneously.

                                                                        edit: to be clear, I am implementing your "I proposed that either the automated cleaning job should first check that the backup location is accessible before cleaning up the entries in the db" solution.

                                                                        avatar1024 1 Reply Last reply Reply Quote 3
                                                                        • avatar1024
                                                                          avatar1024 @girish last edited by

                                                                          @girish Sounds great, thanks Girish.

                                                                          1 Reply Last reply Reply Quote 0
                                                                          • First post
                                                                            Last post
                                                                          Powered by NodeBB