Large filenames cause problems, especially with blob-based storage (e.g. S3-compatible services have a key limit of 1024). This gives problems quite easily with hierarchical storage like syncthing where one might have a set of nested folders that quite easily hit that key limit, especially combined with encryption.

Disabling encryption altogether is not desirable in many cases, but it could be worthwhile to allow disabling filename encryption, in some cases.

(As a longer term project, it may also be worth considering ways to circumvent this limit altogether.)

See https://forum.cloudron.io/topic/3050/backup-feedback-minio/ for the initial discussion leading to this request.

I will move this key length limitation as a feature request. I don't know of a way around it without either a) maintaining an offline index of file names or b) disabling file name encryption altogether.

Yes, I think that there's no other solution, at least, not that I've been able to come up with.

Is disabling filename encrpytion acceptable to you? If so, that is atleast easier to add than a.

That would for sure be more ideal than it not working, so I can make a feature request for it

In terms of the "just work" factor, it would be nicer to have this just handled transparently ('a') though, but I can understand it isn't at all straightforward. Might be worth studying some different backup tools to see how this is handled, and sticking this into the list as a longer term project.

For 2, this seems like a Cloudron bug. How big is your data roughly?

Pretty big then, much smaller now. At the time I was testing the limits to see if I could break things, so I had around 400G stored I think (most of it in a single syncthing instance), don't recall the exact amount.

For the last release, I did some basic performance tests to get the backup speed to be at par with something like rclone. I did not test sshfs specifically but I am curious how long it takes to copy this amount of data using normal tool in your setup?

My guess is that the problem is inherent to sshfs, and is not something specific to Cloudron (sshfs not being fast is pretty widely known/recognized). The only real reason I am reporting this is to perhaps suggest noting that it does not scale well in the docs, or something like that, because I don't think this gives a particularly good impression as-is.

sshfs is very sensitive to latency, so any cases where you run it to a remote server on a different network, the slowdown will be very noticable.

Using a tool more suited to this style of operation (like, say, rsync) will give better results, but it will still not scale well for large files (or large datasets, over say a few hundred GB of small files).

In the other thread, I saw that you were using encrypted backups. Is this the case when you tested this? Encryption requires a decent CPU as well, so a simple cp is not a fair comparison. I wonder if you can enable encryption on the remote disk somehow and not rely on Cloudron encryption?

Correct, encryption was enabled, but it wasn't really a factor. CPU was more or less idle while this was going on. Remote disk encryption isn't really an option in this case because I don't have physical control over the remote host, so I want to keep the data hidden

After struggling with sshfs-based backups, I decided to try object-based storage. I ran into a few problems.

1. Key length limitations (w/ rsync)

As the documentation notes, especially when encrypted, file name length limits can be a problem. I hit this sooner than expected, and wasn't able to successfully back up.

2. max-parts limitation

Switching from rsync to tarball-based backups unfortunately didn't work at some point either:

Error uploading snapshot/app_xxx.tar.gz.enc. Message: Argument max-parts must be an integer between 0 and 2147483647 HTTP Code: InvalidArgument

Not certain, but my guess is that my backup in this case was too large to store?

Yes indeed, a title was also missing. I assumed it was the ID when I went doc searching, but of course title makes more sense. Thanks

Those docs were written from a time when we didn't have custom apps and you had to publish apps in the app store.

Ahh, ok! I had guessed that there was some extra validation step before store publishing indeed.

BTW, where did you see that error "is not responding" ? Notification in the UI or CLI?

The notification was a "regular" admin UI notification, sent some time after it had been running, due to a healthcheck failure caused by a crash I think. I can try dig out the logs if that would be helpful.

Some fields are marked as required in the manifest documentation, but there appears to be no enforcement of this during install/update.

I have encountered this with at least the 'id' field, but I wouldn't be surprised if the same was true of other fields.

This manifests in some UI-side bugs (and presumably, other potential problems behind the scenes), such as this one:

The application undefined installed at xxxx is not responding.

Large filenames cause problems, especially with blob-based storage (e.g. S3-compatible services have a key limit of 1024). This gives problems quite easily with hierarchical storage like syncthing where one might have a set of nested folders that quite easily hit that key limit, especially combined with encryption.

Disabling encryption altogether is not desirable in many cases, but it could be worthwhile to allow disabling filename encryption, in some cases.

(As a longer term project, it may also be worth considering ways to circumvent this limit altogether.)

See https://forum.cloudron.io/topic/3050/backup-feedback-minio/ for the initial discussion leading to this request.

Yes, I think repair would work just fine. I'll give that a shot next time I run into this.

The upgrade process makes sense, as you say, working on custom applications is a bit of a special usecase

I will move this key length limitation as a feature request. I don't know of a way around it without either a) maintaining an offline index of file names or b) disabling file name encryption altogether.

Yes, I think that there's no other solution, at least, not that I've been able to come up with.

Is disabling filename encrpytion acceptable to you? If so, that is atleast easier to add than a.

That would for sure be more ideal than it not working, so I can make a feature request for it

In terms of the "just work" factor, it would be nicer to have this just handled transparently ('a') though, but I can understand it isn't at all straightforward. Might be worth studying some different backup tools to see how this is handled, and sticking this into the list as a longer term project.

For 2, this seems like a Cloudron bug. How big is your data roughly?

Pretty big then, much smaller now. At the time I was testing the limits to see if I could break things, so I had around 400G stored I think (most of it in a single syncthing instance), don't recall the exact amount.

For the last release, I did some basic performance tests to get the backup speed to be at par with something like rclone. I did not test sshfs specifically but I am curious how long it takes to copy this amount of data using normal tool in your setup?

My guess is that the problem is inherent to sshfs, and is not something specific to Cloudron (sshfs not being fast is pretty widely known/recognized). The only real reason I am reporting this is to perhaps suggest noting that it does not scale well in the docs, or something like that, because I don't think this gives a particularly good impression as-is.

sshfs is very sensitive to latency, so any cases where you run it to a remote server on a different network, the slowdown will be very noticable.

Using a tool more suited to this style of operation (like, say, rsync) will give better results, but it will still not scale well for large files (or large datasets, over say a few hundred GB of small files).

In the other thread, I saw that you were using encrypted backups. Is this the case when you tested this? Encryption requires a decent CPU as well, so a simple cp is not a fair comparison. I wonder if you can enable encryption on the remote disk somehow and not rely on Cloudron encryption?

Correct, encryption was enabled, but it wasn't really a factor. CPU was more or less idle while this was going on. Remote disk encryption isn't really an option in this case because I don't have physical control over the remote host, so I want to keep the data hidden

After struggling with sshfs-based backups, I decided to try object-based storage. I ran into a few problems.

1. Key length limitations (w/ rsync)

As the documentation notes, especially when encrypted, file name length limits can be a problem. I hit this sooner than expected, and wasn't able to successfully back up.

2. max-parts limitation

Switching from rsync to tarball-based backups unfortunately didn't work at some point either:

Error uploading snapshot/app_xxx.tar.gz.enc. Message: Argument max-parts must be an integer between 0 and 2147483647 HTTP Code: InvalidArgument

Not certain, but my guess is that my backup in this case was too large to store?

Some feedback on trying to use sshfs as a backup mechanism. Setup was: cloudron node connected to a remote server hosted in a different country using sshfs.

• Using the rsync method, this was unusably slow, likely due to smaller file sizes (~10 minutes just to process a G or two of data, despite having a decent internet connection on both ends).
• Using tarballs was a bit of a better experience, the backup did eventually complete, but this is not really a practical possibility given the size of data I was trying to back up at present (as I understand it, basically, each day would involve a full upload of those tarballs).

Not sure what can really be done about this from the Cloudron side without some pretty serious surgery, but thought that it was worth mentioning.