Cloudron sends health check notifications for each app ("App xxx is down / back online"). In normal use, this is quite useful, but when working on my Cloudron machine (rebooting it, or whatever), I then get a flood of emails for each application, so my inbox looks like:

Aaa is back up
Bbb is back up
Ccc is back up
Aaa is down
Bbb is down
Ccc is down

Suggested improvements:

• Perhaps it might make sense to delay the activation of health check alerts until (host or app) uptime is over a certain value?
• Consider merging notifications for multiple applications into one (or at least, one mail)

Large filenames cause problems, especially with blob-based storage (e.g. S3-compatible services have a key limit of 1024). This gives problems quite easily with hierarchical storage like syncthing where one might have a set of nested folders that quite easily hit that key limit, especially combined with encryption.

Disabling encryption altogether is not desirable in many cases, but it could be worthwhile to allow disabling filename encryption, in some cases.

(As a longer term project, it may also be worth considering ways to circumvent this limit altogether.)

See https://forum.cloudron.io/topic/3050/backup-feedback-minio/ for the initial discussion leading to this request.

Hi,

I noticed that beam.smp is using a lot of CPU inside my cloudron instance. I tracked this down to belonging to onlyoffice, and doing some further digging, I found this post that might be interesting.

tl;dr:

When I set the number of scheduler threads to 1 the CPU usage dropped from 23 % to 3 %.

Setting RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+S 1:1" apparently does the trick, and there's a linked issue on the onlyoffice github that appears to do exactly that as a part of their Dockerfile.

I'm unsure how to test this, but perhaps this may be the same issue / fix for Cloudron's case, too?

I haven't seen the high cpu usage as such

Might be that this is due to the differences in HW config we're running on? As I understand it, this will "scale up" based on the number of cores you have. I'm running on a Threadripper, which has rather a few of them.

Trying to install the fixed version now, observations:

• Before install, beam.smp is taking up a steady 6% CPU
• After upgrading (via cloudron update --app xxx --appstore-id com.onlyoffice.coudronapp@1.6.1), CPU use is more or less nonexistent: I no longer see it in the top consumers at all

So I think this looks successful

I will move this key length limitation as a feature request. I don't know of a way around it without either a) maintaining an offline index of file names or b) disabling file name encryption altogether.

Yes, I think that there's no other solution, at least, not that I've been able to come up with.

Is disabling filename encrpytion acceptable to you? If so, that is atleast easier to add than a.

That would for sure be more ideal than it not working, so I can make a feature request for it

In terms of the "just work" factor, it would be nicer to have this just handled transparently ('a') though, but I can understand it isn't at all straightforward. Might be worth studying some different backup tools to see how this is handled, and sticking this into the list as a longer term project.

For 2, this seems like a Cloudron bug. How big is your data roughly?

Pretty big then, much smaller now. At the time I was testing the limits to see if I could break things, so I had around 400G stored I think (most of it in a single syncthing instance), don't recall the exact amount.

For the last release, I did some basic performance tests to get the backup speed to be at par with something like rclone. I did not test sshfs specifically but I am curious how long it takes to copy this amount of data using normal tool in your setup?

My guess is that the problem is inherent to sshfs, and is not something specific to Cloudron (sshfs not being fast is pretty widely known/recognized). The only real reason I am reporting this is to perhaps suggest noting that it does not scale well in the docs, or something like that, because I don't think this gives a particularly good impression as-is.

sshfs is very sensitive to latency, so any cases where you run it to a remote server on a different network, the slowdown will be very noticable.

Using a tool more suited to this style of operation (like, say, rsync) will give better results, but it will still not scale well for large files (or large datasets, over say a few hundred GB of small files).

In the other thread, I saw that you were using encrypted backups. Is this the case when you tested this? Encryption requires a decent CPU as well, so a simple cp is not a fair comparison. I wonder if you can enable encryption on the remote disk somehow and not rely on Cloudron encryption?

Correct, encryption was enabled, but it wasn't really a factor. CPU was more or less idle while this was going on. Remote disk encryption isn't really an option in this case because I don't have physical control over the remote host, so I want to keep the data hidden

After struggling with sshfs-based backups, I decided to try object-based storage. I ran into a few problems.

1. Key length limitations (w/ rsync)

As the documentation notes, especially when encrypted, file name length limits can be a problem. I hit this sooner than expected, and wasn't able to successfully back up.

2. max-parts limitation

Switching from rsync to tarball-based backups unfortunately didn't work at some point either:

Error uploading snapshot/app_xxx.tar.gz.enc. Message: Argument max-parts must be an integer between 0 and 2147483647 HTTP Code: InvalidArgument

Not certain, but my guess is that my backup in this case was too large to store?

Cloudron sends health check notifications for each app ("App xxx is down / back online"). In normal use, this is quite useful, but when working on my Cloudron machine (rebooting it, or whatever), I then get a flood of emails for each application, so my inbox looks like:

Aaa is back up
Bbb is back up
Ccc is back up
Aaa is down
Bbb is down
Ccc is down

Suggested improvements:

• Perhaps it might make sense to delay the activation of health check alerts until (host or app) uptime is over a certain value?
• Consider merging notifications for multiple applications into one (or at least, one mail)

I haven't seen the high cpu usage as such

Might be that this is due to the differences in HW config we're running on? As I understand it, this will "scale up" based on the number of cores you have. I'm running on a Threadripper, which has rather a few of them.

Trying to install the fixed version now, observations:

• Before install, beam.smp is taking up a steady 6% CPU
• After upgrading (via cloudron update --app xxx --appstore-id com.onlyoffice.coudronapp@1.6.1), CPU use is more or less nonexistent: I no longer see it in the top consumers at all

So I think this looks successful

Hi,

I noticed that beam.smp is using a lot of CPU inside my cloudron instance. I tracked this down to belonging to onlyoffice, and doing some further digging, I found this post that might be interesting.

tl;dr:

When I set the number of scheduler threads to 1 the CPU usage dropped from 23 % to 3 %.

Setting RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+S 1:1" apparently does the trick, and there's a linked issue on the onlyoffice github that appears to do exactly that as a part of their Dockerfile.

I'm unsure how to test this, but perhaps this may be the same issue / fix for Cloudron's case, too?

Yes indeed, a title was also missing. I assumed it was the ID when I went doc searching, but of course title makes more sense. Thanks

Those docs were written from a time when we didn't have custom apps and you had to publish apps in the app store.

Ahh, ok! I had guessed that there was some extra validation step before store publishing indeed.

BTW, where did you see that error "is not responding" ? Notification in the UI or CLI?

The notification was a "regular" admin UI notification, sent some time after it had been running, due to a healthcheck failure caused by a crash I think. I can try dig out the logs if that would be helpful.

Some fields are marked as required in the manifest documentation, but there appears to be no enforcement of this during install/update.

I have encountered this with at least the 'id' field, but I wouldn't be surprised if the same was true of other fields.

This manifests in some UI-side bugs (and presumably, other potential problems behind the scenes), such as this one:

The application undefined installed at xxxx is not responding.

Large filenames cause problems, especially with blob-based storage (e.g. S3-compatible services have a key limit of 1024). This gives problems quite easily with hierarchical storage like syncthing where one might have a set of nested folders that quite easily hit that key limit, especially combined with encryption.

Disabling encryption altogether is not desirable in many cases, but it could be worthwhile to allow disabling filename encryption, in some cases.

(As a longer term project, it may also be worth considering ways to circumvent this limit altogether.)

See https://forum.cloudron.io/topic/3050/backup-feedback-minio/ for the initial discussion leading to this request.

Yes, I think repair would work just fine. I'll give that a shot next time I run into this.

The upgrade process makes sense, as you say, working on custom applications is a bit of a special usecase

I will move this key length limitation as a feature request. I don't know of a way around it without either a) maintaining an offline index of file names or b) disabling file name encryption altogether.

Yes, I think that there's no other solution, at least, not that I've been able to come up with.

Is disabling filename encrpytion acceptable to you? If so, that is atleast easier to add than a.

That would for sure be more ideal than it not working, so I can make a feature request for it

In terms of the "just work" factor, it would be nicer to have this just handled transparently ('a') though, but I can understand it isn't at all straightforward. Might be worth studying some different backup tools to see how this is handled, and sticking this into the list as a longer term project.

For 2, this seems like a Cloudron bug. How big is your data roughly?

Pretty big then, much smaller now. At the time I was testing the limits to see if I could break things, so I had around 400G stored I think (most of it in a single syncthing instance), don't recall the exact amount.

For the last release, I did some basic performance tests to get the backup speed to be at par with something like rclone. I did not test sshfs specifically but I am curious how long it takes to copy this amount of data using normal tool in your setup?

My guess is that the problem is inherent to sshfs, and is not something specific to Cloudron (sshfs not being fast is pretty widely known/recognized). The only real reason I am reporting this is to perhaps suggest noting that it does not scale well in the docs, or something like that, because I don't think this gives a particularly good impression as-is.

sshfs is very sensitive to latency, so any cases where you run it to a remote server on a different network, the slowdown will be very noticable.

Using a tool more suited to this style of operation (like, say, rsync) will give better results, but it will still not scale well for large files (or large datasets, over say a few hundred GB of small files).

In the other thread, I saw that you were using encrypted backups. Is this the case when you tested this? Encryption requires a decent CPU as well, so a simple cp is not a fair comparison. I wonder if you can enable encryption on the remote disk somehow and not rely on Cloudron encryption?

Correct, encryption was enabled, but it wasn't really a factor. CPU was more or less idle while this was going on. Remote disk encryption isn't really an option in this case because I don't have physical control over the remote host, so I want to keep the data hidden