Cloudron Forum

My backup failure issues continue. I wanted to share another Cloudron topic about Minio backup failures and multipart. It got me thinking that perhaps my problem with iDrive E2 is the number of simultaneous connections being initiated as a result of the default Cloudron part size (10MB). I decided to increase the multipart size from 10MB to 64MB and see what happens. In looking at the size of files backed up, for me they range from a few kB to 20GB+ (mail). @joseph Two thoughts: First: should the Cloudron default be increased? Second: Future enhancement... perhaps 2 part size parameters could be defined: small and jumbo. For small tar.gz files, 10MB would be ideal. For jumbo tar.gz files, even 64MB is undersized. But the backup process would be tuned to the task at hand and not cause failures due to too many connections.

@joseph I agree. I looked at some of the implementation requirements and it takes effort. With each Cloudron app update and server reboot, it requires a process to "turn on" the server. That involves presenting several keys to unlock the server. For large companies with dev ops resources, this is doable. But for smaller companies, it is too much effort with a huge risk of bringing down applications if you miss a secrets server reboot.

@robi This looks like a great project and something worthy of being added to Cloudron. I like how routes can easily be added. Looks like I might have to learn me some Flask:)

@robi Good point about different regions at SSDNodes. If you have spare disk on the 2nd Cloudron, then the cost is $0. Something to consider...

@andreasdueren I'll take a look. I use Postmark but the concept should be similar.

@andreasdueren Awesome. Will take a look when I get the chance. Out of curiosity, did you enable any of the features for Paddle integration? It seems that is the way to introduce sending limits.

@girish I know this thread is stale, but I was wondering if the lack on interest in Vault is due to complexity of Vault and Hashicorp's licensing change.

Recently, I have been investigating other secrets management platforms and perhaps Infisical is better suited to our community. Infisical Packaging would require PostgreSQL, Redis, Email, and Auth, all of which is natively supported in Cloudron. Thoughts?

@robi That's a really interesting idea that I hadn't thought about. The question becomes what is the "all in" cost of storage. BackBlaze at $6/TB per month is much cheaper than I have seen from any VPS provider. Of course iDrive E2 was even cheaper, but not as reliable as I would like.

That being said, it might be better to have one server at SSDNodes and another say at Netcup or Hetzer to have geographic diversity (different data centers)... until I find out that all are using the same one in Northern Virginia any way(:

For those interested, just found this thread on using Minio as a backup location

@p44 I haven't tried switching datacenters as I believe you and others have experienced this problem in Miami (and perhaps other locations as well). I am still waiting for a response from iDrive E2, and the lack thereof is also concerning. So rather than trying possible fixes, I'd prefer to get an explanation from iDrive E2 and if they suggest moving to another datacenter due to capacity or networking issues, then it would be worthwhile trying this.

This (early) morning I experienced another failed backup, so the issue is not getting any better.

@jdaviescoates Appreciate your recommendation about Scaleway. I will definitely check them out and consider the alternative.

@james Question: I just saw the advanced backup features link: more memory and multiparts. Given the problems I have been having, would you suggest any changes to the defaults?
And on a completely differnet note, do you think there is any merit in enhancing the backup portion of Cloudron to allow for a secondary backup location? In case primary failed, secondary would be tried instead? I get concerned that the one time I will really need my Cloudron backup, some object storage issue will prevent the backup from happening and then I am really in trouble. In my experience, even Amazon S3 can have issues. No one is infallible.

@p44 I am using a VPS provided by SSDNodes (US provider). It is a large server (12CPU, 48GB RAM, 1TB NVMe) with plenty of capacity to spare. So it is moving from one US datacenter to another.

@mbarria I would start by disabling all site plugins and seeing if the memory leak stops. Then bring them back one by one. I can't explain why the container limit is not enforced, but it could also be that the WP site causes problems with MySQL. Lastly, how much RAM do you have on the Cloudron server and if you disable the WP site, how are the remaining apps functioning relative to service limits?

Does anyone have a recommendation on affordable, high-quality S3 compatible object storage? It seems that iDrive E2 is becoming less reliable for storing Cloudron backups. For comparison, iDrive E2 charges the equivalent of $.008/GB per month [$99 for 1TB per year], which is roughly one-third the cost of Amazon S3.

@james Thank you for checking into this. Yesterday, I opened up a ticket with iDrive E2. They promised a 24 hour response. But seeing these issues appearing 1 year ago, 7 months ago, 4 months ago, etc. and multiple datacenters, it appears that they have a recurring issue that has not been adequately addressed. It could be multiple issues that result in "throttling", but in either case, it makes object store less reliable. (Note: I updated the topic to solicit more feedback from other Cloudronites.)

Once in a while, I received a backup failed notification. It wasn't concerning because it was intermittent. Now I am receiving backup failed notifications more frequently than ever before (failure every few days). There have been no meaningful changes in my Cloudron. Here is the message from the Box Log (referencing s3copy as the source of the issue): "backup error: SlowDown SlowDown: Resource requested is unreadable, please reduce your request rate."

I am using iDrive E2 as an S3-compatible service. Only 50% of my purchased storage is utilized so there is plenty of space for backup objects.

Not every backup fails. Yesterday it failed backup up the mail app. A few days ago, the n8n backup failed. The only common thread appears to be a multipart upload of the object.

Any suggestions on how to remedy the situation?

I have been playing with packaging Keila. Still needs some more work, though. I am particularly interested in the ability to set sending limits (tied to Paddle plans).

@alex-a-soto Great idea. I'm interested. Let's see if anyone else has a similar interest. In either case, I will reach out as you suggested.

@girish I had the same problem with 2 apps (one recently automatically upgraded[n8n] and another installed now [archivebox] and both reported "Starting" in the dashboard despite working URLs (status really = "Running'). systemctl restart box cleared up the dashboard. But in checking box.log, I can see where about 2 days ago, box:apphealthmonitor app health {running, stopped, unresponsive} stopped running. I did not see an error, just nothing in the log. After systemctl restart box, the entry began appearing in box.log. The only unusual aspect of my Cloudron is that I have been experimenting with a privately hosted app. FYI, that app was "Running". Hope this is helpful in figuring out what is going on.

@james @vladimir.d Welcome to the best club in town:) And James, I am also a big Ansible fan. Perhaps we can collaborate/commiserate together.

@BrutalBirdie @jdaviescoates Thanks for reminding me about those little details:)

@leemuljadi I would use the Cloudron backup. Here is the process I followed successfully:

https://docs.cloudron.io/backups/#move-cloudron-to-another-server

As always, taking a provider snapshot of your old VPS is a good safety measure before starting the migration. But the Cloudron process worked like a charm.