iDrive E2 Backups Failing More Frequently
-
Once in a while, I received a backup failed notification. It wasn't concerning because it was intermittent. Now I am receiving backup failed notifications more frequently than ever before (failure every few days). There have been no meaningful changes in my Cloudron. Here is the message from the Box Log (referencing s3copy as the source of the issue): "backup error: SlowDown SlowDown: Resource requested is unreadable, please reduce your request rate."
I am using iDrive E2 as an S3-compatible service. Only 50% of my purchased storage is utilized so there is plenty of space for backup objects.
Not every backup fails. Yesterday it failed backup up the mail app. A few days ago, the n8n backup failed. The only common thread appears to be a multipart upload of the object.
Any suggestions on how to remedy the situation?
-
Hello @crazybrad
SlowDown SlowDown: Resource requested is unreadable, please reduce your request rate.does indicate that the target is not keeping up with the requests send.
Could it be that your target iDrive E2 might be throttling the connection?There are similar reports out there and also quite recent:
-
@james Thank you for checking into this. Yesterday, I opened up a ticket with iDrive E2. They promised a 24 hour response. But seeing these issues appearing 1 year ago, 7 months ago, 4 months ago, etc. and multiple datacenters, it appears that they have a recurring issue that has not been adequately addressed. It could be multiple issues that result in "throttling", but in either case, it makes object store less reliable. (Note: I updated the topic to solicit more feedback from other Cloudronites.)
-
Does anyone have a recommendation on affordable, high-quality S3 compatible object storage? It seems that iDrive E2 is becoming less reliable for storing Cloudron backups. For comparison, iDrive E2 charges the equivalent of $.008/GB per month [$99 for 1TB per year], which is roughly one-third the cost of Amazon S3.
-
@crazybrad It might depend on external factors...
Where exactly are you moving data from/to? Can you share also info's on hardware you're running on?
In one month of use, I had a problem with iDrive e2—a 500 error from their Frankfurt zone. They never responded to my ticket, not even once. (Please read this post).
I tried (expensive) Cloudflare for a bit, but ended up going back to iDrive e2—right now, it’s the sweet spot between quality, speed, and cost.
It all started because I was using a Hetzner Box with an SSHFS mount, but the backup recovery failed due to corruption. Maintaining Hetzner this way felt way too risky.
-
@crazybrad Ok, and what about backup size? Also, did you already tried to switch iDrive e2 data center?
-
Does anyone have a recommendation on affordable, high-quality S3 compatible object storage? It seems that iDrive E2 is becoming less reliable for storing Cloudron backups. For comparison, iDrive E2 charges the equivalent of $.008/GB per month [$99 for 1TB per year], which is roughly one-third the cost of Amazon S3.
@crazybrad said in iDrive E2 Backups Failing More Frequently:
Does anyone have a recommendation on affordable, high-quality S3 compatible object storage? It seems that iDrive E2 is becoming less reliable for storing Cloudron backups. For comparison, iDrive E2 charges the equivalent of $.008/GB per month [$99 for 1TB per year], which is roughly one-third the cost of Amazon S3.
I use Scaleway Object Storage to store my Mastodon and Peertube data. Haven't tried using it for backups (as I just use Hetzner Storage Boxes for that) but haven't noticed any issues.
Pricing here https://www.scaleway.com/en/pricing/storage/ (a bit more expensive that iDrive E2, but perhaps you get what you pay for?
-
@p44 I haven't tried switching datacenters as I believe you and others have experienced this problem in Miami (and perhaps other locations as well). I am still waiting for a response from iDrive E2, and the lack thereof is also concerning. So rather than trying possible fixes, I'd prefer to get an explanation from iDrive E2 and if they suggest moving to another datacenter due to capacity or networking issues, then it would be worthwhile trying this.
This (early) morning I experienced another failed backup, so the issue is not getting any better.
@jdaviescoates Appreciate your recommendation about Scaleway. I will definitely check them out and consider the alternative.
@james Question: I just saw the advanced backup features link: more memory and multiparts. Given the problems I have been having, would you suggest any changes to the defaults?
And on a completely differnet note, do you think there is any merit in enhancing the backup portion of Cloudron to allow for a secondary backup location? In case primary failed, secondary would be tried instead? I get concerned that the one time I will really need my Cloudron backup, some object storage issue will prevent the backup from happening and then I am really in trouble. In my experience, even Amazon S3 can have issues. No one is infallible. -
@p44 I haven't tried switching datacenters as I believe you and others have experienced this problem in Miami (and perhaps other locations as well). I am still waiting for a response from iDrive E2, and the lack thereof is also concerning. So rather than trying possible fixes, I'd prefer to get an explanation from iDrive E2 and if they suggest moving to another datacenter due to capacity or networking issues, then it would be worthwhile trying this.
This (early) morning I experienced another failed backup, so the issue is not getting any better.
@jdaviescoates Appreciate your recommendation about Scaleway. I will definitely check them out and consider the alternative.
@james Question: I just saw the advanced backup features link: more memory and multiparts. Given the problems I have been having, would you suggest any changes to the defaults?
And on a completely differnet note, do you think there is any merit in enhancing the backup portion of Cloudron to allow for a secondary backup location? In case primary failed, secondary would be tried instead? I get concerned that the one time I will really need my Cloudron backup, some object storage issue will prevent the backup from happening and then I am really in trouble. In my experience, even Amazon S3 can have issues. No one is infallible.@crazybrad If you have a second Cloudron node at SSDnodes you can be your own backup with Minio with more app capacity, etc.
