iDrive E2 Backups Failing More Frequently
-
Once in a while, I received a backup failed notification. It wasn't concerning because it was intermittent. Now I am receiving backup failed notifications more frequently than ever before (failure every few days). There have been no meaningful changes in my Cloudron. Here is the message from the Box Log (referencing s3copy as the source of the issue): "backup error: SlowDown SlowDown: Resource requested is unreadable, please reduce your request rate."
I am using iDrive E2 as an S3-compatible service. Only 50% of my purchased storage is utilized so there is plenty of space for backup objects.
Not every backup fails. Yesterday it failed backup up the mail app. A few days ago, the n8n backup failed. The only common thread appears to be a multipart upload of the object.
Any suggestions on how to remedy the situation?
-
Hello @crazybrad
SlowDown SlowDown: Resource requested is unreadable, please reduce your request rate.does indicate that the target is not keeping up with the requests send.
Could it be that your target iDrive E2 might be throttling the connection?There are similar reports out there and also quite recent:
-
@james Thank you for checking into this. Yesterday, I opened up a ticket with iDrive E2. They promised a 24 hour response. But seeing these issues appearing 1 year ago, 7 months ago, 4 months ago, etc. and multiple datacenters, it appears that they have a recurring issue that has not been adequately addressed. It could be multiple issues that result in "throttling", but in either case, it makes object store less reliable. (Note: I updated the topic to solicit more feedback from other Cloudronites.)
-
Does anyone have a recommendation on affordable, high-quality S3 compatible object storage? It seems that iDrive E2 is becoming less reliable for storing Cloudron backups. For comparison, iDrive E2 charges the equivalent of $.008/GB per month [$99 for 1TB per year], which is roughly one-third the cost of Amazon S3.
-
@crazybrad It might depend on external factors...
Where exactly are you moving data from/to? Can you share also info's on hardware you're running on?
In one month of use, I had a problem with iDrive e2—a 500 error from their Frankfurt zone. They never responded to my ticket, not even once. (Please read this post).
I tried (expensive) Cloudflare for a bit, but ended up going back to iDrive e2—right now, it’s the sweet spot between quality, speed, and cost.
It all started because I was using a Hetzner Box with an SSHFS mount, but the backup recovery failed due to corruption. Maintaining Hetzner this way felt way too risky.
-
@crazybrad Ok, and what about backup size? Also, did you already tried to switch iDrive e2 data center?
-
Does anyone have a recommendation on affordable, high-quality S3 compatible object storage? It seems that iDrive E2 is becoming less reliable for storing Cloudron backups. For comparison, iDrive E2 charges the equivalent of $.008/GB per month [$99 for 1TB per year], which is roughly one-third the cost of Amazon S3.
@crazybrad said in iDrive E2 Backups Failing More Frequently:
Does anyone have a recommendation on affordable, high-quality S3 compatible object storage? It seems that iDrive E2 is becoming less reliable for storing Cloudron backups. For comparison, iDrive E2 charges the equivalent of $.008/GB per month [$99 for 1TB per year], which is roughly one-third the cost of Amazon S3.
I use Scaleway Object Storage to store my Mastodon and Peertube data. Haven't tried using it for backups (as I just use Hetzner Storage Boxes for that) but haven't noticed any issues.
Pricing here https://www.scaleway.com/en/pricing/storage/ (a bit more expensive that iDrive E2, but perhaps you get what you pay for?
-
@p44 I haven't tried switching datacenters as I believe you and others have experienced this problem in Miami (and perhaps other locations as well). I am still waiting for a response from iDrive E2, and the lack thereof is also concerning. So rather than trying possible fixes, I'd prefer to get an explanation from iDrive E2 and if they suggest moving to another datacenter due to capacity or networking issues, then it would be worthwhile trying this.
This (early) morning I experienced another failed backup, so the issue is not getting any better.
@jdaviescoates Appreciate your recommendation about Scaleway. I will definitely check them out and consider the alternative.
@james Question: I just saw the advanced backup features link: more memory and multiparts. Given the problems I have been having, would you suggest any changes to the defaults?
And on a completely differnet note, do you think there is any merit in enhancing the backup portion of Cloudron to allow for a secondary backup location? In case primary failed, secondary would be tried instead? I get concerned that the one time I will really need my Cloudron backup, some object storage issue will prevent the backup from happening and then I am really in trouble. In my experience, even Amazon S3 can have issues. No one is infallible. -
@p44 I haven't tried switching datacenters as I believe you and others have experienced this problem in Miami (and perhaps other locations as well). I am still waiting for a response from iDrive E2, and the lack thereof is also concerning. So rather than trying possible fixes, I'd prefer to get an explanation from iDrive E2 and if they suggest moving to another datacenter due to capacity or networking issues, then it would be worthwhile trying this.
This (early) morning I experienced another failed backup, so the issue is not getting any better.
@jdaviescoates Appreciate your recommendation about Scaleway. I will definitely check them out and consider the alternative.
@james Question: I just saw the advanced backup features link: more memory and multiparts. Given the problems I have been having, would you suggest any changes to the defaults?
And on a completely differnet note, do you think there is any merit in enhancing the backup portion of Cloudron to allow for a secondary backup location? In case primary failed, secondary would be tried instead? I get concerned that the one time I will really need my Cloudron backup, some object storage issue will prevent the backup from happening and then I am really in trouble. In my experience, even Amazon S3 can have issues. No one is infallible.@crazybrad If you have a second Cloudron node at SSDnodes you can be your own backup with Minio with more app capacity, etc.
-
@robi That's a really interesting idea that I hadn't thought about. The question becomes what is the "all in" cost of storage. BackBlaze at $6/TB per month is much cheaper than I have seen from any VPS provider. Of course iDrive E2 was even cheaper, but not as reliable as I would like.
That being said, it might be better to have one server at SSDNodes and another say at Netcup or Hetzer to have geographic diversity (different data centers)... until I find out that all are using the same one in Northern Virginia any way(:
For those interested, just found this thread on using Minio as a backup location
-
@robi That's a really interesting idea that I hadn't thought about. The question becomes what is the "all in" cost of storage. BackBlaze at $6/TB per month is much cheaper than I have seen from any VPS provider. Of course iDrive E2 was even cheaper, but not as reliable as I would like.
That being said, it might be better to have one server at SSDNodes and another say at Netcup or Hetzer to have geographic diversity (different data centers)... until I find out that all are using the same one in Northern Virginia any way(:
For those interested, just found this thread on using Minio as a backup location
@crazybrad You can deploy in different regions at SSDNodes too. They don't lie about those, with traceroute and all.
This is what I did and is less costly considering you have a whole other Cloudron on tap, which just happens to run Minio too.
Both my locations have it and backup to each other.
-
J joseph has marked this topic as solved
-
IDrive e2 has been very reliable for me for the cost, however I too ran into that issue a few months back around the “slow down” error where basically they were rate limiting. I suspect it had to do with noisy neighbours ruining it and triggering a rate limiting on the cluster of nodes being backed up to, but that’s just a guess on my part.
What I had to do was delete the region hostname and re-enable it, which created my backup buckets somewhere else and allowed me to proceed without issue. The unfortunate part of this though was that it deletes all the files of course in the prior region, so what I did was first download a few important backups from it to my local disk and reloaded afterwards.
IDrive support team had apologized for the issues and provided me with a couple months free service credit, so that was helpful although the cost is so cheap that it didn’t exactly pay for my time at all, haha.
Just an option for those who may run into that rate limiting issue if it persists for a while. I’d recommend opening a support case and if the issue isn’t resolved quickly enough and you’re able to delete the region, delete it (but first download anything critical from it) and start again and tell support about what you did and they should credit your account for the troubles (you may need to request that though if it isn’t automatic).
-
My backup failure issues continue. I wanted to share another Cloudron topic about Minio backup failures and multipart. It got me thinking that perhaps my problem with iDrive E2 is the number of simultaneous connections being initiated as a result of the default Cloudron part size (10MB). I decided to increase the multipart size from 10MB to 64MB and see what happens. In looking at the size of files backed up, for me they range from a few kB to 20GB+ (mail). @joseph Two thoughts: First: should the Cloudron default be increased? Second: Future enhancement... perhaps 2 part size parameters could be defined: small and jumbo. For small tar.gz files, 10MB would be ideal. For jumbo tar.gz files, even 64MB is undersized. But the backup process would be tuned to the task at hand and not cause failures due to too many connections.
-
IDrive e2 has been very reliable for me for the cost, however I too ran into that issue a few months back around the “slow down” error where basically they were rate limiting. I suspect it had to do with noisy neighbours ruining it and triggering a rate limiting on the cluster of nodes being backed up to, but that’s just a guess on my part.
What I had to do was delete the region hostname and re-enable it, which created my backup buckets somewhere else and allowed me to proceed without issue. The unfortunate part of this though was that it deletes all the files of course in the prior region, so what I did was first download a few important backups from it to my local disk and reloaded afterwards.
IDrive support team had apologized for the issues and provided me with a couple months free service credit, so that was helpful although the cost is so cheap that it didn’t exactly pay for my time at all, haha.
Just an option for those who may run into that rate limiting issue if it persists for a while. I’d recommend opening a support case and if the issue isn’t resolved quickly enough and you’re able to delete the region, delete it (but first download anything critical from it) and start again and tell support about what you did and they should credit your account for the troubles (you may need to request that though if it isn’t automatic).
@d19dotca said in iDrive E2 Backups Failing More Frequently:
What I had to do was delete the region hostname and re-enable it, which created my backup buckets somewhere else and allowed me to proceed without issue. The unfortunate part of this though was that it deletes all the files of course in the prior region, so what I did was first download a few important backups from it to my local disk and reloaded afterwards.
I had the same rate-limiting issue with IDrive E2, but with a twist: my bucket was originally blazing fast in one region. After I deleted and recreated it (like you did), it became painfully slow—the exact opposite of your experience!
I suspect their ‘region reassignment’ doesn’t guarantee consistent performance, maybe due to uneven server loads or hidden throttling.
Moral of the story? With IDrive E2, if your bucket works fine, don’t touch it—even if they tempt you with ‘better regions.’
