Backup failure retry wait times should be shortened from 4 hours
-
@d19dotca in 5.5, we have removed the retry logic entirely. If the backup fails , it gives a notification and only backs up next time based on the schedule. The retry logic doesn't work well because then it might conflict with work hours again and it was the whole reason to make the backup schedule configurable in the first place. I think for general network errors there is already separate retry logic in place independent of this. For any other convoluted errors like mount point not found, the retry wont help.
-
@girish That's good to hear, but if that's true then I think there may be just a small defect because the notification I receive in Cloudron when it fails still has the line about retrying in 4 hours. Not the email, but the actual notification inside of Cloudron.
-
@d19dotca Ah, you are right. I forgot to fix that!
There are some more changes to the backup system (just putting it here, since it's not obvious):
- The backup is now run with a nice of 15. This makes sure that it gets low priority if the Cloudron is doing other things.
- It's run with a configurable memory limit. This memory limit is in Advanced -> Settings. This is useful if you want to do faster upload (by increasing concurrency values). This necessarily means you have to give the task more memory.
- There is currently a timeout of 12 hours for the task. If people are hitting this limit, I will bump this up.
-
@girish This needs to be added to the docs too on the concurrency limits and such. Just went to reference it to set appropriate values and didn’t find it. For what it’s worth, I tried with the defaults (10 concurrency) on all three of those settings, and eventually bumped it to 100. But I didn’t notice any real difference. I guess those settings apply more to remote systems instead of ext4 mounted disks?
-
@girish said in Backup failure retry wait times should be shortened from 4 hours:
by increasing concurrency values
Where can I find these settings? I only see the memory limit setting (using S3)
-
The backup changes worked out very well for us!!!
Cloudron #1 (5.4.1) was 35 minutes for about 14GB (tgz) and now (5.5) 3 minutes with highest concurrency limits / memory in rsync to external Minio.
Cloudron #2 (5.4.1) was about 1 hour for about 38GB (tgz) and now (5.5) 36 minutes with highest concurrency limits / memory in rsync to external Minio.
Thanks @girish for this wonderfull and welcome update (also for being able to set own times and intervals)!
-
@d19dotca I have added some docs now.
https://cloudron.io/documentation/backups/#schedule has info on the timeout and nice.
https://cloudron.io/documentation/backups/#concurrency-settings on the concurrency settings.