Backup uploads time out after 12 hours; can no longer manually adjust timeout to be longer.

[girish]

@shan the timeout is already 24 hours since many releases by now. Which Cloudron version are you on?

[shan]

@girish Sorry yeah it is 24 hours, however that's not enough for my uploads. My uploads take like 3 days usually. Is there a new way to manually change it?

[subven]

@shan 3 days is a loooooot ^^ I guess you have very limited upload speed? Switching from rsync to tgz helped me cutting the time my backup needs in half because of all the small files (wordpress, email...). For larger files that do not change a lot (like media formats), rsync could be way better. You can also exclude apps from automatic backups and do a manual backup from time to time.

Would really wish to have better incremental backups for Cloudron. Uploading ~150GB per day is pretty wasty. Duplicati takes 5 minutes per day to update a 300gb backup copy of my private stuff.

[shan]

@subven I am using tgz because rsync couldn't handle the char length of my encrypted file names, lol. Just need a way to manually adjust this upload time

[robi]

@shan Why not keep using the method you have with larger number?

[girish]

@shan said in Backup uploads time out after 12 hours; can no longer manually adjust timeout to be longer.:

@subven I am using tgz because rsync couldn't handle the char length of my encrypted file names, lol. Just need a way to manually adjust this upload time

Fix for that is coming In 7.3, there is a checkbox to disable filename encryption alone - https://docs.cloudron.io/backups/#filenames

For the moment, you can always bump the timeout here - https://git.cloudron.io/cloudron/box/-/blob/master/src/backups.js#L239 . That file would be /home/yellowtent/box/src/backups.js.

[marcusquinn]

@girish Prob is I can't update to 7.3 because the backup needed before takes over 12h

[girish]

@marcusquinn ah, catch-22 But you should be able to fix up the code in 7.2 accordingly as well and then click the update button.

[shan]

@robi setting the timeout manually now seems to not affect the actual timeout time like it used to

[shan]

@girish yeah that is what I was doing before but it no longer working

[girish]

@shan It shows an error in the eventlog / backups view that it times out? Any screenshot ? How much timeout did you increase it to?

[subven]

@marcusquinn you can maybe backup by exporting your server with a snapshot over your VPS provider? You can also skip the backup entirely

[marcusquinn]

Backup seems to be getting stuck on "Uploading mail snapshot", but I don't have that much email, so I suspect something else going on there. I do have VPS snapshots, so I think you're right on skipping the backup in this case. Ta.

[shan]

@girish increased it to 72 hours; does this file get regenerated perhaps? Looks like the timeout is back to 24 hours when I checked it. Maybe on Cloudron update?

[girish]

@shan yes, on a Cloudron update, the code will revert back.

I am surprised a backup takes that much time though. Have you debugged further as to why? How much data do you have and how long is the upload taking ? (You can get this info from the logs as well).

[CBCUN]

@girish I have also Backups around 600GB (with timeouts too)

[girish]

@CBCUN As in , it hits the same 24 hour timeout? Where are you backing up to?

[CBCUN]

@girish At Backblaze and Netcup

[girish]

@CBCUN Just wondering if I should bump up the timeout to more than 24 hours. Do you think this is some Cloudron upload issue or do you suspect it takes that long to upload from netcup to blackblaze?
Maybe netcup throttles speed after a point ?

I guess another idea is to make this timeout configurable.

[subven]

@CBCUN any decent hoster can upload with a speed of 10MB/s so for a 600GB backup it should take 16-18 hours. My Cloudron backup is ~80GB TGZ and it takes 35-45 minutes to backup from a Netcup RS to a Hetzner storage box. Either your source/target system is slow as hell or something else is bottlenecking. What method do you use? I found TGZ to be way faster because of the many many small files. If you have larger files, maybe RSYNC is the way to go. A good solution would be a (maybe self hosted) S3 Storage together with RSYNC. I use Minio for one of my backup instances and the backup from Cloudron is blazing fast as most of the work is done by the target machine.