Cloudron forum and GIT offline for long time?
Nothing much really, just upgraded to Ubuntu 22 on all our servers
- DigitalOcean takes 2 hours to snapshot our droplet. It then failed to snapshot saying "adapter error". They say it takes 1 min for every 1 GB of hard disk. This means 160GB is over 2 hours to snapshot a server.
- All upgrades from ubuntu 18 to 20 on DO fail. The whole VM gets hosed. Recovery console does not work. Various parts of the UI just error and show that DO shark. I guess all their new stuff is not well tested on Ubuntu 18.
- We restored from Cloudron backups which was fairly smooth.
- The reality is if it takes say 1 hour to backup, 1 hour to restore. Then, 2 hours things are down.
- The restore stalls at times at 0mbps. Have to investigate why.
- Our mail server backups are huge and cloudron restore is essentially "blocked" by mail restore. Only after that can this forum/git etc restore. It's a bit annoying, I would love to fix this but also low priority to fix since this is done so rarely.
@BrutalBirdie Adding to my generic answer with some more details which could help debugging.
I noticed this happening a lot more often with backup with a lot of small files when using rsync.
Example would be the Nextcloud preview folder with its excessive files.
Sometimes I have to check via root shell if the backup process is doing anything at all since there are no logs of what files or folder he is currently backing up.
I already read some users (in my mind), HOW TO DO THAT?!
TL;DR file descriptor
When a backup is running you will have a
box-TASKID.serviceon the system which has the PIDs of the sub-processes.
You can see the file descriptor of the
ls -lah /proc/$PID/fd/
Since the file descriptor change fast a normal
lsmight not catch an open file descriptor, so you can use
watch -d -n 0.1 ls -lah /proc/$PID/fd/