Cloudron forum and GIT offline for long time?

[imc67]

Hi @girish & @nebulon I just experienced a relatively long being offline of at least Cloudron Forum and GIT?

Any experiences to share?

[girish]

Nothing much really, just upgraded to Ubuntu 22 on all our servers

Experiences:

• DigitalOcean takes 2 hours to snapshot our droplet. It then failed to snapshot saying "adapter error". They say it takes 1 min for every 1 GB of hard disk. This means 160GB is over 2 hours to snapshot a server.
• All upgrades from ubuntu 18 to 20 on DO fail. The whole VM gets hosed. Recovery console does not work. Various parts of the UI just error and show that DO shark. I guess all their new stuff is not well tested on Ubuntu 18.
• We restored from Cloudron backups which was fairly smooth.
  • The reality is if it takes say 1 hour to backup, 1 hour to restore. Then, 2 hours things are down.

Possible bugs:

• The restore stalls at times at 0mbps. Have to investigate why.
• Our mail server backups are huge and cloudron restore is essentially "blocked" by mail restore. Only after that can this forum/git etc restore. It's a bit annoying, I would love to fix this but also low priority to fix since this is done so rarely.

[BrutalBirdie]

@girish said in Cloudron forum and GIT offline for long time?:

The restore stalls at times at 0mbps. Have to investigate why.

Same with backups some times.

[BrutalBirdie]

@BrutalBirdie Adding to my generic answer with some more details which could help debugging.

I noticed this happening a lot more often with backup with a lot of small files when using rsync.

Example would be the Nextcloud preview folder with its excessive files.

Sometimes I have to check via root shell if the backup process is doing anything at all since there are no logs of what files or folder he is currently backing up.

I already read some users (in my mind), HOW TO DO THAT?!

TL;DR file descriptor

---

When a backup is running you will have a box-TASKID.service on the system which has the PIDs of the sub-processes.

You can see the file descriptor of the /bin/cp with:

ls -lah /proc/$PID/fd/

Since the file descriptor change fast a normal ls might not catch an open file descriptor, so you can use watch.

watch -d -n 0.1 ls -lah /proc/$PID/fd/

[robi]

Some tar - streaming would help with all the small files.