Long running du process keeps my backup disk spinning

[nebulon]

Currently there is no way to exclude drives from the stat gathering. We do run du to calculate the size each app and volume takes, this is not possible with df, however this only happens once a day https://git.cloudron.io/cloudron/box/-/blob/master/src/cron.js?ref_type=heads#L115 so not sure how long a du for all things on that disk takes. Or if other apps with data on it read/write to it on their own schedules.

[necrevistonnezr]

Thanks for the info. However, as this drive contains an rsynced backup, there are millions of small files causing it to run forever…

Maybe block du something like check for a NO_INDEX file in the root of the drive or something like that?
Or use duc or ncdu as faster alternatives?

[girish]

Maybe we can skip du for non-ssd devices with a high inode count ? @necrevistonnezr can you get the output of df -i /backup-path ? Also, I think lsblk -d -o name,rota should display 1 for your device (indicating hdd)

[necrevistonnezr]

Here we go:

df -i /media/WD4TB/CloudronBackup/
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/sdb1        233M  6,0M  227M    3% /media/WD4TB

and

lsblk -d -o name,rota
NAME    ROTA
loop0      0
loop1      0
loop2      0
loop3      0
loop4      0
loop5      0
loop6      0
loop7      0
loop8      0
loop9      0
loop10     0
loop11     0
sda        0
sdb        1
nvme0n1    0

[girish]

@necrevistonnezr nice, thank you.

I will put some code to skip du if more than 150M inodes (arbitrary) and HDD. actually, in your case only 6M inodes are used! That's a slow disk. Maybe we should base this on read speed instead.

How about hdparm -t /media/WD4TB ?

[necrevistonnezr]

Cool, thank you!

[girish]

@necrevistonnezr sorry, updated the post. I misread your output

[necrevistonnezr]

hdparm -t /media/WD4TB/

/media/WD4TB/:
 Timing buffered disk reads: read() failed: Is a directory
BLKFLSBUF failed: Inappropriate ioctl for device

[girish]

@nebulon I guess it needs the device name /dev/xx

[necrevistonnezr]

Yeah, I guessed the same after reading the debianforum wiki

Results after running it a couple of times:

sudo hdparm -t /dev/sdb1

/dev/sdb1:
 Timing buffered disk reads: 192 MB in  3.00 seconds =  63.92 MB/sec
 Timing buffered disk reads: 200 MB in  3.00 seconds =  66.58 MB/sec
 Timing buffered disk reads: 224 MB in  3.01 seconds =  74.44 MB/sec
 Timing buffered disk reads: 236 MB in  3.01 seconds =  78.31 MB/sec
 Timing buffered disk reads: 248 MB in  3.01 seconds =  82.33 MB/sec
 Timing buffered disk reads: 234 MB in  3.01 seconds =  77.69 MB/sec