Use `gdu` instead of `du` (was "Exlude du from backup check")

[nichu42]

Hey there,

Can I set Cloudron Backup to skip

shell: backuptask: du --dereference-args --summarize --block-size=1 --exclude=*.lock --exclude=dovecot.list.index.log.* /home/yellowtent/appsdata/c5d30482-9837-4fdb-8c21-04ef653d691c

?

It wastes 20 minutes on my Mastodon backup each night. And my S3 won't run out of space any time soon...

[nebulon]

Disabling this is not really possible or exposed as some setting. Once question though is, why does it take so long to just calculate the size of the folder to estimate the backup size?

[nichu42]

@nebulon I guess because my public Mastodon instance has hundreds of thousands of small files.

[necrevistonnezr]

GNU du is single-threaded leading to stalls on scanning hundreds of thousands of files.

Multi-threaded Alternatives:

• gdu (in repositories): Go-based, multi-threaded.
• diskus (Rust, GitHub) : Minimalist CLI total-size calculator.

Disk Usage Benchmark Comparison

Dataset: ~400,000 small files across 100,000 directories (~15 GB on NVMe SSD, 8-Core CPU)

Tool	Cold Cache Execution	Warm Cache Execution	Architecture	Source
gdu	~2.10 s (~1.2x vs diskus)	~500 ms (On par)	Go (Parallel Goroutines, SSD-optimized)	dundee/gdu GitHub Benchmarks
du -sh (GNU)	17.78 s (~10x slower)	1098 ms (~2.2x slower)	C (Single-threaded coreutils)	GNU Coreutils du.c
diskus	1.75 s (Baseline)	~500 ms (Baseline)	Rust (Parallel crossbeam traversal)	sharkdp/diskus GitHub Benchmark

On non-cached disks (e.g., night-time cron tasks), multi-threaded tools (gdu, diskus) bypass the stat() I/O latency bottleneck by querying drive metadata across all CPU threads concurrently.

NOT a full drop-In Replacement for Scripts

Command Comparison
Original GNU du

du --dereference-args --summarize --block-size=1 --exclude=*.lock --exclude=dovecot.list.index.log.* /home/yellowtent/

Script-Compatible gdu Drop-In

gdu -ns -a --no-prefix -L -I ".*\.lock,dovecot\.list\.index\.log.*" /home/yellowtent/ | awk '{print $1"\t"$2}'

Flag Mapping

GNU du	gdu Equivalent	Purpose
-s, --summarize	-s	Output total size only
--block-size=1	-a --no-prefix	Raw byte count (no unit suffix)
-D, --dereference-args	-L	Follow symlinks
--exclude=A --exclude=B	-I "A,B"	Comma-separated exclude patterns
N/A	-n	Non-interactive mode (disables TUI)

Key Caveats

• Delimiter (Tab vs. Space): GNU du separates bytes and path with \t. gdu uses spaces. Piping to awk '{print $1"\t"$2}' prevents broken cut -f1 parsers in scripts.
• Symlink Scope: GNU du --dereference-args resolves symlinks only at the root level. gdu -L resolves all internal symlinks recursively, which can inflate total calculated sizes.
• Exit Codes: gdu -n silently skips unreadable files and exits 0. GNU du exits 1 on permission errors, which backup scripts often monitor.

[necrevistonnezr]

Just tested on my machine: When not excluding anything (i.e. including maildir to simulate a folder with hundreds of thousands of files):

du

time (du -h --dereference-args --summarize --block-size=1 /home/yellowtent/ | sort -hr) 

„Warm Cache“:
real  0m0,617s
user  0m0,139s
sys   0m0,482s

After reboot (to simulate cold cache):
real  0m12,146s
user  0m0,347s
sys   0m1,733s

gdu

time (gdu -ns -a --no-prefix -L -I /home/yellowtent/ | awk '{print $1"\t"$2}')

„Warm cache“:
real  0m0,239s
user  0m0,391s
sys   0m0,372s

After reboot (to simulate cold cache):
real  0m2,844s
user  0m0,729s
sys   0m1,652s

[nichu42]

I have an 8 Core Intel(R) Xeon(R) Gold 6142 CPU @ 2.60GHz VPS.
My Mastodon app data directory has ~ 450,000 files.

[nebulon]

woah that is a lot of files! But I have also seen my personal instance growing the file count a lot. I have set the cache retention days to 1 without noticable issues, it may have to refetch from the federated instance a bit more often if those posts do come up. Maybe this can help your instance also with file counts https://docs.cloudron.io/packages/mastodon#cache-retention-days

[nichu42]

1 day would be good for me, but most of my users log in once a week, so I cut it down to 7 days.
I have also resumed my project of moving to S3, and I think I have everything ready now. I am currently waiting for my user's browser caches to catch up with the new CSP, so images won't break when I do the switch.

But changing to gdu as suggested by @necrevistonnezr seems to be a good idea anyway.

[nebulon]

I have moved it to feature request section and adjusted the title now

[necrevistonnezr]

@nichu42 said:

I have an 8 Core Intel(R) Xeon(R) Gold 6142 CPU @ 2.60GHz VPS.
My Mastodon app data directory has ~ 450,000 files.

FYI my yellowtent folder has ~ 250.000 files on an SSD - just to give you perspective on the benchmarks.