kudos to @girish
he found the real pbl. (out of i-nodes) within minutes.
from there, we could nail down the cause, by following this:
https://unix.stackexchange.com/questions/26598/how-can-i-increase-the-number-of-inodes-in-an-ext4-filesystem
TL,DR:
one cannot raise i-nodes after fs creation. normally, a tar from rootfs, reformat the rootfs, and restore would be necessary.
but to find out, who is consuming all the inodes, one can do the following:
try du -s --inodes * 2>/dev/null |sort -g then cd into the last dir in output and repeat.
Full Disclosure: not all OS's support --inodes flag for du command (my Mac OS does not) but many Linux OS's do.
one has to cd into the dir with the most i-nodes, recursively going down the tree and finally find the dir with the biggest i-node consumption.
in this case, as girish had mentioned, it was caused by not right configured nullmailer, writing tons of error-msg to /var/spool/nullmailer/failed useing 4.4M i-nodes…
deleting that dir eased the situation ad hoc.
rebooting the server and restart all failed apps (GUI & CLI) fixed it.
thanks for all your help
