@atrilahiji that's just poor traffic management 😛

@atrilahiji That'd be the tax monitors, they soon format c: your bank account if you stop pinging their servers 😱

@d19dotca Yes, the limits are there to protect against the noisy neighbor problem which exists when many processes are competing for the same resources and ONE uses up more than their fair share.

Technically we could have all 30 Apps be set to 1+GB on a 16GB RAM system and it would work fine until one App behaved badly. Then the system would be in trouble as the OOM killer would select a potentially critical service to kill.

With limits, the system is happy, and the killing happens in containers instead.

@robi That's my understanding, yes. That it will persist to containers. The article I liked in the earlier comment has the sysctl command to set vm swappiness. But as mentioned earlier, fine tuning these things should be done after investigating and understanding those settings because they usually end up having unexpected side effects.

@robi Yes please. 🙂

I also think next time I encounter a stability issue, I'll explore the provider angle sooner, maybe by replicating it elsewhere the same stack.

@girish said in Disk usage is not showing:

@lolliop @necrevistonnezr I think I found the issue for Ubuntu 20 atleast. Can you please try https://git.cloudron.io/cloudron/box/-/commit/6810d823f542808860955196d959fadaba0a9ab8 ? The file is at /home/yellowtent/box/setup/start/collectd/df.py. After you make the change systemctl restart collectd. You can check the logs in /home/yellowtent/platfordata/logs/collectd/app.log.

I still don't see any data after 10 minutes. The log after the change you mentioned (mind you, I'm still on 18.04):

[2020-12-07 14:06:27] Exiting normally. [2020-12-07 14:06:27] collectd: Stopping 5 read threads. [2020-12-07 14:06:27] error getting the size of /media/SG2TB/CloudronBackup: list index out of range [2020-12-07 14:06:27] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/appsdata/f393a761-8c10-46a6-83e7-486fa129a2d3/data" [2020-12-07 14:06:27] size of /home/yellowtent/appsdata/f393a761-8c10-46a6-83e7-486fa129a2d3/data is 18436096 (time: 1607349987) [2020-12-07 14:06:33] size of docker images is 12.56GB (12560000000) (time: 1607349993) [2020-12-07 14:06:33] collectd: Stopping 5 write threads. [2020-12-07 14:06:33] plugin_load: plugin "aggregation" successfully loaded. [2020-12-07 14:06:33] plugin_load: plugin "cpu" successfully loaded. [2020-12-07 14:06:33] plugin_load: plugin "interface" successfully loaded. [2020-12-07 14:06:33] plugin_load: plugin "load" successfully loaded. [2020-12-07 14:06:33] plugin_load: plugin "memory" successfully loaded. [2020-12-07 14:06:33] plugin_load: plugin "python" successfully loaded. [2020-12-07 14:06:33] plugin_load: plugin "swap" successfully loaded. [2020-12-07 14:06:33] plugin_load: plugin "write_graphite" successfully loaded. [2020-12-07 14:06:33] du plugin: monitoring /home/yellowtent/boxdata/mail [2020-12-07 14:06:33] du plugin: monitoring /home/yellowtent/boxdata [2020-12-07 14:06:33] du plugin: monitoring /home/yellowtent/platformdata [2020-12-07 14:06:33] plugin_load: plugin "table" successfully loaded. [2020-12-07 14:06:33] du plugin: monitoring /home/yellowtent/appsdata/1d6c5d10-67aa-4cbe-b21d-20e9b7688d83/data [2020-12-07 14:06:33] du plugin: monitoring /home/yellowtent/appsdata/29e278a4-c079-4401-942c-e0e17f588482/data [2020-12-07 14:06:33] du plugin: monitoring /home/yellowtent/appsdata/4a0b6b44-5cab-4c04-afae-5e078727dbe4/data [2020-12-07 14:06:33] du plugin: monitoring /home/yellowtent/appsdata/6c1ecf49-b8f4-4a77-ae31-56a6c53868cd/data [2020-12-07 14:06:33] du plugin: monitoring /media/SG2TB/M/Music [2020-12-07 14:06:33] du plugin: monitoring /home/yellowtent/appsdata/91a711c2-51b8-4095-8250-3168e957eb1b/data [2020-12-07 14:06:33] du plugin: monitoring /home/yellowtent/appsdata/b7ace111-fdd3-484f-bc6c-f8ad0b2e811f/data [2020-12-07 14:06:33] du plugin: monitoring /home/yellowtent/appsdata/c5c137fa-0221-4d42-b418-10e544cebad1/data [2020-12-07 14:06:33] du plugin: monitoring /media/SG2TB/CloudronBackup [2020-12-07 14:06:33] du plugin: monitoring /home/yellowtent/appsdata/f393a761-8c10-46a6-83e7-486fa129a2d3/data [2020-12-07 14:06:33] plugin_load: plugin "aggregation" successfully loaded. [2020-12-07 14:06:33] plugin_load: plugin "cpu" successfully loaded. [2020-12-07 14:06:33] plugin_load: plugin "interface" successfully loaded. [2020-12-07 14:06:33] plugin_load: plugin "load" successfully loaded. [2020-12-07 14:06:33] plugin_load: plugin "memory" successfully loaded. [2020-12-07 14:06:33] plugin_load: plugin "python" successfully loaded. [2020-12-07 14:06:33] plugin_load: plugin "swap" successfully loaded. [2020-12-07 14:06:33] plugin_load: plugin "write_graphite" successfully loaded. [2020-12-07 14:06:33] du plugin: monitoring /home/yellowtent/boxdata/mail [2020-12-07 14:06:33] du plugin: monitoring /home/yellowtent/boxdata [2020-12-07 14:06:33] du plugin: monitoring /home/yellowtent/platformdata [2020-12-07 14:06:33] plugin_load: plugin "table" successfully loaded. [2020-12-07 14:06:33] du plugin: monitoring /home/yellowtent/appsdata/1d6c5d10-67aa-4cbe-b21d-20e9b7688d83/data [2020-12-07 14:06:33] du plugin: monitoring /home/yellowtent/appsdata/29e278a4-c079-4401-942c-e0e17f588482/data [2020-12-07 14:06:33] du plugin: monitoring /home/yellowtent/appsdata/4a0b6b44-5cab-4c04-afae-5e078727dbe4/data [2020-12-07 14:06:33] du plugin: monitoring /home/yellowtent/appsdata/6c1ecf49-b8f4-4a77-ae31-56a6c53868cd/data [2020-12-07 14:06:33] du plugin: monitoring /media/SG2TB/M/Music [2020-12-07 14:06:33] du plugin: monitoring /home/yellowtent/appsdata/91a711c2-51b8-4095-8250-3168e957eb1b/data [2020-12-07 14:06:33] du plugin: monitoring /home/yellowtent/appsdata/b7ace111-fdd3-484f-bc6c-f8ad0b2e811f/data [2020-12-07 14:06:33] du plugin: monitoring /home/yellowtent/appsdata/c5c137fa-0221-4d42-b418-10e544cebad1/data [2020-12-07 14:06:33] du plugin: monitoring /media/SG2TB/CloudronBackup [2020-12-07 14:06:33] du plugin: monitoring /home/yellowtent/appsdata/f393a761-8c10-46a6-83e7-486fa129a2d3/data [2020-12-07 14:06:33] Systemd detected, trying to signal readyness. [2020-12-07 14:06:33] custom du plugin initialized with [{'exclude': '', 'name': 'maildata', 'dir': '/home/yellowtent/boxdata/mail'}, {'exclude': 'mail', 'name': 'boxdata', 'dir': '/home/yellowtent/boxdata'}, {'exclude': '', 'name': 'platformdata', 'dir': '/home/yellowtent/platformdata'}, {'exclude': '', 'name': '1d6c5d10-67aa-4cbe-b21d-20e9b7688d83', 'dir': '/home/yellowtent/appsdata/1d6c5d10-67aa-4cbe-b21d-20e9b7688d83/data'}, {'exclude': '', 'name': '29e278a4-c079-4401-942c-e0e17f588482', 'dir': '/home/yellowtent/appsdata/29e278a4-c079-4401-942c-e0e17f588482/data'}, {'exclude': '', 'name': '4a0b6b44-5cab-4c04-afae-5e078727dbe4', 'dir': '/home/yellowtent/appsdata/4a0b6b44-5cab-4c04-afae-5e078727dbe4/data'}, {'exclude': '', 'name': '6c1ecf49-b8f4-4a77-ae31-56a6c53868cd', 'dir': '/home/yellowtent/appsdata/6c1ecf49-b8f4-4a77-ae31-56a6c53868cd/data'}, {'exclude': '', 'name': '6d1f25e8-39ca-46e2-b8a7-fb9db45c7a9b', 'dir': '/media/SG2TB/M/Music'}, {'exclude': '', 'name': '91a711c2-51b8-4095-8250-3168e957eb1b', 'dir': '/home/yellowten [2020-12-07 14:06:34] custom df plugin initialized with [[u'/dev/mapper/beebox--vg-root', u'/', u'476261416', u'421197060', u'30848600'], [u'/dev/sda1', u'/media/SG2TB', u'1922728820', u'1631095852', u'193940908']] [2020-12-07 14:06:34] Initialization complete, entering read-loop. [2020-12-07 14:06:34] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/boxdata/mail" [2020-12-07 14:06:36] size of /home/yellowtent/boxdata/mail is 6544105472 (time: 1607349996) [2020-12-07 14:06:36] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/boxdata" --exclude "mail" [2020-12-07 14:06:36] size of /home/yellowtent/boxdata is 4628480 (time: 1607349996) [2020-12-07 14:06:36] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/platformdata" [2020-12-07 14:06:37] size of /home/yellowtent/platformdata is 2807963648 (time: 1607349996) [2020-12-07 14:06:37] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/appsdata/1d6c5d10-67aa-4cbe-b21d-20e9b7688d83/data" [2020-12-07 14:06:37] size of /home/yellowtent/appsdata/1d6c5d10-67aa-4cbe-b21d-20e9b7688d83/data is 34973192192 (time: 1607349996) [2020-12-07 14:06:37] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/appsdata/29e278a4-c079-4401-942c-e0e17f588482/data" [2020-12-07 14:07:11] size of /home/yellowtent/appsdata/29e278a4-c079-4401-942c-e0e17f588482/data is 345786105856 (time: 1607350030) [2020-12-07 14:07:11] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/appsdata/4a0b6b44-5cab-4c04-afae-5e078727dbe4/data" [2020-12-07 14:07:11] size of /home/yellowtent/appsdata/4a0b6b44-5cab-4c04-afae-5e078727dbe4/data is 1777664 (time: 1607350030) [2020-12-07 14:07:11] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/appsdata/6c1ecf49-b8f4-4a77-ae31-56a6c53868cd/data" [2020-12-07 14:07:11] size of /home/yellowtent/appsdata/6c1ecf49-b8f4-4a77-ae31-56a6c53868cd/data is 12288 (time: 1607350030) [2020-12-07 14:07:11] computing size with command: timeout 1800 du -DsB1 "/media/SG2TB/M/Music" [2020-12-07 14:07:19] size of /media/SG2TB/M/Music is 291166932992 (time: 1607350039) [2020-12-07 14:07:19] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/appsdata/91a711c2-51b8-4095-8250-3168e957eb1b/data" [2020-12-07 14:07:19] size of /home/yellowtent/appsdata/91a711c2-51b8-4095-8250-3168e957eb1b/data is 49152 (time: 1607350039) [2020-12-07 14:07:19] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/appsdata/b7ace111-fdd3-484f-bc6c-f8ad0b2e811f/data" [2020-12-07 14:07:20] size of /home/yellowtent/appsdata/b7ace111-fdd3-484f-bc6c-f8ad0b2e811f/data is 12730368 (time: 1607350039) [2020-12-07 14:07:20] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/appsdata/c5c137fa-0221-4d42-b418-10e544cebad1/data" [2020-12-07 14:07:20] size of /home/yellowtent/appsdata/c5c137fa-0221-4d42-b418-10e544cebad1/data is 40992768 (time: 1607350039) [2020-12-07 14:07:20] computing size with command: timeout 1800 du -DsB1 "/media/SG2TB/CloudronBackup"

Just leaving a note for myself, but it's unclear why the issue got fixed after the update. After an app update, the container is recreated and the collectd config is regenerated. Maybe the collectd restart did the trick.

@nebulon Thanks for your reply.

After thinking about it, the graph chosen is better to view memory usage (which fluctuate much more than disk).

I would then suggest separating both information:

keep the current graph for the app usage (the Y axis adapting to the app using the most memory) add a bar for total memory used, modelled on the total disk usage (could be a single colour for memory usage)
total disk usage

This would help maximize the amount of information you can visualize in one go and help detect spikes better.

@ice The Cloudron disk graphs is computed only twice a day. So, please check after atleast 12 hours of the disk resize.

@smilebasti The /home/yellowtent/appsdata is the location of apps. This size seems to roughly match the nextcloud size. As for docker, you should not use du tools inside docker's image directories since they are overlays and the du tool is not smart enough to figure out the size correctly. Try docker system df to get a better idea about the actual size docker uses (this is what is reported in the graph as ~5GB). The volumes also link into appsdata so they might be double counted the du tools.

To take a wild guess, maybe you were backing up to the file system for some time before you moved to NAS via SMB? If this was the case, then you should remove the old backups manually from /var/backups. You can just safely nuke all the timestamped directories and the snapshot directory inside it.

@nebulon about 15-20 every second. When it stops there is a line box:apphealthmonitor app health: 28 alive / 4 dead. Nothing about accessing any sites though.

@girish That's perfect, thank you so much! 🙂

@ruihildt Thanks for reporting. This is fixed in the coming release - https://git.cloudron.io/cloudron/box/-/blob/master/CHANGES#L1930

@ruihildt The issue in your case was different. There were so many apps that the query parameter limit was getting exceeded. This is fixed in next release. @necrevistonnezr this is most likely your issue as well!

Agree, the memory graphs are quite rudimentary at this point. We will rework them to be more useful and correct at some point.

Finally, one important bit: by default ext4 will reserve 5% of the disk for the root user. You can see this like so:

root@strawberry:~# tune2fs -l /dev/sda1 | grep Reserved Reserved block count: 1562918 Reserved GDT blocks: 1024 Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root)

1562918 blocks * 4096 block size = ~6GB.

You can reduce the percent to say 2 like this:

tune2fs -m 2 /dev/sda1

Unless you have a big disk, I would leave it at 5%.

In the past, I think people have recommended integrating with netdata. I don't have experience with netdata but wanted to throw the idea out there.

Other than that, current graphs is rendered using data collected by graphite/carbon.

This is fixed now, will be part of next release.

Interesting, this was simply missed I guess. I have added it now and it will be part of the next release.

It will look like this then:
Screenshot_2019-11-05 Cloudron.png