Huge memory spikes

[robi]

Something is regularly causing huge memory spikes. Since all apps have limits this shouldn't be possible.
Here's the 6h graphs:

CPU correlates

I don't see anything other than occasional janitor cleanup that lasts on the order of seconds, not minutes, in box log (8MB)

What else?

Here's the 24h graphs:

[girish]

@robi The only CPU/memory intensive thing the box code does is backups. Do the spikes match your backup interval configuration?

[robi]

@girish No, that is why these are concerning. My backups are weekly.

This is happening on the demo cloudron as well.

[girish]

@robi said in Huge memory spikes:

This is happening on the demo cloudron as well.

It seems that they are somehow related to OOM in the demo cloudron atleast:

# grep OOM /home/yellowtent/platformdata/logs/box.log | wc -l
3572

Which now reminds me that I think your server is also on ssdnodes as well right ? They have some strange CPU/memory sharing going on which makes all that I/O unreliable. I have been meaning to move the demo out of them at some point. If you just try to uninstall an app, you will see how slow it is despite it having plenty of RAM (48GB overall!) and it's doing nothing.

[robi]

@girish That's not it. You have OOMs because people install all sorts of apps and try to abuse the system, like WP and trying out large plugins.

My system has 0 OOMs. None.

Mine also has enough of IO as I asked them to limit the number of new customers on the host system.

# dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync ; rm /tmp/test1.img
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.50861 s, 195 MB/s

Doesn't explain 48GB mem spikes.

This only seems to be happening on 20.04 Cloudrons, not 18.04.

Btw, the main issue I think you're having is the choice of that NY datacenter, it's networking isn't great, they seem to allow burst traffic but then quickly throttle down or experience congestion.

Seattle has been way better for me.