PSA: mystery memory pressure / "high system CPU" on a KVM VPS? Check for host-side ballooning

[LoudLemur]

PSA: mystery memory pressure / "high system CPU" on a KVM VPS? Check for host-side ballooning

Sharing this because it cost me two days of chasing ghosts inside my own server. Nothing below touches Cloudron: it is all read-only checks plus standard Ubuntu monitoring packages.

See this?

• System CPU creeping up over weeks, reset by a reboot
• Memory graphs showing huge usage, but your app containers only account for a fraction of it
• Swap constantly churning, kswapd busy, everything slightly sluggish
• free -h says tens of GB "used" that no process owns

Check this

1. How much memory is unaccounted for? Every byte should belong to a process or a kernel counter. This sums the counters and prints what is missing:

awk '/^MemTotal:|^MemFree:|^Buffers:|^Cached:|^SwapCached:|^AnonPages:|^Slab:|^KernelStack:|^PageTables:|^VmallocUsed:|^Percpu:/ {a[$1]=$2}
END {m=a["MemTotal:"]; for (k in a) if (k!="MemTotal:") m-=a[k];
printf "unaccounted: %.1f GB of %.1f GB\n", m/1048576, a["MemTotal:"]/1048576}' /proc/meminfo

Healthy is near zero (a GB or two). Mine showed 45GB of 64GB unaccounted at the worst moment: memory the host had taken back via the virtio balloon.

2. Is the balloon driver actively working?

dmesg | grep -i balloon
ls -l /sys/bus/virtio/devices/*/driver | grep balloon

A virtio_balloon device merely existing is normal on KVM. But if you see lines like this, the host is resizing the balloon so hard the kernel complains:

workqueue: update_balloon_size_func hogged CPU for >10000us 8192 times

3. Catch it in the act. The spikes can last only 10-15 minutes, so a spot check misses them. Drop a one-line cron heartbeat (safe, read-only):

echo '* * * * * root awk "/^MemTotal:/{t=\$2} /^MemAvailable:/{a=\$2} END{print strftime(\"%FT%T\"), \"used_pct=\" int(100*(t-a)/t)}" /proc/meminfo >> /var/log/mem-heartbeat.log' > /etc/cron.d/mem-heartbeat

A sawtooth like used_pct=9 ... 23 ... 80 ... 9 within minutes, with no matching process activity, is the balloon inflating and deflating. Also apt install sysstat and enable it; sar -r and sar -u give you history for the ticket. When a spike hits, capture cat /proc/meminfo, ps aux --sort=-rss | head -25 and docker stats --no-stream in the same minute: processes small + huge unaccounted gap = proof.

Do this to solve

You cannot fix this from inside the guest; the balloon is controlled by the hypervisor. In our case (SSD Nodes, 64GB plan) the host was repeatedly reclaiming up to 45GB, and the "high system CPU" was just my kernel's balloon workqueue plus kswapd feeding it.

(SSDnodes have been great otherwise and would definitely use again and recommend them to the community if you need something with lots of RAM for eg an experimental VPS / hobby.)

1. Open a ticket with your provider with the dmesg lines, the unaccounted-memory numbers and the heartbeat timestamps. Ask directly: is ballooning applied to my VM, and is my plan's RAM dedicated?
2. Add shock absorbers while you wait: generous swap (I added a 16GB swapfile alongside Cloudron's) and vm.swappiness=10. This keeps apps alive through inflation spikes instead of OOMing.
3. If the provider will not stop it, migrate. A box that intermittently loses two thirds of its RAM will never run a busy Cloudron well, and no amount of in-guest tuning changes that.

Hopefully, this will save someone the detour of blaming ClickHouse, their apps, or the kernel first. I did all three.

[james]

Hello @loudlemur
Great find!
That is actually coming from the provider makes it really tricky to understand/find.

[robi]

Wouldn't it be nice to have a small utility monitor for this and surface a notification and possibly suggest mitigations?

Support is usually pretty good about migrating your VM to a different node.

[LoudLemur]

Update — resolved, plus a few things I learned that weren't in the original post.

Outcome first: the provider acknowledged the ballooning was beyond their normal range and made two host-side adjustments. The first one didn't hold. reclaim spikes up to ~96% memused resumed that same evening. The second one worked: 40+ hours flat at ~10% memused, balloon fully deflated, right through the time windows where every earlier event clustered. Ticket closed.

New/helpful bits for anyone landing here later:

1. Verify the fix against your event windows, not the clock. Our events clustered in two daily windows (evening and early morning, presumably the node's busy periods). A VM that looks clean at lunchtime proves nothing; check sar -r across the windows where your events happened before you believe a fix.

2. Don't panic over update_balloon_stats_func in dmesg. That's the host polling guest memory stats, harmless, and arguably a good sign (they're watching, not reclaiming). The line that matters is update_balloon_size_func, that's the balloon actually inflating.

3. "Fixed once" is not fixed. If we'd accepted the first "we've adjusted the node" we'd have concluded the provider fixed it while the VM was still losing half its RAM nightly. Keep the heartbeat cron from the original post running permanently, it's one line, costs nothing, and it's the only reason we caught the failed first fix.

4. No root SSH handy? Containers can still see host memory. /proc/meminfo isn't namespaced, so from any container (docker exec, or Cloudron's cloudron exec --app …) you can read the host's MemTotal/MemAvailable and run the unaccounted-memory check from the original post. dmesg is blocked from containers, but meminfo alone will tell you if the balloon is inflated right now.

5. Ask the provider what they changed. "We doubt it will recur after the current settings" is not the same as "your VM is exempt from auto-ballooning." Get the actual change on record — it determines whether a future recurrence means the fix regressed or was never real.

We still like SSDnodes a lot!

[robi]

I have a server where it seems to stop responding and generates timeouts in monitoring which come back a few minutes later. Perhaps it's this pattern.

[LoudLemur]

TL/DR

The % score on CPU usage in the graph is for CPU Cores, eg 500% means 5 (of your many eg 12 cores) are working at maximum capacity. It doesn't mean that your entire 12 core machine is at maximum. If you have 12 cores, you might see the y-axis reaching 1200% and be wondering how on earth anybody might end up there. It would be because then all of your (in this case) 12 cores would be maxed.

We hope team cloudron could provide this additional context in the panel.

---

The cron heartbeat to catch brief reclamation spikes is a good trick I had not seen before.

Worth adding a companion trap for anyone who arrives here from a search, because it produces the same symptom, "my box looks busy and I do not know why", and it is worth ruling out before you open a ticket with your provider.

Two of the numbers on hand are easy to read as CPU usage when neither of them is.

I went looking at one of my own boxes today, convinced it was still labouring after a genuine runaway had been dealt with. Twelve cores, and the figures were:

• load average: 15.78, steady across 1, 5 and 15 minutes
• CPU: 80 per cent idle, iowait 0 per cent, steal 0 per cent
• memory pressure, from /proc/pressure/memory: 0.00
• one app's graph in the dashboard, earlier in the day: around 700 per cent, which is seven of the twelve cores

The box was completely fine. Nothing needed doing.

The load average is not a CPU utilisation figure. On Linux it counts processes that are runnable or blocked in uninterruptible sleep, so a machine running many containers shows a persistently elevated load while the CPU sits mostly idle. I had 92 running containers at the time. Ninety-two containers' worth of health checks, schedulers and database background threads constantly waking, doing a little work, and sleeping again will hold the load average in the teens on a box that is doing almost nothing. vmstat 5 2 and the idle column is the number that actually answers "is my CPU busy".

The per-app CPU graph is in core units, not machine units. The thing to hold on to is this:

500 per cent does not mean five times your machine. It means 100 per cent of each of five cores, out of the twelve you have. So it is five twelfths of the box, not five boxes.

It follows the Docker convention where 100 per cent means one core, the same as docker stats and the same as top for a multithreaded process, so on a twelve core box the ceiling is 1200 per cent. An app sitting at 700 per cent is saturating seven cores and leaving you five.

That is genuinely useful information and it is the right unit for comparing apps against each other. It is simply very easy, the first time you meet it, to read a number above 100 per cent as "more than my whole machine" and start looking for a fire.

Both of those are correct and conventional. The problem is that neither is interpretable on its own, and the question you actually have when you go looking is "is my machine in trouble", which neither answers.

So, a small request to the Cloudron team, in increasing order of effort.

Put the core count on the per-app CPU graph's axis. Labelling the maximum as 1200% (12 cores), or drawing a ceiling line, would remove the ambiguity completely, and it is about as small a change as a graph can take.

Offer cores as an alternative unit. "5.0 of 12 cores" is unambiguous in a way that "500%" will never be, however conventional the percentage is.

On the system level view rather than the per-app one, show machine-normalised CPU alongside the load average, and say somewhere nearby that the load average is not a utilisation figure. The per-app graphs are fine for comparing apps and cannot answer the question about the machine, and that gap is where people end up guessing.

None of this is a bug, and I am not suggesting the underlying numbers are wrong. It is that the presentation asks the reader to bring context that is not on the screen, and in a thread like this one you can watch that cost people real time before they get as far as suspecting their hypervisor.