PSA: mystery memory pressure / "high system CPU" on a KVM VPS? Check for host-side ballooning

[LoudLemur]

PSA: mystery memory pressure / "high system CPU" on a KVM VPS? Check for host-side ballooning

Sharing this because it cost me two days of chasing ghosts inside my own server. Nothing below touches Cloudron: it is all read-only checks plus standard Ubuntu monitoring packages.

See this?

• System CPU creeping up over weeks, reset by a reboot
• Memory graphs showing huge usage, but your app containers only account for a fraction of it
• Swap constantly churning, kswapd busy, everything slightly sluggish
• free -h says tens of GB "used" that no process owns

Check this

1. How much memory is unaccounted for? Every byte should belong to a process or a kernel counter. This sums the counters and prints what is missing:

awk '/^MemTotal:|^MemFree:|^Buffers:|^Cached:|^SwapCached:|^AnonPages:|^Slab:|^KernelStack:|^PageTables:|^VmallocUsed:|^Percpu:/ {a[$1]=$2}
END {m=a["MemTotal:"]; for (k in a) if (k!="MemTotal:") m-=a[k];
printf "unaccounted: %.1f GB of %.1f GB\n", m/1048576, a["MemTotal:"]/1048576}' /proc/meminfo

Healthy is near zero (a GB or two). Mine showed 45GB of 64GB unaccounted at the worst moment: memory the host had taken back via the virtio balloon.

2. Is the balloon driver actively working?

dmesg | grep -i balloon
ls -l /sys/bus/virtio/devices/*/driver | grep balloon

A virtio_balloon device merely existing is normal on KVM. But if you see lines like this, the host is resizing the balloon so hard the kernel complains:

workqueue: update_balloon_size_func hogged CPU for >10000us 8192 times

3. Catch it in the act. The spikes can last only 10-15 minutes, so a spot check misses them. Drop a one-line cron heartbeat (safe, read-only):

echo '* * * * * root awk "/^MemTotal:/{t=\$2} /^MemAvailable:/{a=\$2} END{print strftime(\"%FT%T\"), \"used_pct=\" int(100*(t-a)/t)}" /proc/meminfo >> /var/log/mem-heartbeat.log' > /etc/cron.d/mem-heartbeat

A sawtooth like used_pct=9 ... 23 ... 80 ... 9 within minutes, with no matching process activity, is the balloon inflating and deflating. Also apt install sysstat and enable it; sar -r and sar -u give you history for the ticket. When a spike hits, capture cat /proc/meminfo, ps aux --sort=-rss | head -25 and docker stats --no-stream in the same minute: processes small + huge unaccounted gap = proof.

Do this to solve

You cannot fix this from inside the guest; the balloon is controlled by the hypervisor. In our case (SSD Nodes, 64GB plan) the host was repeatedly reclaiming up to 45GB, and the "high system CPU" was just my kernel's balloon workqueue plus kswapd feeding it.

(SSDnodes have been great otherwise and would definitely use again and recommend them to the community if you need something with lots of RAM for eg an experimental VPS / hobby.)

1. Open a ticket with your provider with the dmesg lines, the unaccounted-memory numbers and the heartbeat timestamps. Ask directly: is ballooning applied to my VM, and is my plan's RAM dedicated?
2. Add shock absorbers while you wait: generous swap (I added a 16GB swapfile alongside Cloudron's) and vm.swappiness=10. This keeps apps alive through inflation spikes instead of OOMing.
3. If the provider will not stop it, migrate. A box that intermittently loses two thirds of its RAM will never run a busy Cloudron well, and no amount of in-guest tuning changes that.

Hopefully, this will save someone the detour of blaming ClickHouse, their apps, or the kernel first. I did all three.

[james]

Hello @loudlemur
Great find!
That is actually coming from the provider makes it really tricky to understand/find.

[robi]

Wouldn't it be nice to have a small utility monitor for this and surface a notification and possibly suggest mitigations?

Support is usually pretty good about migrating your VM to a different node.

[LoudLemur]

Update — resolved, plus a few things I learned that weren't in the original post.

Outcome first: the provider acknowledged the ballooning was beyond their normal range and made two host-side adjustments. The first one didn't hold. reclaim spikes up to ~96% memused resumed that same evening. The second one worked: 40+ hours flat at ~10% memused, balloon fully deflated, right through the time windows where every earlier event clustered. Ticket closed.

New/helpful bits for anyone landing here later:

1. Verify the fix against your event windows, not the clock. Our events clustered in two daily windows (evening and early morning, presumably the node's busy periods). A VM that looks clean at lunchtime proves nothing; check sar -r across the windows where your events happened before you believe a fix.

2. Don't panic over update_balloon_stats_func in dmesg. That's the host polling guest memory stats, harmless, and arguably a good sign (they're watching, not reclaiming). The line that matters is update_balloon_size_func, that's the balloon actually inflating.

3. "Fixed once" is not fixed. If we'd accepted the first "we've adjusted the node" we'd have concluded the provider fixed it while the VM was still losing half its RAM nightly. Keep the heartbeat cron from the original post running permanently, it's one line, costs nothing, and it's the only reason we caught the failed first fix.

4. No root SSH handy? Containers can still see host memory. /proc/meminfo isn't namespaced, so from any container (docker exec, or Cloudron's cloudron exec --app …) you can read the host's MemTotal/MemAvailable and run the unaccounted-memory check from the original post. dmesg is blocked from containers, but meminfo alone will tell you if the balloon is inflated right now.

5. Ask the provider what they changed. "We doubt it will recur after the current settings" is not the same as "your VM is exempt from auto-ballooning." Get the actual change on record — it determines whether a future recurrence means the fix regressed or was never real.

We still like SSDnodes a lot!

[robi]

I have a server where it seems to stop responding and generates timeouts in monitoring which come back a few minutes later. Perhaps it's this pattern.