TL/DR
The % score on CPU usage in the graph is for CPU Cores, eg 500% means 5 (of your many eg 12 cores) are working at maximum capacity. It doesn't mean that your entire 12 core machine is at maximum. If you have 12 cores, you might see the y-axis reaching 1200% and be wondering how on earth anybody might end up there. It would be because then all of your (in this case) 12 cores would be maxed.
We hope team cloudron could provide this additional context in the panel.
The cron heartbeat to catch brief reclamation spikes is a good trick I had not seen before.
Worth adding a companion trap for anyone who arrives here from a search, because it produces the same symptom, "my box looks busy and I do not know why", and it is worth ruling out before you open a ticket with your provider.
Two of the numbers on hand are easy to read as CPU usage when neither of them is.
I went looking at one of my own boxes today, convinced it was still labouring after a genuine runaway had been dealt with. Twelve cores, and the figures were:
load average: 15.78, steady across 1, 5 and 15 minutes
CPU: 80 per cent idle, iowait 0 per cent, steal 0 per cent
memory pressure, from /proc/pressure/memory: 0.00
one app's graph in the dashboard, earlier in the day: around 700 per cent, which is seven of the twelve cores
The box was completely fine. Nothing needed doing.
The load average is not a CPU utilisation figure. On Linux it counts processes that are runnable or blocked in uninterruptible sleep, so a machine running many containers shows a persistently elevated load while the CPU sits mostly idle. I had 92 running containers at the time. Ninety-two containers' worth of health checks, schedulers and database background threads constantly waking, doing a little work, and sleeping again will hold the load average in the teens on a box that is doing almost nothing. vmstat 5 2 and the idle column is the number that actually answers "is my CPU busy".
The per-app CPU graph is in core units, not machine units. The thing to hold on to is this:
500 per cent does not mean five times your machine. It means 100 per cent of each of five cores, out of the twelve you have. So it is five twelfths of the box, not five boxes.
It follows the Docker convention where 100 per cent means one core, the same as docker stats and the same as top for a multithreaded process, so on a twelve core box the ceiling is 1200 per cent. An app sitting at 700 per cent is saturating seven cores and leaving you five.
That is genuinely useful information and it is the right unit for comparing apps against each other. It is simply very easy, the first time you meet it, to read a number above 100 per cent as "more than my whole machine" and start looking for a fire.
Both of those are correct and conventional. The problem is that neither is interpretable on its own, and the question you actually have when you go looking is "is my machine in trouble", which neither answers.
So, a small request to the Cloudron team, in increasing order of effort.
Put the core count on the per-app CPU graph's axis. Labelling the maximum as 1200% (12 cores), or drawing a ceiling line, would remove the ambiguity completely, and it is about as small a change as a graph can take.
Offer cores as an alternative unit. "5.0 of 12 cores" is unambiguous in a way that "500%" will never be, however conventional the percentage is.
On the system level view rather than the per-app one, show machine-normalised CPU alongside the load average, and say somewhere nearby that the load average is not a utilisation figure. The per-app graphs are fine for comparing apps and cannot answer the question about the machine, and that gap is where people end up guessing.
None of this is a bug, and I am not suggesting the underlying numbers are wrong. It is that the presentation asks the reader to bring context that is not on the screen, and in a thread like this one you can watch that cost people real time before they get as far as suspecting their hypervisor.
