We talk about CPU (or processors, or cores, or)
How we talk about CPU (or processors, or cores, or...)



There are a bunch of ways that we talk about CPU, and they're not all 
measuring the same thing.  We're not even always talking about the same 
physical part of the computer!  More on that in a bit, though.  Let's
talk about the numbers first.


There's the load average sense, where you get to see the run queue 
averaged over three spans of time.  Those are the number of things which 
are actually running or are trying to run.  This has changed subtly in 
some places over the years to include or exclude different things.


A load average can be particularly fun since it usually (on Linux) 
includes stuff that's in the kernel and trying to run but is stuck on 
some kind of lock, like if 
reading /proc/pid/cmdline gets stuck,
or a NFS mount is hung, or whatever.  If you stack a bunch of those up 
from cron jobs or other repetitive things, you might end up with a load 
average in the hundreds or thousands... and still have a perfectly 
responsive machine!


On the other hand, if you have just a couple of jobs, but they are 
consuming every resource they can, then you might have a load average 
barely higher than the CPU (or processor, or thread... see below) count 
on the box and yet it'll take 5 minutes to just display the shell prompt 
when you log in.


Clearly, the load average is only one way to look at things.


Then there's what you see when you look at a process in 'ps'.  It shows 
a number like "%CPU", but that's not instantaneous.  It's usually the 
result of dividing the elapsed CPU time into the elapsed wall time.  If 
your process has been running for a day and has used 1000 seconds of CPU 
time, then that's about 1% CPU utilization over that whole period.


If the process suddenly ramps up and gets very busy, that number will go 
up, but if it's been running for a long time, it won't happen right 
away.  After all, when the denominator is large, incrementing the 
numerator doesn't always make the outcome change a lot.


The numbers from 'ps' are also really fun because a process with a bunch 
of threads can totally use more than 100% CPU over time.  If you have 32
CPUs (or processors, or whatever, read on for more on that vocabulary 
issue) on your machine, you could totally have a single process 
approaching "3200% CPU" in a ps output.  (Ah, my old web server 
machines.)


Meanwhile, a tool like 'top' (or atop) is going to show you what's going 
on more-or-less right now.  I suspect it's taking snapshots, calculates 
the delta in CPU time and then divides it into the amount of elapsed 
wall time between those snapshots.  If the program used 1 second of CPU 
time in the past 2 real (wall time) seconds, then that's 50%.  You can 
tell that top is doing this kind of shorter-timescale analysis by 
starting something that chews CPU, watching it spike, and then suspend 
it and watch it drop back to 0.  'ps', meanwhile, would still be 
showing the amount used over the entire lifetime of the process so far.


Next, you have process scheduling things like Borg back in the old days 
(like, over a decade ago).  This is where we'd define a job and say 
"cpu=1.0", and that basically meant it reserved approximately one core 
for itself.  It wasn't an exact science.  Sometimes people would do fun 
things like "cpu=0.99" so they'd be able to use almost the same amount 
of CPU time but it wouldn't need to reserve a whole core for it.


We also have tools like WMCPULoad, which is a fun little X widget from 
the days of Window Maker.  Oh yeah, I'm talking about the late '90s 
here.  It just shows a single % number which ranges from 0 to 100.  That 
number is talking about the entire box, no matter how many CPUs you 
have!


I have an eight-way machine that used to do a lot of software-defined 
radio work.  At its busiest, wmcpuload used to show 50% or more.  That 
basically meant that four of its processors were cranking away 
full-tilt, or equivalent.  In all likelihood, all eight of them were 
doing stuff, and some of them might have been fully pegged, but the 
others were somewhere less, and it all summed up to 50% of its absolute 
peak.


So now let's talk about these hardware names: packages, CPUs, 
processors, cores, and threads.  These things show up all over the 
place.


If you look in /proc/cpuinfo on a Linux box, you will see stanzas 
labeled "processor", but it's not quite that simple, because a CPU with 
hyperthreading actually shows up with multiple "processors".  My 
eight-way box is actually a "four core" machine with two "threads" per.  
These are not the same threads as the ones that we talk about when 
running processes in parallel, but their numbers definitely influence 
each other -- if you have a lot of (process) threads, a machine with 
more (CPU) threads will probably be able to make more progress in 
parallel than one with fewer.  Confusing, right?


Try explaining this to someone who's never heard any of this before.


The "package", then, is the physical thing you slap into a socket or 
slot on the motherboard (or not, if it's soldered down).  Your machine 
might have multiple packages.  Each package might have multiple cores.  
Then each core might have multiple threads.


What you see in /proc/cpuinfo on Linux... is that last number, all the 
way at the bottom of the multiplication.


All of this means that when someone says "this thing is at half a CPU", 
it probably means that it's consuming 50% of one thread, while by 
saying "this thing is using half the box", it's probably using 50% of 
*everything*.


Then, if someone talks about CPUs, or processors, or threads, or 
packages, you really have to pay attention because they might be mixing 
and matching things, while someone else has a slightly different take on 
this whole business while using the same words.


So, top and atop show you almost current utilization (unless you're back 
in atop's history), ps shows you total utilization for the entire 
lifetime, load averages talk about what's running and also *trying* to 
run, even if it's not doing any work, and resource schedulers sometimes 
have weird inflection points at integer boundaries.  What a mess!


Don't even get me started on virtualization and stolen time.  Ugh!


May 22, 2021: This post has an update.
