join_buffer_size causes high cpu load
Please try the experiment again; this time capture these for the two values of join_buffer_size:
 EXPLAIN SELECT ... (including the query)
 how long the query took
 SHOW CREATE TABLE ...  -- of reach table in the query

Meanwhile, I will guess at this explanation.  The Optimizer has several ways it might perform a query.

Ignore the join_buffer and use lookups (indexed or not) for joining.
Use the join_buffer:  Load the second table completely into a hash in RAM, then reach into it for lookups.

The potential problem include

The Optimizer may not have the &quot;right&quot; statistics for deciding between the two choices.  (There is no reliable way to fix this for all cases.)
There may be a LIMIT or something else that makes loading the entire table the &quot;wrong&quot; approach.
Once the join_buffer is loaded, the rest of the query is CPU-bound.  When not using the join_buffer, the query may be I/O-bound and is likely to take longer (elapsed time).  (Hence my request for the timings.)
Etc.