How to benchmark Switchable Optimizations in MySQL
(I realize that this does not &quot;answer&quot; the Question.  But it is too long for a Comment.)
Good luck.  re &quot;Almost identical&quot; -- Probably because they were &quot;identical&quot; except for random noise in the timing.  The switchable optimizations are, in general, &quot;binary&quot; and &quot;optional&quot;.
Consider an obvious situation:  You have an index that could be used in a query.  Still, the Optimizer will deliberately consider whether to ignore the index and simply scan the table.  It may be more costly to bounce between the index and the table, hence using the index would slow down execution.  This is a &quot;binary&quot; decision -- use vs ignore index.  The Optimizer cannot know the exact cutoff between the choices.  It is influenced by HDD vs SDD, buffer_pool_size, other activity, statistics, etc, etc.
That is, a benchmark would be complicated by having more than 2 choices (an index exists or not):

FORCE INDEX vs IGNORE INDEX vs 'allowed' the Optimizer to choose
for &quot;allowed&quot; the Optimizer picks the better choice vs fails to

Any of these could change whether the Optimizer would pick a particular optimization.  Also, the timings would change by a relatively unpredictable amount in cases such as these:

Adding more rows to a table
Changing the size of a 'range'
ANALYZE TABLE will sometimes lead to changing things significantly.
OPTIMIZE TABLE -- ditto.

Over the years, I occasionally see &quot;index merge intersect&quot;.  (INDEX(a), INDEX(b) with WHERE a=1 AND b=2)  In every case, a composite index would be better (INDEX(a,b)).
Change that AND to OR and no index is useful -- except when &quot;index merge union&quot; is used.  This almost never happens.  It takes two index range scans, a merge, and then something like a &quot;join&quot;.
EXPLAIN does not show much info.  EXPLAIN FORMAT=JSON is better.  The Optimizer Trace shows a different set of info.
I don't know of a test set like what you seek.  I see the writing of even one test to be quite a challenge (see above).  Furthermore, interpreting the results would be challenging.
If you do succeed with your goal, please blog about it.