Constructing a dataset that scores well only for a specific set of hyper parameter values
I'm not familiar with any existing, robust methods to generate such a dataset. Here are some thoughts though.

You propose using an MLP with a single hidden layer. That means we have two weight matrices, and two activation functions (one for hidden layer, one for output layer). Some notation:


$d$: dimensionality of input vectors
$n$: dimensionality of hidden layer
You mentioned binary output, so I'll assume that dimensionality is $2$
$W_n^{(1)} \in \mathbb{R}^{d \times n}$: the first weight matrix in the scenario where we're using $n$ hidden nodes
$W_n^{(2)} \in \mathbb{R}^{n \times 2}$: the second weight matrix in the scenario where we're using $n$ hidden nodes
$g$: first activation function
$h$: second activation function


Then, given an input vector $x \in \mathbb{R}^d$, our neural network will generate output $f_n(x) = h(W_n^{(2)} (g(W_n^{(1)}x)))$.

Now, in general we of course expect Neural Networks to only find a local optimum, but if you want a robust solution you'll want it to be able to handle the worst case, and the worst case for your "adversarial" task is when the Neural Network manages to find the global optimum. So, we'll assume it can find the global optimum.

Essentially, what you're looking for is a dataset $D_n$ containing a number of input vectors $x$, such that:


$f_n(x) = h(W_n^{(2)} (g(W_n^{(1)}x)))$ provides good / optimal results (after training to the global optimum)
$f_{n - 1}(x) = h(W_{n - 1}^{(2)} (g(W_{n - 1}^{(1)}x)))$ provides poor results (even after training to the global optimum of this setup).


In other words, you want to find a collection of vectors $x$ such that it becomes impossible to find a collection of weights in the $n - 1$ case where $f_{n-1}(x)$ is a good approximation of $f_n(x)$. You want to make it impossible that $f_{n - 1}(x) \approx f_n(x)$.



Now this has turned into a clear mathematical problem. I'm not familiar with any established methods in mathematics to solve a problem like this, maybe there are though. My best guess at this point in time would be a procedure like the following:


Generate random "ground truth" versions of the weight matrices $W^{(1)*}_{n}$ and $W^{(2)*}_{n}$ (just completely random matrices).
Generate completely random input vectors $x$. Compute the corresponding ground truth labels as $f^*_n(x) = h(W_n^{(2)*} (g(W_n^{(1)*}x)))$.
Hope that the neural network with $n$ hidden nodes can recover the ground truth weight matrices that were previously generated randomly.
Hope that the neural network with $n - 1$ hidden nodes cannot find an accurate approximation.


In theory, the MLP with $n$ hidden nodes should be able to learn the exact ground truth function. In theory, under certain conditions, the MLP with $n$ hidden nodes should not be able to learn the exact ground truth function. I suspect those "certain conditions" would be that the rows/columns of the weight matrices should be linearly independent, which is likely with randomly generated matrices, but I'm not 100% sure on this. Even if it can't learn the exact ground truth, it may still be capable of learning an approximation... there may be ways to find upper bounds on how close such an approximation could get, but I'm not sure.