What does it mean to 'gradient check a few dimensions for every separate parameter'?
As you might know, biases are used in the CNN only when there is no batch normalization layer is included because this automatically adds a type of bias to the activation outputs.

So, this is what I understood:

Let us assume a layer of CNN (without batch norm) with 50 kernels each of size of 5 x 5. So, total weight updates will be 5 x 5 x 50 = 1250 and total biases updates will be 50 (a bias for each kernel).

For the part "...people combine the parameters into a single large parameter vector for convenience. In these cases, for example, the biases could only take up a tiny number of parameters from the whole vector..." :-


If you merge all these parameters into one single vector and randomly sample the parameters to check gradients for, you may end up checking very few or none of the gradient updates for biases as they make a very less portion of that vector (only 50 out of 1300).


For the part "One issue to be careful with is to make sure to gradient check a few dimensions for every separate parameter..." :-


On the other hand, you can choose some kernels and take all parameters (weights and biases) of only those kernels to check gradient for and so in this case you will definitely be checking gradients of some biases which they say is a good way of doing gradient checking.