Performance of MATLAB's `conv2()` vs. `imfilter()`
First, I think conv2 and imfilter will give you the same result if you change the filter option of imfilter to conv instead of corr. imfilter uses correlation to filter images by default that starts from one side of the image, whereas covolution starts from the other, so there may be some small differences in the filter output.

Secondly, my test shows imfilter on 2D image is faster if the filter size if the same as images. 

clc
data = (rand(200));
kernel = ones(200)/4e4;
tic
blurredData = conv2(data, kernel,'same');
conv2t=toc
tic
blurredData = imfilter(data, kernel,'same','conv');
imfiltime=toc

&gt;&gt; conv2t =

       2.7425


  imfiltime =

       0.1552


Both imfilter and conv2 are hardware optimized and multi-threaded. For floating point inputs, the performance of them are supposed to be roughly the same. But in most cases in image processing, the filter kernel is linearly separable, imfilter does separation of the kernel into row and column vectors. As a result, they are much faster.  Yet if the filter kernel is (1) not linearly separable, or (2) a small kernel such as a 5*5 filter kernel. conv2 wins in the processing time. I guess the reason is some time is spent on checking the separability of the kernel in imfilter. I am not sure whether there are other different mechanisms that leads to the processing time difference between the two.

Finally, if the filter size is the same as image. fft is always the best one (O(nlogn)):

clc
data = (rand(200));
kernel = ones(200)/4e4;
tic
blurredData = conv2(data, kernel,'same');
conv2t=toc
tic
blurredData = imfilter(data, kernel,'same','conv');
imfiltime=toc
tic
D=fft2(data);
K=fft2(kernel);
blurdata=ifft2(D.*K);
fftime=toc


&gt;&gt;conv2t =

          2.7429


  imfiltime =

          0.1327


  fftime =

          0.0164


EDIT

@geometrikal's detailed analysis:
The reason fft is faster than imfilter with same size images is that imfilter pads the inputs (even with 'same'). For small filter kernels, not much padding is needed for imfilter, whereas to use the fft the kernel will have to be padded to the size of the image. In these cases imfilter is faster.