Weird results when using MATLAB's $\tt butter$ function?
The answer to your question can mainly be found in the Matlab documentation. The butter function can design filters either in the analog domain as well as in the discrete domain. In order to make an analog filter, you can do so by giving the string 's' as a third parameters. The transfer function can be defined by the zeros/poles/gain or by numerator/denominator coeefficients. The two available syntaxes are


[z,p,k] = butter(n,Wn,'s')
[b,a] = butter(n,Wn,'s')


When designing a filter in the discrete domain, the normalized frequency is a ratio of the cutoff frequency over the biggest frequency that you can represent according to Nyquist theorem. For instance : if you have a sample time of 1ms, hence a sampling frequency of 1Khz, the biggest frequency you can represent is 500Hz.  Designing a filter with a normalized frequency of 0.5 will put the cutoff frequency at 250Hz, since 250/500 = 0.5

In your question, you specify that the cutoff frequency didn't seems to match the -3dB point. Can you give us more detail about your issue. Maybe show us the work you have done until now ?