Designing Butterworth Filter
I'm assuming you are using Matlab to calculate the filter, judging from the syntax and function parameters.

butter calculates a normalized low pass filter response. If you are in the z domain, the actual cut-off frequency depends on your nyquist frequency (half your sampling frequency).
If you sample with say 2MSPS, you have a Nyquist frequency of 1MHz and butter with Wn=0.5 would have a cut-off frequency of 500kHz.
See also this discussion on matlab central.

You should read "doc butter" to get started and might also want to read a textbook on filter design. 
From the documentation:

[b,a] = butter(n,Wn) designs an order n lowpass digital Butterworth filter with normalized cutoff frequency Wn. It returns the filter coefficients in length n+1 row vectors b and a, with coefficients in descending powers of z.