Identify DC motor parameters using Least Squares Estimator
You need to find out how the input and output relate in the FREQUENCY space.
The transfer function is your frequency model.

Here is the equation you should use if you want to fit your data to a first order filter. 

  

The input is a voltage you control.
The output is the value from the encoder, which you probably have to convert to rotational postion or rotational velocity.

What your doing is called system identification, and it's easy to do if you have a model with a low order.

So take your data, vary the input perhaps by incrementing the voltage in steps (like 0.5V). Then get the output in rotational velocity. If the motor speed starts to trail off as the voltage gets higher, then a first order transfer function would be great to describe this behavior.

 

Then in matlab, figure;plot( Input, Output) (I think I have the axis right) if you see something that has an amplitude like the picture shown from your data then you are on the right track. (remember the input is volts, the output is rotational velocity.)

Now you need to come up with parameters from your low pass filter model. You could start punching values into tf([a],[tau a]) by hand to match your data, or you could use an algorithm. Here is a good tutorial on how to do this

Make sure you pay attention to units, your units will be in volts vs rpm or something like that. If your model doesn't look like a low pass filter and it has a resonance point, then you need to switch to a higher order model. The motor may respond like a bandpass (it doesn't turn at low speeds and then has a flat passband and then falls off the more voltage that you put into it) Then you would need to switch to a bandpass transfer function.