System identification on a non linear system
Since your pan-tilt system controls to a velocity it's difficult to back out a transfer function especially if it's a high sensitivity system. So the best way is to close the loop using some type of position sensors - perhaps your pan-tilt system already has these built in?

In any event the feedback will allow you to stabilize the pan-tilt system at an operating point somewhere between the hard-stops which will allow you excite the system without running into the stops. If the open loop transfer function of the pan-tilt system is $$G(s)$$
and you close the loop using a proportional gain of K then the closed loop (position control) system transfer function will be $$T(s)=\frac{KG(s)}{1+KG(s)}$$
When you excite and measure the response of the position control system it will be the transfer function $$T(s)$$
But you are interested in getting $$G(s)$$
Since you know the structure of T(s) and the value of K, you can solve for G(s) in terms of T(s):
$$G(s)=\frac{T(s)}{K(1-T(s)}$$
and that gives you what you were after.

Another word though on nonlinearity. By nonlinearity I've assumed that you were referring to the hard stops (limits). The technique I described above just lets you keep the system in its linear range away from the stops. But if the action of applying an input voltage results in a non-linear velocity -voltage relationship this method will not provide you with a good model. The act of closing the loop will linearize control about the operating point, but you could find that the dynamic response and even stability could be different depending on where you operate the device. In this case you have to separately evaluate the nature of the nonlinear velocity-voltage relation and somehow augment that with the linear model derived above. That's allot more difficult to approach since the nonlinearity might be structured in so many different ways that might not be so obvious.