General tips for PID tuning of super low-friction actuator? I realize this question is not directly related to signal processing, however, it's relevant to system analysis which is relevant to most signal processing engineers. There's also no good alternative stack exchange for me to post this question to (please let me know if there is though).

I'm currently struggling to tune an air-bearing actuator. Because this actuator has such low friction, most of the energy put into it will not dampen itself out naturally. Instead, I'm relying on the derivative gain of my PID controller to do that.

However, I'm struggling to achieve a stiff system (high proportional gain) without it going unstable. I'm using a high powered amp with this stage which is plenty capable of creating stiff control, however, when ever I increase the proportional gain the stage goes unstable. My hope was that I could increase the derivative gain in response to this, yet even doing this it doesn't seem to make the system stable. 

What I'm running into is a system that is "squishy" controlled that then goes unstable when I try to increase the proportional gain. 

What exactly am I doing wrong?
1 - Do you have some general idea of what your transfer function looks like? Or at least a frequency response? Could you perform some kind of step response? You could then try to fit an order-2 model on your step response and then try to tune your controller.

2 - You mention that your system is underdamped. I assume that you have underdamped stable poles (in the left-hand plane) in your process transfer function. Perhaps you could place zeroes near the poles to damp the system.

3 - I never use a PID with a "real derivative" i.e. derivative = x[n] - x[n-1], the derivative is always filtered in order to stabilize the system. Otherwise modeling errors and high-frequency noise will be amplified by the derivative which is something you don't want. Or instead of a PID you could use a PI with a lead controller cascaded.

4 - I recommend that you consult this book http://www.cds.caltech.edu/~murray/books/AM05/pdf/am08-complete_22Feb09.pdf