Why did fly-by-wire systems take so long to implement?
A simple answer is that the aircraft control system has ultimate responsibility for survival of the aircraft. If it completely fails, you are as good as dead in most cases. Short of in-flight destruction, hardly anything can be worse.

Thus, full-authority (i.e. without mechanical backup) FBW gained acceptance only when the confidence in its overall reliability was sufficiently high. Naturally, it appeared on military airplanes first, where demands to reliability can be lower (the pilot can eject, after all), while demands to the control system sophistication are often higher.

(Others have already commented that A320 wasn't the first commercial airliner with FBW; Concorde had it much earlier. As for your understanding of FBW, it's basically correct. Sometimes, strain gauges rather than potentiometers are used as control stick sensors: force as the primary input is generally better than displacement).