Downsampling fft results to a graphics display
Ok, here is the answer...hopefully this will help someone else.  Read this thesis:

http://skemman.is/stream/get/1946/15343/37285/3/SS_MSthesis.pdf

This gives a very easy and comprehensive overview of how to do just what I needed. I do believe this particular case is different than the usual downsampling. It fits the situation perfectly.
The thesis describes several methods of downsampling for display purposes. For example the practice of just dropping every nth sample, performing an average over the bucket of samples corresponding to the downsample ratio and more advanced methods.  The one I will be looking at is called Largest Triangle Three Bucket.
Basically you divide your original samples into buckets consisting of samples equal to your downsampling factor.  For example if you have a sample set of 10,000 points and you want to only display 2000, you will divide your original samples into 2000 buckets containing 5 sample each.
You then rank every point in the bucket by calculating the area of a triangle it forms
with the selected point in the last bucket and the average point in the next
bucket.
This seems to work really well in preserving the visual representation of data.