Does this VCO produce a sine wave or square wave
This VCO outputs a frequency between 5.5 and 6.1 GHz.

It is sort of "implied" that the output signal will be a sinewave because:


a "decent" square wave at that frequency would need a signal bandwidth of several hundreds of GHz. That's because a square wave relies on harmonic frequencies (multiples of the base frequency) to become "square". Read up on Fourier analysis to understand this
this chip is designed to be used in RF transceivers, usually as a local oscillator for mixing up/down RF signals. This works best using sinewaves as RF designers want one frequency to deal with, not one frequency and all its harmonics (like a square wave would have).