Frequency deviation in VCO chips
To produce an FM modulated carrier using a VCO, you directly modulate the frequency control voltage. 

FM demodulation by VCO requires a little more, closing a PLL around the VCO, so that the VCO tracks the incoming signal, and the demodulated signal is the voltage on the VCO.

The obvious first order parameters to take account of are the voltage to frequency gain, and the range over which both will operate. You would use these to set the gain and range, deviation is just the maximum variation in frequency within the range. You choose a deviation within this range to suit your system.

In order to set the PLL parameters, you would also need to know the maximum modulation rate, and keep the PLL bandwidth well below this. You would need to choose a VCO that was capable of (rule of thumb for PLL stability) a rate at least 5 times higher than the system modulation rate.

Some important second order parameters that affect the quality of the modulator/demodulator you produce are the voltage to frequency linearity, which will affect the distortion, the control voltage noise, and if you want the modulation to go down to DC, the control voltage drift and tempco.

It might be easier to start with a VCO like that found in a 4046 PLL IC. This is specified for linearity and drift, and already has the PLL circuitry on chip (apart from the filters) to complete the demodulator. For the modulator, you can simply ignore the PLL section.

Modulation index is deviation/rate. You can get any index you like from 0 to infinity by manipulating the rate or deviation down to zero, so that's not really a design criterion. Depending on the application, you find different modulation indexes. Digital transmission tends to use indexes in the order of 1 for bandwidth efficiency, in FM audio the maximum index varies between a few and a few thousand depending on audio frequency.