Algorithm to estimate coherence when measuring a transfer function with a swept sine
For each frequency, divide the measurement time into chunks.  For each chunk, do your estimation of the amplitude at that frequency in both the input and output channels.  Then compute the coherence for this ensemble of measurements at each frequency.  

For example, if you measure at each frequency for Ncyc cycles, then divide these into Navg measurements each of Ncyc/Navg cycles.

A nice reference on this subject is Bendat and Piersol.