AD9850 comparator problem. How to calculate is frequency?
It appears there's a bit of circuitry between the output of the dac and the input of the comparator on that board.  Connect a four channel scope to the outputs and inputs directly on the chip and view the waveforms present.  I suspect you'll find your answer.  

You'll notice in the datasheet that the comparator jitter graph is performed at 40MHz, so it's capable of running at that speed.  Chances are good you have low signal quality at VINP, an improperly set VINN threshold, or both. As the frequency goes up, the circuit impedance reduces the signal power, and it's having a hard time seeing it.