Do you recognise this signal?
That looks like a sine wave with the y-axis wrapped around.  Here's my attempt at recreating it:



This is a plot of the function \$1.25 \cdot \sin(t) - \operatorname{round}(1.25 \cdot \sin(t))\$, where \$\operatorname{round}(x)\$ rounds \$x\$ to the nearest integer.

Perhaps the highest bit of your signal is getting cut off?  That would seem likely to produce such a waveform.