PCM1804 Output distorted
The ADC is returning the data as signed (two's complement).  But you're treating it as if it was unsigned.

https://en.wikipedia.org/wiki/Two%27s_complement

Here's a table (taken directly from Wikipedia) that explains what happens:

Bits   Signed    Unsigned

011    3         3
010    2         2
001    1         1
000    0         0
111    -1        7
110    -2        6
101    -3        5
100    -4        4


The "distortion" you see is caused by that transformation.

There should be some option in the software you use where you can select if the data is signed or unsigned.  Anyone who knows more about that is free to edit my answer.