Software/Hardware Reset of MCU sometimes makes 24 Bit External ADC conversion goes wrong in Tiva C series
The fact a POR fixes it sometimes indicates to me that something is not being properly initialized. 

I would suggest controlling the /RESET input of the ADC with the Tiva and giving it plenty of time before releasing it from reset- tens or hundreds of ms.