PIC18F4580 - PORTD not being read correctly
I think it's the comparators still being attached - try adding:

CMCON = 0x07;

If you notice in the table CM2:CM0 have to be 111 (rather than 000) to have the pins disconnected and digital. Plus they correspond to pins RD0:RD3: