W5100 Ethernet Shield to Custom Board Problem
A bit late to the party but... it may be that you are using pins 1, 2, 3 and 4 instead of 1, 2, 3 and 6 on the RJ45. It is unclear whether you are connecting via a header, in which case you my disregard this, or straight to the RJ45, in which case you need to change your pin:

1. white/orange
2. orange
3. white/green
6. green