Verifying Cat5e cabling
As you increase in speed the quality of the construction and length of the cable makes a big difference. CAT5E can support Gigabit speeds, but only when constructed at high quality. While CAT6 can have a slightly poorer construction and still work.

The main thing I would check is how much of the twisted pair you have untwisted. You should have none except for what is in the slots of the connector.

The other things to check is to make sure that your connector is actually rated as CAT5e. Just because your cable is CAT5e doesn't mean your plugs are.

Do you have access to CAT6 connectors? If so try putting them on your CAT5e cable and see if you get improved performance. The CAT6 connectors are designed for a slightly thicker cable, which is not ideal for your situation, but may help you get the untwisted part to be less.