is it possible to convert a 328 design to use a 32U4?
Just had a quick look at the datasheets, and the main difference I see physically is the 32u4 is 44 pin package and the 328 is either 32 or 28 pin.
Also the 32U4 seems to have a lot more peripherals.

So based on that yes you will have to modify your circuit. Atmel are fairly consistant with ports and what their alternate functions are though.

eg SPI on portB, uart on portD

But you are just going to have to compare the pinouts on the datasheets see how they differ. And adjust pin connections accordingly.

Datasheets are easy to find. Good Luck!