Multichannel AoE Hardware
You don't want to send a single packet for every sample. That's because the packet has headers as well, and that means inefficiency in the transfer. Putting several samples into a single packet will both increate the efficiency and get you over your minimum packet size. The upper limit on the number of samples is the maximum latency that you can tolerate. At 48kHz, adding one more sample into your packets would increase the latency by ~21\$\mu\$s. You should also calculate the latency introduced by the whole system end-to-end. Having 8-16 samples per packet would increase your efficiency above 70-80%.
I'd go for a microcontroller with a built-in Ethernet MAC. Having a built-in PHY as well would be a nice plus, however, that extra constraint would decrease your choice by large. There many-many microcontrollers with a built-in 100Mbps MAC, but only a few with a built-in PHY as well.