AM335x FPGA Memory Integration
I will make some assumptions: since this is mixing audio, you will want to sequentially read all of the ADCs at some fixed synchronous rate (like 96kHz), and sequentially write all of the DACs at the same rate. I think the PRU will be the easiest way to implement a fast data-pipe to/from an FPGA.

There are two PRU processors in a AM335x Sitara processor, and each has 32 input pins and 32 output pins. Since you would probably need some control signals and are likely using 24-bit codecs, that should work out nicely. You could dedicate one PRU to receive data and the other to send data. With tight C code, I estimate that you could get at least 10 samples per microsecond in each direction (30 megabytes/second), and more with assembly language.

I have put multiple I2S audio interfaces into an FPGA, and it's not that hard. As you said, the codecs could move data to/from FPGA sram, and after each codec cycle (every 10.417 microseconds at 96kHz) the PRUs could read/write the data as a contiguous block. A simple sequencer in the FPGA could respond to strobes from the PRU to walk through the block of data. In the Sitara, a Linux process can allocate a block of memory to share with the PRU. I've put C structures and arrays into shared ARM/PRU memory to allow the Linux process and the PRU to share data as C variables.

Hope that helps.