2D convolution on 32x32 grayscale image on FPGA using verilog for inference of CNN
The problem of processing a 2-d kernel of data over a large dataset (not just convolution) comes up so regularly in HD video processing that I came up with a generic way of handling it that I use all the time.

I developed a generic "kernel generator" that uses line buffers and registers to present all of the input data for a given output pixel in parallel. An N&times;N kernel requires N-1 line buffers and N-1 registers. It assumes that the data is arriving in "raster-scan" order, like a TV signal.







simulate this circuit &ndash; Schematic created using CircuitLab

The next stage could be your multipliers, but more often than not, I need to handle the edges in some special way, such as zeroing out the values that fall outside the input data, or reflecting the data across the edge, or whatever. Therefore, I have some standard modules (N2 pixels in, N2 pixels out) that consist of counters and multiplexers that do this edge processing before passing the data to the actual data processing module.

For a convolution, you can do all of the multiplies in parallel in the same clock period, but then the adds will have to be pipelined. For example, if you can only add two numbers in one clock period, you'll need a "tree" of adders that's 5 levels deep for a 5&times;5 kernel. If you can add three or four numbers at a time, you will only need 3 levels of pipeline.

Obviously, the same kernel generator can feed multiple convolutions in parallel if that's what you're doing, but I second Harry Svensson's notion of using FFT techniques if you're doing more than a few of them.