ASIC verification of a multiport switch
I think I would approach this by maintaining 16 separate queues (FIFO buffers), one for each of the possible paths through the DUT, since from your description, that is the only sense in which order is preserved. When a packet is transmitted into one of the input ports, it is also placed in the appropriate queue.

When a packet comes out of an output port, check each of the four queues associated with that port and make sure that it is the first packet in one of them. If so, remove it from the queue and keep going.

If not, the level of diagnostics to include in the testbench is up to you. For example, you might search all of the queues to their full depth to determine whether a packet was dropped or misrouted. Or you could simply halt the simulation and debug it interactively from there.