SI tools for SDI video signals
This looks like a shopping question!

That said, for 3G-SDI I would probably not bother simulating it, by the time you get the connector model really right (And how would you be sure) you may as well just prototype the relevant bits and measure them (12G is a whole other ball game). 

Don't forget that SDI extends pretty much down to DC on the pathologicals so your simulation runs will take some time, but I see no reason to think MWO could not do it.

Keep the layout compact, trace length you don't have does not cause you problems. 

Most of the line equaliser parts have suggested layouts in the datasheets, and the eval board documentation will usually give you a workable layer stack. 

Oh yea, be careful about that 4.7uF coupling capacitor and its pad geometry, you often want to knock a cutout in the L2 ground plane to reduce the parasitic cap. 

You will need access to something that measures physical layer eye and both timing and alignment jitter, none of which are exactly cheap.