Visual equivalent of VST
It really depends on how much you care about realtime operation.  VSTs tend to be used both for production and performance, but with video, because there's so much more processing involved, things tend to be split between production and performance.  That being said, for production, After Effects plugins are probably the closest analogy.  For performance, probably Freeframe.

Freeframe is a format that's specifically tailored to be cross-application, but on the production side of things, I don't know of anything that transcends applications the way that VSTs do.  Each video production app tends to have its own format.  The main reason I say AE plugins are the closest analogy to VSTs is because AE itself is almost as ubiquitous in the same application price range/platform.

So there's not really an exact analogy (if you consider all the various factors), but there are similar things out there.