Multi modal learning from separate modal datasets
Back in 2012, this was accomplished by Srivastava and Salakhutdinov in "Learning Representations for Multimodal Data with Deep Belief
Nets" where they mixed textual and visual features using the following Deep Belief network architecture:



Note the specially defined hidden layer that can be trained independently.

More recently in 2015, this was accomplished by Huang, Y.; Wang, W. and Wang, L. in "Unconstrained Multimodal Multi-Label Learning" where they use a conditional Restricted Boltzmann Machine applied to creating text labels for images:



Note that each observed variable, t for tags and m for images are connected to h, but there is the built in assumption that m is more frequently available than t.

One problem with both of these approaches is I'm unclear on how well they're going to scale.