What is the fundamental difference between neural networks for classifying and generating data
The formal name for this difference is &quot;generative&quot; vs &quot;discriminative&quot; models.
By default, a supervised learning process using a simple feed-forward neural network and a set of training data with expected answers will produce a discriminative model. It is hard to use such a model to generate content directly.
Differences between discriminative and generative models tend to be at the architecture level or deeper, although a few generative techniques are simple variants or re-purposing of discriminative networks. There is no single &quot;fundamental&quot; difference in terms of NN design. About the only common theme is that generative models are more complex and harder to work with than discriminative ones.
Images

Probably the most popular currently for image generation are Generative Adversarial Networks or GANs. These train a Generator to create an image from random inputs, alongside a Discriminator that tries to spot fakes compared to real images. Training them together results in a generator network that gets progressively better at creating fake data.

Also suitable for image generation are Variational Autoencoders (VAEs) which learn essentially by compressing a set of images down into a small representation, and as a result become able to &quot;uncompress&quot; similar representations.

There are also VAEGANs, which combine VAEs and GANs

There are other models which generate examples from a dataset. For example, Restricted Boltzmann Machines (RBMs) are similar to neural networks, but the neurons fire randomly, and RBMs require a different training process to NNs.

GANs, VAEs, VAEGANs, RBMs can be used to generate data in any simple non-sequential datasets, they are not restricted to just images, but recent work with GANs for example has excelled there.

For a different kind of generator, you can look at how Deep Dream works. The interesting thing here is that it is a modification of a feed forward network that has been trained by supervised learning. Essentially to run Deep Dream, you take an existing image and change it so that it maximises some internal part(s) of the existing neural network, &quot;training&quot; the image as if it were a set of weights.

The difference between Deep Dream and other generative techniques is that it does not generate content that could be in the training set - it is no good for creating &quot;realistic&quot; images.



Text

Recurrent Neural Networks, and specifically Long Short-Term Memory (LSTM) networks can be used to model word sequences by training them to predict the next word. Because the resulting model is probabilistic (e.g. chance of next work being &quot;Hello&quot; is ~0.001%) then a simple method to make this generative is to sample from the predictions randomly, and then feeding them back into the sequence to see what it will predict next. There is a great blog post about this technique by Andrei Karpathy.

Audio and Music

DeepMind's WaveNet architecture is similar to a CNN predicting the next samples of audio from a given input. It can be used in speech or music generation.

LSTMs are a popular choice here, as well as for language models.


One important caveat. Nearly all these models are hard to understand, work with and train, compared to simpler supervised techniques. Probably the easiest to get to grips with if you want to understand them well enough to implement your own versions, are Deep Dream, VAEs and LSTMs.
It is also worth noting that there are many other ways to generate content than neural networks. For instance, sound and music generation has a long history completely independent of recent AI developments.