What is the difference between encoders and auto-encoders?
Theory

Encoder


In general, an Encoder is a mapping $f : X \rightarrow Y $ with $X$ Input Space and $Y$ Code Space 
In case of Neural Networks, it is a Generative Model hence a function which is able to compute a Representation out of some input (like GAN)


The point is: how would you train such an encoder network ? 


The general answer is: it depends on what you want your code to be and ultimately depends on what kind of problem the NN has to solve, so let's pick one  


Signal Compression

The goal is to learn a compressed representation for your input that allows to reconstruct the original input minimizing the loss of information 

In this case hence you want the dimensionality of $Y$ to be lower than the dimensionality $X$ which in the NN case means the code space will be represented by less neurons than the input space 

Autoencoder

Focusing on the Signal Compression problem, what we want to build is a system which is able to 


take a given signal with size N bytes 
compress it into another signal with size M&lt;N bytes 
reconstruct the original signal, starting from the compressed representation, as good as possible 


To be able to achiebve this goal, we need basically 2 components 


an Encoder which compresses its input, performing the $f : X \rightarrow Y$ mapping 
a Decoder which decompresses its input, performing the $f: Y \rightarrow X$ mapping 


We can approach this problem with the Neural Network Framework, defining an Encoder NN and a Decoder NN and training them 

It is important to observe this kind of problem can be effectively approached with the convenient learning strategy of unsupervised learning : there is no need to spend any human work (expensive) to build a supervision signal as the original input can be used for this purpose 

This means we have to build a NN which operates essentially between 2 spaces 


the $X$ Input Space 
the $Y$ Latent or Compressed Space 


The general idea behind the training is to make a certain input go along the encoder + decoder pipeline and then compare the reconstruction result with the original input with some kind of loss function 

To define this idea a bit more formally 


The final autoencoder mapping is $f : X \rightarrow Y \rightarrow X$ with 


the $x$ input 
the $y$ encoded input or latent representation of the input 
the $\hat x$ reconstructed input 

Eventually you will get an architecture similar to 





You can train this architecture in an unsupervised way, using a loss function like $f : X \times X \rightarrow \mathbb{R}$ so that $f(x, \hat x)$ is the loss associated to the $\hat x$ reconstruction compared with the $x$ input which is also the ideal result 


Code

Now let's add a simple example in Keras related to the MNIST Dataset 




from keras.layers import Input, Dense 
from keras.models import Model 

# Defines spaces sizes 

## MNIST 28x28 Input 
space_in_size = 28*28

## Latent Space 
space_compressed_size = 32 

# Defines the Input Tensor 
in_img = Input(shape=(space_in_size,))

encoder = Dense(space_compressed_size, activation='relu')(in_img)

decoder = Dense(space_in_size, activation='sigmoid')(encoder)

autoencoder = Model(in_img, decoder)

autoencoder.compile(optimizer='adam', loss='binary_crossentropy')