What is the feedforward network in a transformer trained on?
Let's take the common translation task which transformers can be used for as an example: If you would like to translate English to German one example of your training data could be 

("the cat is black", "die Katze ist schwarz"). 

In this case your target is simply the German sentence "die Katze ist schwarz" (which is of course not processed as a string but using embeddings incl. positional information). This is what you calculate your loss on, run backprop on, and derive the gradients as well as weight updates from.

Accordingly, you can think of the light blue feed forward layers of a transformer



as a hidden layer in regular feed forward network. Just as for a regular hidden layer its parameters are updated by running backprop based on transformer $loss(output,target)$ with target being the translated sentence.