Can BERT be used for sentence generating tasks?
For newbies, NO.

Sentence generation requires sampling from a language model, which gives the probability distribution of the next word given previous contexts. But BERT can't do this due to its bidirectional nature.



For advanced researchers, YES.

You can start with a sentence of all [MASK] tokens, and generate words one by one in arbitrary order (instead of the common left-to-right chain decomposition). Though the text generation quality is hard to control.

Here's the technical report BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model, its errata and the source code.



In summary:


If you would like to do some research in the area of decoding with
BERT, there is a huge space to explore
If you would like to generate
high quality texts, personally I recommend you to check GPT-2.