Question of pretraining text-generation task, it seems that pretraining is not work for a small model?
1, I pad some zeros in the input tokens for multi sentences. The output positions of output tokens should be exactly same to the input tokens, which means I should keep the padding zeros in the output tokens.
2, The pretraining time should be longer.