A challenge in speech production research is to predict future tongue movements based on a short period of past tongue movements. This study tackles speaker-dependent tongue motion prediction problem in unlabeled ultrasound videos with convolutional long short-term memory (ConvLSTM) networks. The model has been tested on two different ultrasound corpora. ConvLSTM outperforms 3-dimensional convolutional neural network (3DCNN) in predicting the 9 th frames based on 8 preceding frames, and also demonstrates good capacity to predict only the tongue contours in future frames. Further tests reveal that ConvLSTM can also learn to predict tongue movements in more distant frames beyond the immediately following frames. Our codes are available at: this https URL.
@cite_19: A challenge in speech production research is to predict future tongue movements based on a short period of past tongue movements. This study tackles speaker-dependent tongue motion prediction problem in unlabeled ultrasound videos with convolutional long short-term memory (ConvLSTM) networks. The model has been tested on two different ultrasound corpora. ConvLSTM outperforms 3-dimensional convolutional neural network (3DCNN) in predicting the 9th frames based on 8 preceding frames, and also demonstrates good capacity to predict only the tongue contours in future frames. Further tests reveal that ConvLSTM can also learn to predict tongue movements in more distant frames beyond the immediately following frames. Our codes are available at: https: github.com shuiliwanwu ConvLstm-ultrasound-videos.
The contributions of the current work can be summarized as follows: 1) We applied ConvLSTM to predict tongue motions in ultrasound image sequences based on past motions, demonstrating that in most cases ConvLSTM network outperforms the 3DCNN adopted in @cite_19 ; 2) @cite_19 only investigated predictions of tongue motions in the immediately following frames. However, we also tested the performance of our method in predicting tongue motions in more distant ultrasound frames and ConvLSTM consistently outperforms 3DCNN in multiple quantitative comparisons.