Is it legal to scrape YouTube videos for training data?
It will depend on the rights of the videos themselves, although probably the terms of service of youtube wouldn't agree with it anyway.

But you have the YouTube 8M dataset, released by Google for research purposes.


  YouTube-8M is a large-scale labeled video dataset that consists of millions of YouTube video IDs and associated labels from a diverse vocabulary of 4700+ visual entities. It comes with precomputed state-of-the-art audio-visual features from billions of frames and audio segments, designed to fit on a single hard disk. This makes it possible to get started on this dataset by training a baseline video model in less than a day on a single machine! At the same time, the dataset's scale and diversity can enable deep exploration of complex audio-visual models that can take weeks to train even in a distributed fashion. 


Edit: And Facebook and MIT just released SLAC (dataset to be released soon, as well as pretrained models for transfer learning).


  This project presents a novel video dataset, named SLAC (Sparsely Labeled ACtions), for action recognition and localization. It consists of over 520K untrimmed videos and 1.75M clip annotations spanning 200 action categories.