Training a NN classifier on a single\double channel out of a surround sound dataset
Generally the order of a 5.1 channel signal is as follows: L (Left), R (Right), C (Center), LFE (Low frequency enhancement), Ls (Left surround), Rs (Right surround).
What you need is a downmix and there are many downmixing methods. Here's the recommendation of ITU:
$$
L' = L+0.7071C+0.7071Ls
$$$$
R' = R+0.7071C+0.7071Rs
$$
However, multichannel signal may contain some hidden information for your classification task. To get a better performance, I don't think downmixing to single channel is a good idea. Stereo signals may be a compromise between performance and training time.