Music/Speech Classification: How are frame based features used in a feature vector?
Let us say you have a set of songs you want to classify by genre.

There are two approaches if the classification algorithm you use works on unidimensional vectors (eg: neural networks, SVM...):


Either you have one feature vector per song; and your feature vector contains statistics (mean, standard deviation, etc.) summarizing all the MFCCs computed at the frame level. In this case, your training set contains one vector per song; and during classification, the classification algorithm is run once per song.
Or you have one feature vector per frame. In this case, your training set is made of frames; and the classification algorithm is run once per frame. Voting is used to select the label for the whole songs. It is common to weight the votes to give more important to the middle of the signal (for example, for musical genre classification, the song might start with a slow/soft intro which might bias the classification).


A particularity of both solutions (the so-called "bag of frames" representation) is that they are invariant to permutations in the sequence of MFCC vectors - they do not take into account the temporal structure of the signal. Another approach is to use machine learning algorithms or models (such as HMMs) which can intrinsically deal with time-series - in which case your representation is the sequence of feature vectors.

The approach you describe in which MFCC vectors for each frame are concatenated to yield a gigantic feature vector is not practical: the large feature vectors would make the learning task computationally extensive, it would yield feature vectors of different dimensions for signals of different durations; and two signals identical modulo a delay would have very different feature vectors.