how to handle different durations of audio data?
In this case, usually, Normalization is done.
For example in your training and testing data, you need the same shape, so that you should try something like,
mean = np.mean(X_train_features, axis=0)
std = np.std(X_train_features, axis=0)
X_train_features = (X_train_features - mean)/std

Here X_train_features can be a data frame of spectrogram or mfcc features.
The same thing you can apply for testing features. One thing to remember is the number of columns in shape of training and testing should be the same.
You can also check some Kaggle kernels which are related to Audio processing.