Which model to use for multiclass audio classification?
500 samples for 6 classes is not so much. You should put aside about 100 samples for validation and 100 for testing, leaving 300 samples for training.
I'm assuming that these drum loop are on the order of 1 second long (0.5-5 seconds). Then I would recommend trying a pretrained audio model, which typically have 1 second analysis windows. For example OpenL3, which is a powerful and easy to use audio embedding. It even has pretrained versions trained on audio.
On top of the audio embeddings try a simple linear classifier like LogisticRegression or a RandomForest.