ESC-50 Audio data for binary classifier
On the github page of the ESC-50 dataset, there is a list of tried classifiers as well as links to the relevant papers. The best ones are all using some kind of deep learning, mostly CNNs, and currently the best score is 85.5%. A baseline Random forest (notebook available here) achived 44.3% and SVM (available here) achieved 39.6%.

I would recommend you take a look at the approaches linked above and see if you can improve upon them. However, for higher accuracy you probably need some kind of deep neural network.