Objective Noise Classification for speech enhancement
Environmental Sound Classification might be suitable for what you are asking for. Datasets include Urbansound8k, ESC-50 and Audio set. Pretrained classifier like VGGish are available, which cover the 527 classes in Audioset.