Hugging Face's logo

khouloudCh15
/
multitaskwav2vec2

Safetensors
English
wav2vec2
audio
speech
emotion-recognition
Model card Files Community
khouloudCh15 commited on
Commit
b9db7a4
·
verified ·
1 Parent(s): af687c6

Upload 9 files

Browse files
Files changed (9) hide show
  1. README.md +120 -3
  2. config.json +130 -0
  3. gitattributes +35 -0
  4. model.safetensors +3 -0
  5. preprocessor_config.json +10 -0
  6. special_tokens_map.json +6 -0
  7. tokenizer_config.json +50 -0
  8. training_args.bin +3 -0
  9. vocab.json +34 -0
README.md CHANGED
@@ -1,3 +1,120 @@
1
- ---
2
- license: apache-2.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ tags:
4
+ - audio
5
+ - speech
6
+ - emotion-recognition
7
+ - wav2vec2
8
+ datasets:
9
+ - TESS
10
+ - CREMA-D
11
+ - SAVEE
12
+ - RAVDESS
13
+ license: mit
14
+ metrics:
15
+ - accuracy
16
+ - f1
17
+ ---
18
+
19
+ # wav2vec2-emotion-recognition
20
+
21
+ This model is fine-tuned on the Wav2Vec2 architecture for speech emotion recognition. It can classify speech into 8 different emotions with corresponding confidence scores.
22
+
23
+ ## Model Description
24
+
25
+ - **Model Architecture:** Wav2Vec2 with sequence classification head
26
+ - **Language:** English
27
+ - **Task:** Speech Emotion Recognition
28
+ - **Fine-tuned from:** facebook/wav2vec2-base
29
+ - **Datasets:** Combined emotion datasets
30
+ - [TESS](https://www.kaggle.com/datasets/ejlok1/toronto-emotional-speech-set-tess)
31
+ - [CREMA-D](https://www.kaggle.com/datasets/ejlok1/cremad)
32
+ - [SAVEE](https://www.kaggle.com/datasets/barelydedicated/savee-database)
33
+ - [RAVDESS](https://www.kaggle.com/datasets/uwrfkaggler/ravdess-emotional-speech-audio)
34
+
35
+ ## Performance Metrics
36
+ - **Accuracy:** 79.57%
37
+ - **F1 Score:** 79.43%
38
+
39
+ ## Supported Emotions
40
+ - 😠 Angry
41
+ - 😌 Calm
42
+ - 🤢 Disgust
43
+ - 😨 Fearful
44
+ - 😊 Happy
45
+ - 😐 Neutral
46
+ - 😢 Sad
47
+ - 😲 Surprised
48
+
49
+ ## Training Details
50
+
51
+ The model was trained with the following configuration:
52
+ - **Epochs:** 15
53
+ - **Batch Size:** 16
54
+ - **Learning Rate:** 5e-5
55
+ - **Optimizer:** AdamW
56
+ - **Weight Decay:** 0.03
57
+ - **Gradient Accumulation Steps:** 2
58
+ - **Mixed Precision:** fp16
59
+
60
+ For detailed training process, check out the [Fine-tuning Notebook](https://colab.research.google.com/drive/1VNhIjY7gW29d0uKGNDGN0eOp-pxr_pFL?usp=drive_link)
61
+
62
+ ## Limitations
63
+
64
+ ### Audio Requirements:
65
+
66
+ - Sampling rate: 16kHz (will be automatically resampled)
67
+ - Maximum duration: 1 minute
68
+ - Clear speech with minimal background noise recommended
69
+
70
+ ### Performance Considerations:
71
+
72
+ - Best results with clear speech audio
73
+ - Performance may vary with different accents
74
+ - Background noise can affect accuracy
75
+
76
+
77
+ ## Demo
78
+ https://huggingface.co/spaces/Dpngtm/Audio-Emotion-Recognition
79
+
80
+ ## Contact
81
+ * **GitHub**: [DGautam11](https://github.com/DGautam11)
82
+ * **LinkedIn**: [Deepan Gautam](https://www.linkedin.com/in/deepan-gautam)
83
+ * **Hugging Face**: [@Dpngtm](https://huggingface.co/Dpngtm)
84
+
85
+ For issues and questions, feel free to:
86
+ 1. Open an issue on the [Model Repository](https://huggingface.co/Dpngtm/wav2vec2-emotion-recognition)
87
+ 2. Comment on the [Demo Space](https://huggingface.co/spaces/Dpngtm/Audio-Emotion-Recognition)
88
+
89
+ ## Usage
90
+
91
+ ```python
92
+ from transformers import Wav2Vec2ForSequenceClassification, Wav2Vec2Processor
93
+ import torch
94
+ import torchaudio
95
+
96
+ # Load model and processor
97
+ model = Wav2Vec2ForSequenceClassification.from_pretrained("Dpngtm/wav2vec2-emotion-recognition")
98
+ processor = Wav2Vec2Processor.from_pretrained("Dpngtm/wav2vec2-emotion-recognition")
99
+
100
+ # Load and preprocess audio
101
+ speech_array, sampling_rate = torchaudio.load("path_to_audio.wav")
102
+ if sampling_rate != 16000:
103
+ resampler = torchaudio.transforms.Resample(orig_freq=sampling_rate, new_freq=16000)
104
+ speech_array = resampler(speech_array)
105
+
106
+ # Convert to mono if stereo
107
+ if speech_array.shape[0] > 1:
108
+ speech_array = torch.mean(speech_array, dim=0, keepdim=True)
109
+
110
+ speech_array = speech_array.squeeze().numpy()
111
+
112
+ # Process through model
113
+ inputs = processor(speech_array, sampling_rate=16000, return_tensors="pt", padding=True)
114
+ with torch.no_grad():
115
+ outputs = model(**inputs)
116
+ predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)
117
+
118
+ # Get predicted emotion
119
+ emotion_labels = ["angry", "calm", "disgust", "fearful", "happy", "neutral", "sad", "surprised"]
120
+ predicted_emotion = emotion_labels[predictions.argmax().item()]
config.json ADDED
@@ -0,0 +1,130 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "facebook/wav2vec2-base-960h",
3
+ "activation_dropout": 0.1,
4
+ "adapter_attn_dim": null,
5
+ "adapter_kernel_size": 3,
6
+ "adapter_stride": 2,
7
+ "add_adapter": false,
8
+ "apply_spec_augment": true,
9
+ "architectures": [
10
+ "Wav2Vec2ForSequenceClassification"
11
+ ],
12
+ "attention_dropout": 0.1,
13
+ "bos_token_id": 1,
14
+ "classifier_proj_size": 256,
15
+ "codevector_dim": 256,
16
+ "contrastive_logits_temperature": 0.1,
17
+ "conv_bias": false,
18
+ "conv_dim": [
19
+ 512,
20
+ 512,
21
+ 512,
22
+ 512,
23
+ 512,
24
+ 512,
25
+ 512
26
+ ],
27
+ "conv_kernel": [
28
+ 10,
29
+ 3,
30
+ 3,
31
+ 3,
32
+ 3,
33
+ 2,
34
+ 2
35
+ ],
36
+ "conv_stride": [
37
+ 5,
38
+ 2,
39
+ 2,
40
+ 2,
41
+ 2,
42
+ 2,
43
+ 2
44
+ ],
45
+ "ctc_loss_reduction": "sum",
46
+ "ctc_zero_infinity": false,
47
+ "diversity_loss_weight": 0.1,
48
+ "do_stable_layer_norm": false,
49
+ "eos_token_id": 2,
50
+ "feat_extract_activation": "gelu",
51
+ "feat_extract_dropout": 0.0,
52
+ "feat_extract_norm": "group",
53
+ "feat_proj_dropout": 0.1,
54
+ "feat_quantizer_dropout": 0.0,
55
+ "final_dropout": 0.1,
56
+ "gradient_checkpointing": false,
57
+ "hidden_act": "gelu",
58
+ "hidden_dropout": 0.1,
59
+ "hidden_dropout_prob": 0.1,
60
+ "hidden_size": 768,
61
+ "id2label": {
62
+ "0": "LABEL_0",
63
+ "1": "LABEL_1",
64
+ "2": "LABEL_2",
65
+ "3": "LABEL_3",
66
+ "4": "LABEL_4",
67
+ "5": "LABEL_5",
68
+ "6": "LABEL_6",
69
+ "7": "LABEL_7"
70
+ },
71
+ "initializer_range": 0.02,
72
+ "intermediate_size": 3072,
73
+ "label2id": {
74
+ "LABEL_0": 0,
75
+ "LABEL_1": 1,
76
+ "LABEL_2": 2,
77
+ "LABEL_3": 3,
78
+ "LABEL_4": 4,
79
+ "LABEL_5": 5,
80
+ "LABEL_6": 6,
81
+ "LABEL_7": 7
82
+ },
83
+ "layer_norm_eps": 1e-05,
84
+ "layerdrop": 0.1,
85
+ "mask_feature_length": 10,
86
+ "mask_feature_min_masks": 0,
87
+ "mask_feature_prob": 0.0,
88
+ "mask_time_length": 10,
89
+ "mask_time_min_masks": 2,
90
+ "mask_time_prob": 0.05,
91
+ "model_type": "wav2vec2",
92
+ "num_adapter_layers": 3,
93
+ "num_attention_heads": 12,
94
+ "num_codevector_groups": 2,
95
+ "num_codevectors_per_group": 320,
96
+ "num_conv_pos_embedding_groups": 16,
97
+ "num_conv_pos_embeddings": 128,
98
+ "num_feat_extract_layers": 7,
99
+ "num_hidden_layers": 12,
100
+ "num_negatives": 100,
101
+ "output_hidden_size": 768,
102
+ "pad_token_id": 0,
103
+ "proj_codevector_dim": 256,
104
+ "tdnn_dilation": [
105
+ 1,
106
+ 2,
107
+ 3,
108
+ 1,
109
+ 1
110
+ ],
111
+ "tdnn_dim": [
112
+ 512,
113
+ 512,
114
+ 512,
115
+ 512,
116
+ 1500
117
+ ],
118
+ "tdnn_kernel": [
119
+ 5,
120
+ 3,
121
+ 3,
122
+ 1,
123
+ 1
124
+ ],
125
+ "torch_dtype": "float32",
126
+ "transformers_version": "4.44.2",
127
+ "use_weighted_layer_sum": false,
128
+ "vocab_size": 32,
129
+ "xvector_output_dim": 512
130
+ }
gitattributes ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:780dce74c397bb0cf5876136676bdc3aa2e5408fdc24c66e699cdeb602dfab62
3
+ size 378308536
preprocessor_config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "do_normalize": true,
3
+ "feature_extractor_type": "Wav2Vec2FeatureExtractor",
4
+ "feature_size": 1,
5
+ "padding_side": "right",
6
+ "padding_value": 0.0,
7
+ "processor_class": "Wav2Vec2Processor",
8
+ "return_attention_mask": false,
9
+ "sampling_rate": 16000
10
+ }
special_tokens_map.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "eos_token": "</s>",
4
+ "pad_token": "<pad>",
5
+ "unk_token": "<unk>"
6
+ }
tokenizer_config.json ADDED
@@ -0,0 +1,50 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "0": {
4
+ "content": "<pad>",
5
+ "lstrip": true,
6
+ "normalized": false,
7
+ "rstrip": true,
8
+ "single_word": false,
9
+ "special": false
10
+ },
11
+ "1": {
12
+ "content": "<s>",
13
+ "lstrip": true,
14
+ "normalized": false,
15
+ "rstrip": true,
16
+ "single_word": false,
17
+ "special": false
18
+ },
19
+ "2": {
20
+ "content": "</s>",
21
+ "lstrip": true,
22
+ "normalized": false,
23
+ "rstrip": true,
24
+ "single_word": false,
25
+ "special": false
26
+ },
27
+ "3": {
28
+ "content": "<unk>",
29
+ "lstrip": true,
30
+ "normalized": false,
31
+ "rstrip": true,
32
+ "single_word": false,
33
+ "special": false
34
+ }
35
+ },
36
+ "bos_token": "<s>",
37
+ "clean_up_tokenization_spaces": true,
38
+ "do_lower_case": false,
39
+ "do_normalize": true,
40
+ "eos_token": "</s>",
41
+ "model_max_length": 1000000000000000019884624838656,
42
+ "pad_token": "<pad>",
43
+ "processor_class": "Wav2Vec2Processor",
44
+ "replace_word_delimiter_char": " ",
45
+ "return_attention_mask": false,
46
+ "target_lang": null,
47
+ "tokenizer_class": "Wav2Vec2CTCTokenizer",
48
+ "unk_token": "<unk>",
49
+ "word_delimiter_token": "|"
50
+ }
training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:12e850157104fb63d82373c93c9020d69e433ce6c0f4b77daf03698dd4b742c0
3
+ size 5304
vocab.json ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "'": 27,
3
+ "</s>": 2,
4
+ "<pad>": 0,
5
+ "<s>": 1,
6
+ "<unk>": 3,
7
+ "A": 7,
8
+ "B": 24,
9
+ "C": 19,
10
+ "D": 14,
11
+ "E": 5,
12
+ "F": 20,
13
+ "G": 21,
14
+ "H": 11,
15
+ "I": 10,
16
+ "J": 29,
17
+ "K": 26,
18
+ "L": 15,
19
+ "M": 17,
20
+ "N": 9,
21
+ "O": 8,
22
+ "P": 23,
23
+ "Q": 30,
24
+ "R": 13,
25
+ "S": 12,
26
+ "T": 6,
27
+ "U": 16,
28
+ "V": 25,
29
+ "W": 18,
30
+ "X": 28,
31
+ "Y": 22,
32
+ "Z": 31,
33
+ "|": 4
34
+ }