@Saumya_Mishra (Saumya Mishra) posted a new topic, subject ""Transcription wrong on giving audio from Mozilla Common voice dataset". It read:

   Hi all,  
    I am using “deepspeech-0.6.1-model(Pre-trained model)” to transcript wave files from mozilla common voice dataset but getting incorrect transcription.I want to ask if our model is trained on this dataset than why we are getting incorrect transcription??

Then @lissyx ((slow to reply) [NOT PROVIDING SUPPORT]) said

    Common Voice dataset is quite large both in number of releases and in languages, could you please be more precise on what you tested ?
    
    Also, because a sample was in the training or validation dataset does not ensure it will be 100% correctly recognized.

So @Saumya_Mishra (Saumya Mishra) replied

    Language:-English  
    here I am unable to upload audio here I have tested on sample-000000.wav,sample-000001.wav,sample-000002.wav.  
    Loading model from file deepspeech-0.6.1-models/output_graph.pbmm  
    TensorFlow: v1.14.0-21-ge77504a  
    DeepSpeech: v0.6.1-0-g3df20fe  
    2020-04-08 05:22:51.588842: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA  
    Loaded model in 0.00798s.  
    Loading language model from files deepspeech-0.6.1-models/lm.binary deepspeech-0.6.1-models/trie  
    Loaded language model in 0.000177s.  
    Running inference.  
    **without the data that the article useless** :-by deepspeech_model-V.0.6.1  
    Inference took 2.208s for 3.192s audio file.  
    for sample-000000.wav the correct transcription is “ **Without the dataset the article is useless** ”

Then @lissyx ((slow to reply) [NOT PROVIDING SUPPORT]) said

    Saumya_Mishra:
    
    > Language:-English  
    >  here I am unable to upload audio here I have tested on sample-000000.wav,sample-000001.wav,sample-000002.wav.
    
    You have not answered the question about which common voice release you are referring about. There have been multiples english dataset release.
    
    Saumya_Mishra:
    
    > for sample-000000.wav the correct transcription is “ **Without the dataset the article is useless** ”
    
    I don't know about that specific example, but I have already explained to you that this not unexpected.

Then @Saumya_Mishra (Saumya Mishra) said

    I referred version “en_1488h_2019-12-10” of common voice dataset.

So @lissyx ((slow to reply) [NOT PROVIDING SUPPORT]) replied

    So you expect that a model release from before the common voice release would include the data? No. The 0.6 was trained on previous common voice release.
    
    Mozilla Discourse a écrit :

To which @Saumya_Mishra (Saumya Mishra) replied

    @lissyx,Please let me know which version of common voice dataset to download for deepspeech 0.6 or please share the document where it is given, it would be really helpful.
    
    Thanks in advance.

Then @lissyx ((slow to reply) [NOT PROVIDING SUPPORT]) said

    Saumya_Mishra:
    
    > @lissyx,Please let me know which version of common voice dataset to download for deepspeech 0.6 or please share the document where it is given, it would be really helpful.
    
    I don't know what you want exactly … Unfortunately, the Common Voice team does not yet allows to access older version of the dataset through their website.
    
    Could you please clearly state what you are trying to achieve ?

To which @Saumya_Mishra (Saumya Mishra) replied

    I am sorry for making you confused.
    
    I want the samples in which deepspeech model gives correct predictions without any error in the transcription for a given wave. So,should I have to check it manually by running each sample and checking the transcription?

After which, @lissyx ((slow to reply) [NOT PROVIDING SUPPORT]) added

    Saumya_Mishra:
    
    > I want the samples in which deepspeech model gives correct predictions without any error in the transcription for a given wave.
    
    You are chasing ghosts here. First, we don't have such a list. Second, as I already stated, Common Voice is one of the hardest dataset according to several benchmarks. So even on samples that have been on the training set, it would not be surprising there is small errors like the ones you reported.
    
    The error could also be triggered by the language model.
    
    Saumya_Mishra:
    
    > So,should I have to check it manually by running each sample and checking the transcription?
    
    As of now, yes, that looks like the only solution.
    
    It would really be useful if you articulated clearly why you need that.

@Saumya_Mishra (Saumya Mishra) replied

    Thanks for the information and sorry for the inconvenience caused.

Then @othiele (Olaf Thiele) said

    You really have a strange request, sounds like you are trying to cheat on a benchmark or what else do you need that for?
    
    Usually people want to know how well DeepSpeech can transcribe language and not find perfect examples … Just overfit a small model yourself and it will detect your phrases perfectly

To which @lissyx ((slow to reply) [NOT PROVIDING SUPPORT]) replied

    Saumya_Mishra:
    
    > Thanks for the information and sorry for the inconvenience caused.
    
    So, can we know why you want to do that ? It's frustrating that you don't want to share and thus that we cannot help you correctly.

To which @Saumya_Mishra (Saumya Mishra) replied

    No, it's not like cheating or anything.I want to perform adversarial attack on this model so, that's why I want to take 100% correctly recognized samples .

@Saumya_Mishra (Saumya Mishra) replied

    Ya,sure. I want to perform adversarial attack on this model so, that's why I want to take 100% correctly recognized samples .

So @lissyx ((slow to reply) [NOT PROVIDING SUPPORT]) replied

    Saumya_Mishra:
    
    > I want to perform adversarial attack on this model so, that's why I want to take 100% correctly recognized samples
    
    You could have exposed that from the begining, it would have saved everyone a lot of time.
    
    I guess overfitting on a smaller subset of Common Voice would be the most efficient way to achieve this. You would be more in control of the model, and you could thus perform your attack in a reproductible manner.

Then @othiele (Olaf Thiele) said

    Exactly, that way you could figure out what works best which is a lot harder to see in the larger model.

To which @Saumya_Mishra (Saumya Mishra) replied

    Yes I will definitely take care of this before raising an issue from next time .


{'source': 'discourse', 'url': 'https://discourse.mozilla.org/t/transcription-wrong-on-giving-audio-from-mozilla-common-voice-dataset/57487', 'title': '"Transcription wrong on giving audio from Mozilla Common voice dataset', 'views': 472, 'has_accepted_answer': False, 'word_count': 942, 'like_count': 1, 'reply_count': 14, 'participant_count': 3, 'posts': 18, 'tags': '', 'author': '@Saumya_Mishra (Saumya Mishra)', 'participants': '@Saumya_Mishra (Saumya Mishra), @lissyx ((slow to reply) [NOT PROVIDING SUPPORT]), @othiele (Olaf Thiele)', 'image_url': [None], 'created_at': '2020-04-08T06:17:43.287Z', 'last_posted_at': '2020-04-09T08:47:15.228Z'}