SpeechFake: A Large-Scale Multilingual Speech Deepfake Dataset Toward Cutting-Edge Speech Generation Methods
dataset;deepfake detection;anti-spoofing;speech generation
Yes
No ethics review needed.
- As noted in the weaknesses, additional analyses would elevate the paper’s contribution. The inclusion of diverse use cases for the dataset could also significantly enhance its impact and utility.
- Why do TTS-Tortoise and VC-OpenVoiceTTS contain significantly more data than other systems?
- Why is BD-UT excluded (or separated) from the result discussion table? Including access dates or version details for the API services would also be advisable, as these may vary over time.
The proposed database stands out for its extensive coverage of languages and speech generation tools, providing a robust foundation for deepfake detection research. Additionally, the dataset includes rich metadata, which is invaluable for in-depth analysis of anti-spoofing performance or targeted optimizations. The exploration of generator types and the effects of language and speaker variability is particularly engaging and relevant to recent advances in the field.
The paper introduces a novel database for speech deepfake detection, comprising a large-scale collection of samples generated by 40 different speech synthesis tools across 46 languages. Benchmark results accompany the dataset, with detailed analyses provided on performance across various subsets, each with a specific focus.
While the dataset is comprehensive, the overall scope is closely aligned with previous work in the field, as acknowledged in Table 1. Although the paper offers a solid contribution, it represents an incremental rather than groundbreaking addition to the anti-spoofing community. The reviewer appreciates the authors' efforts but suggests that more extensive analyses or use cases could further strengthen the contribution. Potential extensions might include exploring additional anti-spoofing model variants, conducting detailed error analyses, or leveraging the dataset for novel applications using pre-trained anti-spoofing models.
Yes
No ethics review needed.
- Generation Process Details: It would be helpful if the authors could clarify how they determine the number of utterances generated using different speech generation tools for a particular original utterance.
- Choice of Test Sets (Table 3): Referring to Table 3, could the authors provide insights into why test sets from 2019 were selected rather than more recent versions? In fact, evaluating on all available benchmarks might have provided a more comprehensive analysis.
- Performance on "Other" Datasets (Table 3): Referring again to Table 3, could the authors explain why the BD (combined set) does not consistently achieve better performance on the "Other" testing datasets compared to its individual subsets?
- Selection of 40 Systems: What was the rationale behind selecting 40 speech generation systems? It would be valuable to understand if all these systems can produce natural-sounding speech.
- Human Verification of Generated Samples: It would also be useful to know if the dataset was subjected to any human verification, as generative models often produce a variety of artefacts, including silences or truncated words. Verifying whether the generated samples are indeed suitable deepfake candidates would add credibility to the dataset.
- SpeechFake Dataset: A large-scale and diverse dataset for speech deepfake detection, encompassing a wide range of speech generation methods. The dataset consists of 3 million utterances, totalling 3,000 hours. It would be beneficial if the authors open-source the entire dataset (including hidden), rather than just a subset.
The paper introduces SpeechFake, a large-scale multilingual dataset designed explicitly for speech deepfake detection. It offers a vast collection of over 3 million samples (amounting to 3,000 hours of audio) generated using 40 advanced speech synthesis tools across 46 languages. The paper provides a comprehensive baseline evaluation for SpeechFake, analyzing factors like generation methods, language diversity, and speaker variation.
- Multilingual Focus: While the work prioritizes multilingualism, it primarily focuses on English and Chinese. Training datasets are lacking for 37 out of the 46 languages, with only a set of 5,000 utterances released for testing purposes for each of these 37 languages.
- Coverage of Speech Generation Tools: Although the dataset includes 40 different speech generation tools, it does not encompass all current techniques, nor does it provide a way to account for future techniques that may emerge. This limitation suggests that the dataset may not remain fully relevant over time and could require updates or recreation as the field advances.
Yes
There may be no violation here, but I am a bit concerned that the paper doesn't fully specify the license that will be used when releasing the dataset publicly.  The appendix only states that the authors have checked the licenses of the deepfake generation software that they used, and thus ensured that the dataset "complies with non-commercial research usage policies".  Considering that the dataset contains deepfakes of real people (if I've understood correctly), I think more detail is needed about how the authors ensure this and how the data will be licensed.
Yes, Privacy, security and safety
- How is the quality filtering done?  The text states "generated speech with noticeable distortions, excessive noise, or unnatural artifacts is discarded".  Is this done automatically, and if so how?

- The detection performance results in Tables 5 and 6 are much better for all settings than in Tables 3 and 4.  Why is that?  For example, why are the English and Chinese results in Table 4 much better than those in the BD-EN and BD-CN columns of Table 3?

- The extremely low EERs in Table 5 suggest that the multilingual dataset is too easy and has very limited value for deepfake detection research.  Is this a misunderstanding of the results?  Please correct me if I'm missing something.

- For the settings with very low EERs, it would be helpful to include confidence intervals or significance estimates.  It is hard to tell whether we can really draw conclusions based, for example, on the differences between speaker settings in Table 6.

More minor details:

- The title is a bit hard to parse.  I think the word "toward" should probably be something else, e.g. "generated using".

- The third contribution in the bullet list (at the end of the introduction) doesn't seem like an independent contribution (more like an elaboration of the others).

- In Table 1, what does "-" mean?

- The first sentence in Related Work suggests that TTS and VC are the only two speech generation tasks (though NV is added on later).  However, there are other tasks like speech content editing, style conversion, and enhancement.

- The related work sub-section on existing datasets doesn't explicitly state what is new/different about the proposed one (though this is briefly stated elsewhere).

- In the appendix, what does "speaker information" include?  The description is "details about the speaker or the generated voice identity", but I'm not sure what this means.

- In the appendix, the figures and surrounding descriptions comparing waveforms/mel spectrograms are unclear and make questionable statements (as far as I can tell) about the examples shown.  For example, A.2 states that the mel spectrograms show "variations in energy distribution, frequency patterns, and clarity".  What exactly is meant by this?  Another similar statement is "the resulting mel-spectrograms share some common characteristics, particularly in their overall structure and energy distribution across frequencies".  What does "overall structure" and "energy distribution" mean, and how do you measure that they are similar here vs. in other examples across multiple speakers?  Yet another statement of this type is about "... distinct characteristics, such as pitch, timbre, and pacing, which are clearly reflected in the waveforms and mel-spectrograms".  How do you measure the timbre in the waveforms or mel spectrograms?  On a more minor note, labeling the axes in the waveform and spectrogram plots would help.
+ The paper addresses an important problem.

+ The study of performance across multiple dimensions (languages, speakers, seen/unseen generation methods) is valuable.

+ The dataset will probably be useful for research beyond deepfake detection, like data augmentaiton.
The paper introduces a new dataset, SpeechFake, for training and evaluating speech deepfake detectors.  The dataset is larger than previous such datasets, and includes deepfakes generated by a larger number (40) of speech generation systems, including text-to-speech (TTS), voice conversion (VC), and neural vocoders (NV).  The paper also provides baseline results using existing deepfake detection models, demonstrating that training the detectors on SpeechFake is more effective across multiple evaluation sets than training on a previous such dataset, ASV19, and measuring the detection performance across speakers, languages, and deepfake generation methods.
- The paper doesn't include enough empirical comparisons to make a convincing case for the value added of the new dataset.  It includes a comparison of training on the new dataset vs. on ASV19, but not vs. other existing datasets, or vs. combinations of existing datasets, which might provide a similar generalization benefit.

- The contribution is a bit narrow for ICLR.  A dataset + baseline paper like this would be a better fit in a speech conference or related workshop.

- The description of dataset construction needs more detail, e.g. how multiple speaker IDs were sampled and generated with each model.  Ideally the paper would include an "algorithm" for sampling examples using each method.
@inproceedings{
anonymous2024speechfake,
title={SpeechFake: A Large-Scale Multilingual Speech Deepfake Dataset Toward Cutting-Edge Speech Generation Methods},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=GpUO6qYNQG},
note={under review}
}
As speech generation technology continues to evolve, the risk of misuse through deepfake audio has become a pressing concern, which underscores the critical need for robust detection methods.  However, many existing speech deepfake datasets fall short in terms of size, diversity, and linguistic coverage, limiting the ability of models to generalize effectively to unseen deepfakes. To address these limitations, we present SpeechFake, a large-scale dataset specifically designed for speech deepfake detection. With over 3 million deepfakes totaling more than 3,000 hours of audio, SpeechFake was generated using 40 different speech generation tools, including cutting-edge techniques, and spans 46 languages. This paper provides a detailed overview of the dataset’s composition and statistics, emphasizing its scale and diversity. Additionally, we establish baseline results for SpeechFake and explore how factors such as generation methods, language diversity, and speaker variation influence detection performance. We believe SpeechFake will be a valuable resource for advancing speech deepfake detection research, offering opportunities to explore new detection strategies and improve model robustness across diverse and evolving generation techniques. The dataset will be publicly available soon.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
dataset deepfake detection anti-spoofing speech generation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/dca99af26f2c3b25bbc06ef81edde84c6b29316e.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
SpeechFake: A Large-Scale Multilingual Speech Deepfake Dataset Toward Cutting-Edge Speech Generation Methods
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission