OASIS Uncovers: High-Quality T2I Models, Same Old Stereotypes
Stereotype Measurement;Responsible AI;Trustworthy AI;Interpretability
Yes
No ethics review needed.
S3.2 - WALS is quite confusingly described.  In particular (6) doesn't really make sense as an equation.  If E(D) is the SVD that is just estimating the dataset? Do you mean the rank or something else?  Even the motivation doesn't make much sense to me of why this is necessary.

"Existing bias definitions are not applicable for some attributes studied in Tab. 1. For example, in the case of wearing turban, a T2I model needs to depict 50% of the images of Iranian with turban to be unbiased according to Eq. (1), which incorrectly represents Iranian people." - This seems like an odd remark in that this is largely what is done here just conditioned on gender.  What do you see as the primary difference here?
1. Measuring sterotyping is an important and challenging problem for generative models.

2. The diversity of metrics seem quite interesting to get a more complete picture of how bias and stereotypes arise in these models.

3. The paper is fairly thorough in its testing.
In this paper, the authors propose multiple new metrics to understand stereotyping in T2I models.  In particular, they propose a metric for stereotyping that compares the prevalence of a given attribute to a societally-anchored prior probability.  They go on to present numerous other analyses for diversity of images and measuring how differnt stereotypical concepts relate to demographics in latent space.  They show that generally newer models exhibit less stereotyping and that stereotypes arise mostly in the earlier stages of optimization.
1. The biggest weakness in my opinion is that the paper is quite hard to follow for many of the metrics. I believe this is due to making the description of the metrics unnecessarily complex, and I would encourage the authors to try to describe all of the concepts more simply.  In addition to making the paper harder to understand, it also obfuscates the insights and contributions.

2. One of the papers claimed main contributions is measuring stereotyping relative to societal priors on the distribution.  However, as the authors state in the appendix, the prior is often unknown and thus falling back to 50% is a reasonable choice.  That said, this seems to limit the contribution over prior work.

3. The paper does not validate their work with human judgements

4. More minor: the paper only presents the evaluations but do not work on mitigations.

5. The intro could be more clear and sensitive in how it characterizes what are stereotypes, why they matter, and how they define which stereotypes are important.
Yes
No ethics review needed.
- In the abstract the authors said "incorrectly categorizes biases as stereotypes", do you mean the opposite (i.e. "incorrectly categorizes stereotypes as biases")?
- How about using other measures of visual variations of the generated images (e.g. a similarity measure based on feature extraction (e.g. HoG))?
Addressing a research gap
Clear mythology
Application to existing models
The proposed work has a novel contribution in an existing research gap. Defining, separating, and quantifying a measure for stereotypes and biases in Image generation models (from text prompt) is a strong case of Ethical Machine Learning. The authors built a foundation of definitions that allowed them to measure estimates of data and image stereotypes. Then they apply that on well-known Text to Image models to uncover the stereotypes risks of applying those models according to the proposed measures/estimates.
It would have been nice if the writeup included a detailed case study of qualitative analysis of cases of stereotypes
Yes
This work labels people's images to their nationalities, which might sound offensive or unethical to some. Further review from the ethics experts would be necessary.
Yes, Discrimination / bias / fairness concerns
Please, refer to the weaknesses section.
- The paper is well-written, clear, and easy to follow.
- The paper addresses a complex and socially significant problem by evaluating biases and stereotypes in T2I models. Additionally, it provides new methods to trace the origins of these stereotypes in T2I models, offering valuable insights to help mitigate these biases.
- The findings of this work highlight the need for greater inclusion and representation of underrepresented communities, particularly those from regions with limited internet presence, in the development of T2I models.
This paper introduces OASIS, a toolbox for auditing visual stereotypes in text-to-image (T2I) models. OASIS offers novel methods for measuring stereotypes and analyzing their origins within T2I models. OASIS stereotype scores reveal that newer T2I models produce significant stereotypical biases, potentially limiting their applicability.
- This work focuses exclusively on generated images based on nationality, which raises several concerns. For instance, in racially diverse countries like the United States, how can an image indicate that someone is a U.S. citizen without relying on stereotypes and social biases? Additionally, in regions where people from neighboring countries may share similar facial features or cultures (such as North and South Korea), how useful can OASIS be in distinguishing these cases?
- How can OASIS be useful beyond nationality domains? It is not clear in the paper how OASIS can be extended to evaluate other biases in T2I models, such as gender biases, racism, etc.
- Overall, this is a solid piece of work; however, a deeper discussion of its limitations and the specific types of stereotypes for which OASIS may or may not be effective would be beneficial.
Yes
No ethics review needed.
Is this definition of stereotypes correct? Does it not implicitly push the stereotypes in society that cause imbalances to exist? Relying on census data to define an expected rate of a given attribute is perpetuating the stereotypes that exist in today's society. For example, using the example provided in the paper, if there are less women than male doctors in the world, it doesn't necessarily mean that the model is creating a 'stereotype' by generating an equal number of woman and male doctor images. 

For experimental details, what is each point in Figure 3? What does each point correspond to in terms of attribute? 

In Figure 5, there is a claim that "We observe that some models have lower stereotypes at the cost of lower attribute variance". This suggests a trade-off can exist between these two, but there is not much correlation observed for any model across nationalities. I agree that both metrics are important to measure, but it would be great to get more clarification on if there indeed is a trade-off that occurs often. 

I would appreciate any additional discussion about the accuracy of the metrics as well and if they work as intended.
The paper was very well-written, and it was quite easy to follow. I greatly appreciated how the terminology was constantly linked to concrete examples. The general formatting was also intuitive. 

The paper did a good job of thinking about the broader context of this work, thinking a bit more critically about what phrases like biases actually mean and what we should think about when studying these models. Also incorporating real-world census data was a neat idea. 

I liked the results about how the biases are being generated early in the image-generation process.

The formulation for the stereotype score was quite intuitive and seems easy enough to compute at scale when provided with accurate descriptors from these LLMs.
The authors present OASIS, a set of 4 metrics/methods to analyze the presence of stereotypes in T2I models, how those stereotypes form, and the connections between them. The authors use 3 popular models for this analysis. In the paper, OASIS shows the performance of these models on various prompts (e.g. "Show me a picture of a Mexican person") and how certain stereotypes are strongly associated with the prompt, at rates greater than those stereotypes in the real world. The paper shows weaknesses in models in their attempts to prevent stereotypes such as revealing stereotypes when prompts or combined or in how they produce low-variance along the stereotype's dimension. The paper also shows how certain stereotypes are clustered together and at what stage in the image generation process these models form (which is almost always in the early stages). 

In all, OASIS provides a useful toolkit for understanding stereotypes in T2I models and highlights the importance for further work in this very important area.
I feel that overall, there was a lack of bench-marking for the methods presented in this paper beyond providing a few visual examples (which are still useful!). There were a few parts of the paper where I felt this significantly 

1. Computing the stereotype score: How accurate is the probability computation? The intuition of the metrics' formulation is clear, but there is no validation that the score provides accurate results in practice. As the paper mentions, CLIP is a biased VLM on its own, so its not fair to assume that this score will work perfectly. A smaller dataset demonstrating the accuracy of the score could be quite useful. 

2. Spectral variety along a stereotype. There was a nice description of each method for measuring $\delta A$ in the appendix, but its not clear which one is used in the paper and which one is more accurate in practice. 

3. The biases of models like CLIP are well-documented (e.g. Birhana, Prabhu, & Kahembwe, 2021). If CLIP is being used extensively to identify additional biases, I think it makes sense to add more validation about how said biases may or may not be affecting the performance of OASIS. 

4. I understand the SPI claims about how certain stereotypes are intrinsically associated with a concept. However, the paper only provides a handful of examples of where this is occurring. I would appreciate a larger-scale analysis that includes something like 100 contexts generated in some way (i.e. top 100 most common ethnicities), and the average step at which each stereotype attribute occurs. This would be much more convincing than what could be cherry-picked examples. 

I think that the breadth of the experiments/validation were also a bit limited. I understand that gaining access to real-world data for a ground truth distribution is not always feasible, but consistently focusing on broad nationalities feels like it lacks nuance. 

Another broad critique is that certain figures were unclear. I'll bring up those missing details in the following sections
Yes
No ethics review needed.
I would appreciate it if the authors addressed the questions mentioned in the weaknesses section, and I will raise the score if their responses are convincing.
1. The paper tackles the significant problem of stereotypes in T2I models, which has important implications for ethical AI use and social impact.
2. The paper is clearly written and organized. The visual presentations are good. 
3. It proposes StOP and SPI that try to understand the origin of the stereotypes, which may offer further insights.
This paper aims to measure and understand stereotypes in Text-to-Image models. 

* Measurement: it introduces metrics to quantify the presence of stereotypes in generated images by comparing them to real-world distributions, and also assess diversity along stereotypical attributes. 

* Track Origins:  The author attempts to identify hidden associations that T2I model makes with certain concepts and trace how stereotypical attributes emerge over the generation process
1. The authors emphasize that their method is innovative in focusing specifically on stereotypes rather than just bias in the abstract. However, this distinction is not fully explained in the main content, and the methodology still seems quite similar to bias detection to me.

2. The way they identify stereotypes is by prompting LLMs to generate stereotypical attributes for a given concept, but this method is already common in other papers. For example, [1] prompts LLMs to generate stereotypical / anti-stereotypical sentences for certain concepts and then identifies the stereotypical / anti-stereotypical attributes from those sentences.

3. Although the paper includes many formulas, I found that the core idea is actually quite simple, and the formulas only make it harder to follow. If I understand correctly, the main process is: identify a concept, use LLMs to generate stereotypical attributes, generate images, measure the appearance of these attributes, and compare to real-world distributions & similarities across groups. This could be communicated in just a few sentences, which would make the approach clearer.

4. I am concerned with the idea of comparing to real-world distributions. For example, men are dominant in many professions even today. In society, an educated person can be aware of stereotypes to some extent, but telling the model that it’s okay to learn patterns such as “men are more likely to be doctors” because it reflects real-world distributions doesn’t seem fully convincing to me.

[1] chatgpt based data augmentation for improved parameter-efficient debiasing of llms
We propose a toolbox to quantify stereotypes in Text-to-Image models
@inproceedings{
anonymous2024oasis,
title={{OASIS} Uncovers: High-Quality T2I Models, Same Old Stereotypes},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=L6IgkJvcgV},
note={under review}
}
Images generated by text-to-image (T2I) models often exhibit visual biases and stereotypes of concepts such as culture and profession. Existing quantitative measures of stereotypes are based on statistical parity that does not align with the sociological definition of stereotypes and, therefore, incorrectly categorizes biases as stereotypes. Instead of oversimplifying stereotypes as biases, we propose a quantitative measure of stereotypes that aligns with its sociological definition. We then propose OASIS to measure the stereotypes in a generated dataset and understand their origins within the T2I model. OASIS includes two scores to measure stereotypes from a generated image dataset: **(M1)** Stereotype Score to measure the distributional violation of stereotypical attributes, and **(M2)** WALS to measure spectral variance in the images along a stereotypical attribute. OASIS
also includes two methods to understand the origins of stereotypes in T2I models: **(U1)** StOP to discover attributes that the T2I model internally associates with a given concept, and **(U2)** SPI to quantify the emergence of stereotypical attributes in the latent space of the T2I model during image generation. Despite the considerable progress in image fidelity, using OASIS, we conclude that newer T2I models such as FLUX.1 and SDv3 contain strong stereotypical predispositions about concepts and still generate images with widespread stereotypical attributes. Additionally, the quantity of stereotypes worsens for nationalities with lower Internet footprints.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Stereotype Measurement Responsible AI Trustworthy AI Interpretability
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/8ab25601c2fc69ba8528b05b084d786efdeb9de7.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
OASIS Uncovers: High-Quality T2I Models, Same Old Stereotypes
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission