Identifying Fairness Issues in Automatically Generated Testing Content
Kevin Stowe Benny Longwill Alyssa Francis Tatsuya Aoyama Debanjan Ghosh Swapna Somasundaran

Virtual Takeovers in the Metaverse: Interrogating Power in Our Past and Future(s) with Multi-Layered Narratives
    Jessa Dickinson
    April 29, 2024
===============================================================================================================


Natural language generation tools are powerful and effective for generating content. However, language models are known to display bias and fairness issues, making them impractical to deploy for many use cases. We here focus on how fairness issues impact automatically generated test content, which can have stringent requirements to ensure the test measures only what it was intended to measure. Specifically, we identify test content that is focused on particular domains and experiences that only reflect a certain demographic or that are potentially emotionally upsetting; both of which could inadvertently impact a test-taker's score. This kind of content doesn't reflect typical biases out of context, making it challenging even for modern models that contain safeguards. We build a dataset of 621 generated texts annotated for fairness and explore a variety of methods for classification: fine-tuning, topic-based classification, and prompting, including few-shot and self-correcting prompts. We find that combining prompt self-correction and few-shot learning performs best, yielding an F1 score of .791 on our held-out test set, while much smaller BERT- and topic-based models have competitive performance on out-of-domain data.[Code and dataset will be released upon publication.]






§ INTRODUCTION


    
    

    
    Large language models (LLMs) have become ubiquitous in the space of natural language generation (NLG) due to recent advances in model capability <cit.>. However, these improvements come with the potential for various negative societal impacts. These negative impacts include the generation of misinformation/propaganda, allocation harms of systems providing benefits only to certain groups of people, and representational harms revolving around bias and stereotyping. Natural language processing (NLP) models including LLMs are known to reflect and repeat harmful biases and stereotypes <cit.>, and research into how the community addresses the societal harms engendered by NLP technology is critical <cit.>. 
    These issues are particularly dangerous when applied to language learning and assessment; tests with inherent biases have the potential to directly harm test takers by unfairly disadvantaging certain subgroups.
    
    Many of these types of bias in language generation are well studied. Biases based on gender <cit.>, race <cit.>, nationality <cit.>, and disability <cit.> have been identified in language models, and many modern LLMs incorporate deliberate safeguarding measures in an attempt to alleviate these issues <cit.>. 
    
    In the area of language assessment, there exists a tangential set of issues regarding fairness to test takers and score users <cit.>. Content that is irrelevant to the skills and abilities being tested should be avoided (Figure <ref>). This includes content that could disadvantage anyone based on their culture, location, or experiences (e.g., focusing on barbeques on the 4th of July could disadvantage test-takers who are unfamiliar with U.S. culture); their emotions (e.g., health hazards and diseases can evoke negative emotional responses among some people); their  worldviews (e.g., luxury cruises or designer clothing may make some people feel excluded); and other issues. We refer to these types of issues as fairness issues.

    Knowing how to better understand, detect, and mitigate bias related to fairness in NLG not only raises awareness of the issue but also enables researchers and developers to create more fair and inclusive NLP systems, evaluation metrics, and datasets in the language assessment space.  
    
    
    


    Our goal is to build a system for identifying fairness-violating content in automatically generated texts. It is of course still necessary to have human review of content before it is publicly used, but by adding a filtering process after generation and before manual review, we can significantly reduce the time taken for reviewing and the chance that fairness-related content is mistakenly allowed. To accomplish this goal, we explore four different approaches: fine-tuning, topic-based classification, few-shot prompting, and prompt-self correction.

    Our methods need to adapt to new contexts: our definition of fairness is operationally defined by the particular testing context, and may not apply to others, so the guidelines, prompts, and models may not apply generally to new contexts. For this reason, we assess our methods on two held-out test sets and analyze how our methods could be applied to new contexts. We release our resulting dataset, consisting of 620 samples of which 19.4% contain fairness issues[Each sample we used was rejected for deployment in actual tests. Using rejected samples for our experiments allows us to release the dataset: accepted stimuli cannot be made public.], to facilitate improvements in the fairness-detection community.

    Our contribution consists of the following:

    
    0em
        
  * We define a new fairness problem around issues faced in developing fair testing content.
        
  * We build and release a dataset of 620 generated samples for use in evaluating fairness detection methods.
        
  * We analyze the relative effectiveness of a variety of well-known classification techniques.
        
  * We provide a new mechanism for prompting self-correction which yields significant improvements over other prompting strategies.
    

    We start with data collection and analysis. We collect 620 samples over 7 different types of content generated using LLM prompting. We annotate each sample and identify whether it contains a fairness issue, and if so, whether that fairness issue pertains to knowledge, skill, or expertise or emotion (more on these categories and how they relate to fairness in Section <ref>). We then use this dataset to experiment with a series of classification models for identifying fairness issues.

    We show that fine-tuning and filtering by topic can be cheap and effective options, although overall prompting strategies with GPT4 tend to be more effective. Few-shot prompting along with self-correcting prompt strategies yield strong performance with relatively little data, and combining both yields the best results on our in-domain test set, with an F1 score of .773. Interestingly, using a shorter, more generic prompt combined with our self-correction method yields the best result on our out-of-domain test set, with an F1 score of .462.    


§ RELATED WORK

    Bias, fairness, and responsible AI has been at the forefront of education technology, with contemporary research focusing on automated scoring, writing assistance, and other nuances of applying NLP technology to this sensitive domain <cit.>.   
    baffour-2023 find that assisted writing tools may exhibit moderate bias depending on the task, while wambsganss-2023 found no significant gender bias difference in writing done with and without automated assistance. wambsganss-2022 explore bias in educational tools for German peer review, and kwako-2023,kwako-2022 propose novel methods for detecting bias in automated scoring algorithms.

    We are specifically interested in applications to language generation, and there is also substantial work in using LLMs and other NLP technology to generate content for educational assessments <cit.>. However, this work largely fails to address bias and fairness issues in content generation. Our work is specifically focused on fairness issues in automatically generated language testing content.

    In the context of language models, fairness and bias have emerged as critical concerns. Existing detection and mitigation tools generally diverge from our work: some are overly domain-specific like the focus on news articles in <cit.>, while others are focused on assessing issues within the language models and datasets <cit.>, rather than the outputs. Other works rely on retrospective metrics that assess a model's fairness through aggregated predictions and subgroup analysis, and/or focus on classification rather than generation problems <cit.>. Although these tools enhance transparency and accountability for evaluating language model issues, they fundamentally differ from our bias detection approach tailored for evaluating generated text in real-time for a production environment.





§ DATA

    
    In the language testing context, we face a unique set of fairness challenges in generating content. Specifically, fair testing requires content that does not contain irrelevant factors that negatively impact the assessment of a test taker. 

    A primary concern is to ensure that the test content measures only what it is intended to measure. For English-language proficiency tests, this means that the test must measure only the skills and abilities needed to communicate effectively in English, and not other constructs such as background knowledge of specific jobs, events, or cultures.  
    
    Consider the following question and an example of a response to that question: 
    
    

    
    -.225em 
    
  * Question: Are you planning to go to one of The Eras Tour shows? 
    
  * Response: Yes—I love Taylor Swift! 
    
    

    
    If the task were to identify whether the response is an appropriate response to the question, even some native English speakers would likely get it wrong. This is because, in addition to needing to know features of English proficiency (in this case, the ability to infer gist, purpose, and basic context based on information stated in short spoken texts), one would also need to know about Taylor Swift and her concert tour. Thus, those familiar with Taylor Swift would have an unfair advantage in identifying the correct answer, and those who are unfamiliar with Taylor Swift would be unfairly disadvantaged. 
    
    Eliminating the fairness issue for this type of question would result in the following revision: 

    

    
        -.225em  
        
  * Question: Are you planning to go to the music concert on Friday? 
        
  * Response: Yes—I’m looking forward to the performance! 
     
    

    
    
    In addition to avoiding testing outside knowledge, it is also important that language proficiency tests do not include content that is offensive or disturbing. For example, the following question and response refer to serious health issues, which have the potential to evoke deep negative emotions. 

     

    
        -.225em
        
  * Question: Did you hear that Luis has been hospitalized?  
        
  * Response: No, but I knew he had a bad case of Covid-19. 
    
      

    
    Content like this that could prompt strong feelings of anger, sadness, or anxiety should be avoided because it could derail a test taker’s concentration, resulting in lower performance on the test. How a test taker interacts with this test content may tell more about their ability to concentrate under emotional strain than about their ability to identify a response's linguistic appropriateness. Eliminating this construct-irrelevant content helps to ensure that the test measures only the skills and abilities it is intended to measure. 
    
    
    

    

 §.§ Dataset


     
     
 
    Our goal is to identify and mitigate these fairness issues in testing content. We build a dataset spanning seven different item or task types from standardized English language proficiency tests all generated using GPT4 <cit.>. Item and task types can contain up to four components: the stimulus (main text the question is based around), stem (question asked about the stimulus), key (correct answer to the stem), and distractors (a set of alternative answers that are not correct).
    
    Fairness issues are possible in all components, but we focus on only the stimuli: these are the longest, most feature-rich components of the test content, and thus are most likely to reflect fairness and bias issues. Issues in the stimuli can leak through to stems, keys, and distractors, making the stimulus the likely culprit for the majority of issues.

    

  
Annotation
    For each stimulus, we aim to identify whether or not the stimulus contains fairness/bias issues, and if so, what type of issue is present. We start with a dataset of automatically generated stimuli. These have been reviewed by experts, and each was rejected for use in live assessments. These rejected stimuli have the relevant language and structure: our goal is to identify which of those stimuli were rejected (at least in part) for fairness reasons. We employ content development experts to annotate these samples, yielding a binary classification between non-fairness and fairness-related rejections.

    However, there are many different ways for bias and fairness considerations to impact individual stimuli. To better understand and mitigate these issues, we split them into two main categories:
    
    
        -.15em
        
  * knowledge, skill, and ability (KSA): content that contains construct-irrelevant information that may be unavailable to test takers in different environments or with different experiences and abilities. These include content with reference to specific skills, regionalisms, or unfamiliar contexts.
        
  * emotion: content in which language, scenarios, or images are likely to cause strong emotions that may interfere with the ability of some groups of test takers to respond. These include offensive, controversial, upsetting, or overly negative content.
    

    We annotate each sample flagged with a fairness issue with one or both of these categories. This allows further analysis to address these specific fairness categories and to better understand the impact of specific fairness issues.

    Our dataset is comprised of stimuli from seven different item and task types: a summary of the collected data is shown in Table <ref>, with examples for each type in Appendix <ref>. These stimuli represent various structures, depending on the item/task type: Read a Text Aloud, Talks, Announcements, and Text Completion stimuli are short text paragraphs, while Conversation stimuli involve turns between two or more speakers. Respond to Questions Using Information Provided and Respond to a Written Request task stimuli are structured content: the generation process creates text that is filled into a structured template; we use only the raw text.
    
    Overall we collect 620 samples, of which 19.4% exhibit evidence of fairness issues, with 9.3% reflecting KSA issues and 12.2% Emotion issues.

    We build a validation set of 48 samples reflecting a balance of the item and task types from the training types (Read a Text Aloud, Talks, Text Completion, and Respond to Questions Using Information Provided), and an equal-sized "in-domain" dataset from these stimuli is held separately for testing. These datasets contain an even number of positive and negative classes for fairness evaluations: as our goal is to able to identify positive cases where fairness issues exist, we intend for our validation and test sets to have a substantial number of this class. We use the three remaining types (Conversations, Respond to a Written Request, and Announcements) as a separate "out-of-domain" test set to evaluate performance on unseen content.



§ METHODS

    

    Our goal is to detect whether a generated stimulus contains an issue as a binary classification task. We experiment with standard transformer-based classification baselines, topic detection, and a variety of GPT4-based prompting, including methods for automatic prompt-self correction. We describe each method below: each is tuned on the validation set, and we report the best model performance on that set. We then evaluate model performance on two separate test sets in Section <ref>.

    

  
Classification with Fine-Tuning
    As a baseline, we fine-tune standard pre-trained transformer models for sequence classification. We experiment with , , and . We perform a hyperparameter search and report results using the model with the best-performing results.

    

  
Topic-based Filtering
    We observe that many samples are flagged for fairness due to the topic of the material: many topics contain content that violates our fairness guidelines directly, while others are simply more likely to include unacceptable content. Motivated by this, we explore topic detection as a method for identifying fairness issues.

    We first identify topics found within the data. We use BERTopic <cit.>, a state-of-the-art neural topic modeling framework to extract topic representations from two sources of training data: (1) all samples from the training partition of our dataset and (2) our fairness guidelines. In this method, SentBERT <cit.> converts each training document into a dense vector representation which are grouped by semantic similarity, creating clusters that represent different topics. For each of the two training sets, topic descriptions made up of the most important words in a cluster are generated for the clusters containing at least five supporting documents. We manually assess each topic description for themes that should be avoided based on their relation to known fairness issues and which topics are acceptable. Finally, for each unseen sample in test and validation datasets, we make predictions based on the single nearest topic cluster. If a sample falls within the boundaries of restricted topics, it is classified as a violation.

    

    Results for these methods are shown in Table <ref>. The fine-tuned bert-based models perform fairly well, with F1 scores for  and  both nearing .65. The Topic-based Filtering models are worse, with the data-based system yielding an F1 score of .579. In all cases precision is much higher than recall: these models are conservative with predictions.

    
    

 §.§ Prompting

    
    We initially experiment with five different “base” prompts. We pair these with stimuli and use GPT4 to return “True” if the stimulus contains a fairness issue and “False” otherwise. These prompts represent different strategies[Prompts in Appendix <ref>]:
    
    
    -.15em
        
  *   53 tokens: Drawing from general knowledge of fairness and bias in LLMs, we write a generic prompt designed to combat attested LLM biases. This prompt is designed as weak baseline. Our goal is to determine if short, simple prompt can capture relevant issues, and whether or not it can be easily improved via self-correction or few-shot learning (Sections <ref> and <ref>)
        
  *   191 tokens: This is a longer, more detailed version of the above, containing nearly 200 tokens.
        
  *   197 tokens: We craft a prompt based on guidelines for writing fair assessments. Using documentation that defines what constitutes fair assessment items and how to write them, we build a prompt capturing the important components of a fair question based on guidelines. The goal of this prompt is to determine if human-written guidelines based on theoretical issues will accurately capture these issues in real data.
        
  *   1081 tokens: We construct a “long” version of the previous guidelines by summarizing the entire fairness guidelines with the help of GPT4, asking for concise versions of relevant sections and combining them into a document that fully captures all the relevant aspects of the guidelines. This prompt is our longest, but still fully based on documentation. The goal of this prompt is to determine the efficacy of a longer, more comprehensive prompt.
        
  *   142 tokens: We craft a prompt based on annotations in our data. We identify what topics and language cause fairness issues in the training data and build the prompt to reflect both these issues specifically, and how they might generalize to unseen item/task types and topics. This method is hypothesized to be the most effective, as it will address known issues in the data, but may not extend to unseen data, as it is built specifically around the given training samples.
    

    These prompts are run through GPT4 via the Azure interface <cit.>. Each prompt was updated manually to correct obvious potential issues. Our goal here is not to overoptimize prompt writing, which could lead to overfitting the validation set, but rather to develop a generic prompt likely to be effective for both known fairness issues and novel issues possible in generated content.

    Initial experiments on the validation set showed two things: the  prompt performs similarly to the  in all cases, and the  prompt failed to identify any fairness issues. We therefore focus our efforts on the three other prompts:     and .

    A primary issue in identifying the fairness issues we are concerned with is that they are only applicable in the context of certain testing environments. The guidelines and data we use are specific to a certain test: for other tests, other environments, and other contexts, different definitions of fairness and different alignments will be applicable. For this reason, the ability to adapt to new environments is critical in applying fairness constraints. We aim to build a system that can learn to adapt to new guidelines with minimal information.

    We explore several methods to improve the robustness of prompting: few-shot prompting and self-correcting prompting techniques. 

    

  
Few-Shot prompting
    
    We follow few-shot prompting strategies that have been successful for LLM classification tasks <cit.>. For each of our base prompts, we provide a set number n of additional samples for both the positive and negative classes. Each of the samples contains the text of the stimulus along with a brief description of why it does or does not violate our fairness guidelines.

    
    

  
Prompt Self-correction
    
    
    We leverage our annotated data using a prompt self-correction strategy. We start with a base prompt and iterate through a batch of samples taken from our training data. For each sample, we insert text into the prompt, perform a pass through GPT4, and evaluate the resulting output. If it's correct, we continue. If not, we instruct the LLM to make adjustments to the prompt, either to disallow the sample (when the error is a false negative) or to allow it (when the error is a false positive). This process is run up to e epochs, or stopped early if accuracy reaches 1 or the predictions don't change. We run over b batches of n samples randomly drawn from the training data. We use the best-scoring prompt from the final batch for evaluation.[For an example of the process, see Appendix <ref>]


    

  
Combining Few-Shot and Self-Correction
    Few-shot and self-correction are inherently complimentary, as the self-correction method returns an optimized prompt and few-shot learning reinforces it by providing in-domain examples. We combine them by concatenating additional few-shot learning samples to the self-correcting prompts.

    For each of these improvements to prompting, we perform a hyperparameter search over the number of total training/few-shot samples and batch size. We experiment with the     and  prompts.[Experiments with the longer guideline-based prompt were unsuccessful: the LLM invariably returns either a commentary on a single testing procedure or rewrites the prompt entirely to handle a single sample.] We hypothesize the  and  prompts should be able to benefit quickly from adaptive methods, while the  prompt should be nearly optimized, as it is already based on observations from the data.


    We run each of our experiments on the validation set to tune the prompts and parameters to optimize F1 score. Figure <ref> shows the best results on the validation set. We explore each model's effectiveness on unseen data in Section <ref>.
    
       
    
    The base generic prompt fails, as the traditional bias and stereotyping issues are less likely to occur in our generated content, and the fairness issues we are concerned with are unlikely to be deemed as problematic out of context. Using a simplified version of our guidelines yields .364 F1 score for identifying fairness issues.  The  based on observations in the training data yields much better results (.7 F1). However, this may not extend well to novel cases, as the prompt is driven purely by our validation data.
    
    Few-shot learning displays some interesting properties: we see significant improvements across all three prompts, using 3 samples (this yielded the best results across all validation runs). Even the minimal  prompt rises to over .6 F1 with minimal few-shot prompting. 
        
    We see small improvements over the baseline using prompt self-correction for all three prompts. For the  prompt, results using self-correction equal those using few-shot learning. This aligns with previous work showing that language models themselves tend to write better prompts <cit.>: after only a few iterations of self-correction, the  prompt surpasses the performance of a human-written prompt, even in cases where the human describes the dataset explicitly.

    Combining self-correction and few-shot learning yields improvements over base prompts and few-shot prompting alone. This approach yields the best results for all three prompts, with the best-performing model being the  prompt with self-correction and few-shot learning. This may be due to overfitting, however: the prompt is written to reflect the data. To explore the efficacy of these methods on unseen data, we evaluate them on our two held-out test sets.


§ TEST RESULTS


    
    
    

The previous experiments describe our attempts to identify the best-performing model for fairness classification on our validation set. Our goal is to develop a system that generalizes: for this, we evaluate the best performing of the above model types on two held-out test sets:


-.15em
    
  * In-domain: The 48 held-out samples drawn from the item/task types used for training.
    
  * Out-of-domain: All samples (85) from the three held-out types: Conversations, Respond to a Written Request, and Announcements.


    Figure <ref> shows the results on the test set. We evaluate the best-performing models of each type: fine-tuned transformer models, topic-based classification, base prompts, few-shot learning, self-correction, and combining few-shot and self-correction. We here note some key facts about model performance on our test set.
    
 

  
Best Performance
    Combining the  prompt with self-correction and few-shot learning performs the best on the in-domain test. This shows this is the best approach if there is available data and expertise to support hand-crafting a  prompt and running self-correction. On the out-of-domain data, the smaller initial prompts,  and   both outperform the  prompt, perhaps due to their more generic nature: the  prompt is too specific to this dataset, and understandably doesn't generalize well. The self-correct+few-shot methodology performs the best in both cases: few-shot learning alone is better than self-correction alone, but the combination is typically the best.

    

  
Strong Results from Small Models
    Traditional transformer-based classification performs remarkably well, especially in generalizing to the out-of-domain data. On the in-domain data, the best performing model  performs on par with the best base prompting model (.571 compared to .6 F1 score). On the out-of-domain data,  performs nearly as well as the best-performing overall model, just .04 behind the  prompt with self-correction and few-shot learning. If the goal is to quickly and cheaply build a system that is applicable to a wide variety of domains, there appears to be significant value in relying on these relatively small transformer-based classification models. The Topic (data) approach is also competitive on out-of-domain data, and doesn't even require model training: it lags only slightly behind the  model.

    

  
Self-Correction
    We found significant success in our proposed self-correction mechanism. While it typically doesn't outperform few-shot learning in isolation, the methods are naturally complementary, and the combination often yields the best-performing model.

    In our experiments, we noted some issues. First, when run using too many samples or batches, the prompts tend to degrade: once the LLM makes an error and returns a prompt that doesn't match the specifications, the run needs to be aborted. Even when the LLM sticks to the instructions, after many iterations the prompts become unwieldy and self-contradictory, and performance rapidly declines. We suggest using somewhere between 6 and 20 total samples for prompt self-correction: it's best to not continue to make corrections indefinitely.

    
    

  
Use-cases and metrics
    We here report F1 score as a balance between precision and recall (for full scores, see Appendix <ref>). Depending on the end use-case, other metrics may be more appropriate. In our case we advocate always including humans in the evaluation process to ensure that only fair content is accepted: we then value both precision (as we don't want to unnecessarily flag excessive content as having an issue, which could reduce diversity) and recall (as we don't want to let fairness issues through). Optimizing for recall seems reasonable, as it is likely more important to prevent fairness issues from being released, but it is critical to note that no system is perfect: even optimizing for recall, these fairness issues are likely to persist, and the models should not be used as failproof safeguards.
    
      

  
KSA and Emotion

           
     
        
    We evaluate performance on the test set for the two subcategories annotated: Knowledge, Skill, and Ability (KSA) and Emotion (Table <ref>). Models built on data (the  prompts (.47) and Topics from Data (.571)) perform best on the Emotion, likely due to the inclusion of negative emotional issues in the text. The data-based Topic classification model performs best in this category, highlighting the benefits of quick, untrained models. The  prompt combined with few-shot and self-correction performs best in KSA identification (.445), highlighting the difficulty of the task.
    


§ CONCLUSIONS

    

    This work delivers four key contributions: an exploration of a novel fairness detection task, a dataset of 620 samples annotated for fairness issues, evaluation of a variety of classification models for this task, including fine-tuning, topic-based approaches, and prompting, and a novel prompting strategy, which, combined with few-shot learning, achieves state-of-the-art performance on the task.

    This work is aimed to explore the space of fairness and bias issues in generated content, especially in the education context. We aim to highlight the difficulties of accounting for fairness particularly in specific contexts unlikely to be accounted for by traditional model guardrails. As language model usage becomes more prevalent, the need for proper bias and fairness strategies from those training, deploying, and using these models is paramount.


§ LIMITATIONS

Our work is limited largely by the type of content evaluation and the models used. We focus specifically on a small number of item and task types that fall under very specific fairness constraints: the evaluation of the methods used specifically applies to these items under these constraints. This is apparent in the evaluation on the "unseen" item types in Section <ref>. Applying these methods to new item and task types, even those annotated under the same fairness guidelines, yields significantly reduced results: this is evidence that the methods and models designed work only for the specific contexts in which they are trained and developed.

Similarly, we explore a relatively small space of models and approaches. We use relatively basic prompt strategies: there exist many other approaches and improvements that are likely to be valuable that we do not evaluate. The same is true of fine-tuned models, and topic classification: we present relatively basic, well-known strategies to better understand the difficulty of our data, with the understanding that there are substantial improvements that could be applied.


§ ETHICS

Content generation comes with inherent ethical concerns relating to fairness, bias, factuality, and sensitivity. Our work aims to mitigate some of these issues with regard to fairness, but it is important to consider potential issues that might arise from using large language models and other NLP technology in generating assessment content. Models may introduce subtle biases against disadvantaged groups, or produce content that appears to be factual, but is not: these are critical failures that need to be accounted for.

In practice, the generation of assessment content requires human intervention: large language model generations are not at the point where they are immune to these negative impacts, and thus for any content that goes into production, a human with relevant expertise needs to evaluate it. The methods we propose support this human intervention: these methods can remove obviously offensive content before the human even reviews, or assist in human reviews by flagging potentially harmful content.

While our dataset is unlikely to contain any content that is triggering (our framework of fairness is focused on more nuanced contexts), it must be noted that there is potential for it to be used maliciously, for example, by someone designing a system to adapt to and deceive a fairness detection systems. In releasing this data we hope to bring awareness to this issue, and better understand the potential negative impacts. Primarily, we stress that any fairness detection system should not be used in isolation or without supervision as a catchall for potential issues.
    



acl_natbib





§ ITEM TYPES


    Table <ref> gives examples for each item type.
    
  



§ PROMPTS



Below are the five base prompts used. Each also contains a slot to insert a stimulus and a suffix instruction: "Consider the instructions. If the text violates the instructions, return only the word True. If not, return only the word False."



 §.§  

Ensure it is fair to for a test taker. It should not contain biased, toxic, or unethical content. It should not reflect any special skills, abilities, or culture, and it should not contain any negative emotion.



 §.§  

Develop a set of criteria to filter out negative test items that may violate fairness, bias, or sensitivity requirements. Consider the following aspects:

Fairness:
Are the test items equally applicable and appropriate for all individuals regardless of their background, demographics, or personal characteristics?
Do the test items avoid favoring or disadvantaging any specific group based on race, ethnicity, gender, religion, sexual orientation, socioeconomic status, disability, or any other protected characteristic?
            
Bias:
Do the test items avoid stereotypes or assumptions that may lead to unfair advantages or disadvantages for certain groups?
Are the language, examples, scenarios, and references used in the test items neutral and inclusive?
Have the test items been reviewed to ensure they do not unintentionally favor or discriminate against any particular group?

Sensitivity:
Are the test items respectful and considerate of individuals' diverse experiences, perspectives, and identities?
Do the test items avoid potentially triggering or offensive content?



 §.§ 

Consider the following text. The text should avoid any content or information that would unfairly disadvantage someone tested on it. It should not do any of the following: 
            
1. It should not contain content specific to a region, including brand names, celebrities, sports, entertainment, culture, customs, geography, government, history, holidays, or laws. Any of these topics that are specific to a certain region should be excluded.

2. It should not include topics that require specialized knowledge that would be easier for one group than another. 

3. It should not include unnecessarily difficult language including words, phrases, and concepts more likely to be known by one group than another. 

4. It should avoid contexts that not all people may have experienced. The contexts should not require direct, personal experience to understand. 

5. It should not mention religion. 

6. It should not include contemptuous, derogatory, or exclusionary language. It should not induce any negative emotions. 

7. It should not advocate for particular causes or ideologies, or include anything divisive. 

8. It should avoid sensitive and controversial topics, including political issues, natural disasters, accidents, or other negative topics.



 §.§ 

Below are a set of guidelines. These guidelines aim to enhance the fairness and validity of tests, communications, and other materials. These guidelines assist users in understanding fairness in assessment, including the right content, eliminating unfair content, promoting diversity and inclusivity, addressing accessibility and inclusion issues and reducing subjective fairness decisions. The guidelines cover the fairness of various subjects including the National Assessment of Educational Progress (NAEP), K–12 tests, artificial intelligence (AI) algorithms and includes information to help use plain language and a quick reference guideline list.

  Understanding fairness in testing is crucial for proper application of guidelines, though its definition varies. One common definition sees fairness as absence of any inequity, affecting individuals and groups alike, such as unfair test questions or biased content affecting diverse groups. Another definition argues that tests seeming harder for certain groups aren't necessarily unfair, as differences in results may reflect real differences in knowledge or ability, not test bias. Group score differences don't prove bias, but should be explored to rule out bias. Furthermore, fairness definitions based on outcomes are contested and of limited use during test design. Fairness is also defined based on test validity. The test validity indicates quality, and represents the accuracy of inferences and actions based on scores, which must be equally valid for all test-takers for a test to be fair. Therefore, an effective definition of fairness in assessment is rooted in validity, creating an interconnected relationship between the two. Lastly, fairness in testing relates to the effectiveness of related educational products and services in fulfilling their intended purposes.
  
  These guidelines should ideally cater to everyone, particularly focusing on groups discriminated against due to factors such as age, appearance, citizenship, disability, ethnicity, gender, national origin, native language, race, religion, sexual orientation, and socioeconomic status. It's crucial to also account for intersectionality, a framework recognizing how overlapping identities like race and gender can impact the experiences of individuals with multiple marginalized identities. For instance, Black women may perceive test material differently than Black men or White women.
  
  Principles and Guidelines for Fairness 
  
  Fairness in assessment requires adherence to key principles: Tests should focus on essential aspects of the intended construct and avoid construct-irrelevant hurdles. They must offer design, content, and conditions facilitating valid inferences about diverse test takers' knowledge and abilities. Also, they should provide scores that allow valid group-wise inferences. The subsequent sections offer specific guidelines related to these principles. In case of interpretational conflicts, choose the one that upholds fairness principles.
  
  Construct-Irrelevant KSA Barriers to Success 
  
  Construct-irrelevant Knowledge, Skills, and Abilities (KSA) barriers to test success can arise when unrelated KSAs are required to answer a question correctly. For example, a math item asking for the conversion of kilometers to meters is construct-irrelevant to multiplication skills. If a specific group lacks this irrelevant knowledge, the test's validity and fairness are diminished. 
  Construct-irrelevant sources of KSA often include unfamiliar contexts, disabilities, difficult language, regionalisms, religion, specialized knowledge, translation issues, unfamiliar item types, and topics specific to the U.S. 
  
  The content and context of test stimuli should be familiar and accessible to all test takers. 
  Tests shouldn't require personal experiences that may not be available to test takers with disabilities.
  
  Language should be simple and clear, and shouldn't require knowledge of jargon or specialized vocabulary unless relevant to the test.
  
  Regionalisms, words or phrases specific to a certain region, should not be required unless relevant to the test construct.
  
  Tests shouldn't require unrelated knowledge about religion.
  
  Construct-irrelevant specialized knowledge should be avoided unless the test is designed to assess that specific knowledge.
  
  Tests need to be culturally adapted along with translations to ensure fairness.
  
  Test takers should be familiar with the technology used in assessments.
  
  
  Tests taken by an international audience shouldn't require specific knowledge of U.S. dominant cultures or conventions unless meant to measure such knowledge.
  
  Construct-Irrelevant Emotional Barriers to Success
  
  Construct-irrelevant emotional barriers to success occur during testing when certain language, scenarios or images elicit strong emotions that disrupt a test taker's ability to answer a question. This can happen due to offensive content, controversial material, or content that challenges a test taker's personal beliefs. The stress and pressure of testing can heighten these reactions. It's important to avoid potentially offensive material, especially content that may trigger negative reactions in diverse groups of test takers.
  
  Test content about groups that have been discriminated against should be carefully reviewed for any offensive or emotionally triggering material. Test developers should strive for diversity in their team and aim to use content written by diverse authors. However, offensive content should be avoided even in multiple choice items where the wrong answer may potentially be seen as the viewpoint of the test creators or institution.
  
  A list of topics likely to trigger negative reactions is provided, including topics like abduction, abortion, and drug use among others, and should be avoided in test materials unless they are crucial for test validity. On the other hand, while some topics may not trigger negative reactions, they need careful handling to ensure balance and objectivity. This includes topics like advocacy, biographical material, conflicts and others.
  
  The document concludes with a detailed discussion on specific topics that should either be avoided or handled with care in tests, including religion, personal questions, role playing, sexual behavior, stereotypes, and violence among others. It emphasizes the importance of fair, balanced and objective representation in testing material, and the avoidance of content that may trigger strong negative emotions or construct-irrelevant barriers to test performance.
  
  Plain Language
  
  Tests should contain plain language. Plain language benefits all test-takers, minimizing score differences unrelated to test construct. It is not designed to override client-specific guidelines or simplify complex language inherent to the construct being tested. Plain language applies to all irrelevant elements of tests and associated materials, and examples where it isn't suitable include reading comprehension tests, subject-matter tests, historical documents, and language proficiency assessments.
  
  Plain language includes writing short, clear paragraphs with one main idea and avoiding complex sentences and vocabulary unless necessary. Relative clauses should be used sparingly, nouns should be clear, transition words used for clarity, and personal pronouns used when aiding communication. Testing material layout and formatting should be clear, and verb forms simple and in active voice. Test-specific guidelines also apply, with clear and precise stem presentation. Avoiding negative stems or emphasizing them if necessary is advised.



 §.§ 

No mention of anything difficult or challenging. 
        
Nothing about luxury or elitist activities such as cruises or golf, or expensive vacations. 
        
Nothing upsetting or dangerous such as car accidents, health hazards, disease, or disasters that might make the test taker nervous. 
        
No topics that might be anxiety-inducing, like romance, testing, or relationships. 
        
No health-related topics like health codes, health hazards, or anything negative related to health. General well-being is okay. 
       
No fitness discussion that is focused on weight loss or negative health impacts. 
        
No negativity involving overcrowding, housing shortages, or general bad living conditions or housing concerns. 
        
No content and vocabulary specific to the US or other regions such as Fahrenheit units, local holidays, or local events. 
        
No controversial and political topics such as fake news. 
        
No mentions of any alcohol, drugs, or other controlled substances. 
        
No specific skills or abilities that wouldn't be generally known, such as law or medicine. No specialized vocabulary, acronyms, or jargon. 
        
No hazardous or potentially dangerous technology like viruses or artificial intelligence. 
    
No trademarked or well-known products, places, or people.




§ SELF-CORRECTION PROMPTS


Below is an example of prompt self-correction:


  
Initial Prompt Consider the following text. Ensure it is fair for a test taker. It should not contain biased, toxic, or unethical content. It should not reflect any special skills, abilities, or culture, and it should not contain any negative emotion.



  
After one batch of corrections Ensure the test content is unbiased, non-toxic, and ethical, without favoring any specific skills, hobbies, or activities not related to the test's subject. It should not evoke negative emotions or assume uniform experiences or opportunities among test takers. The content should avoid asking about personal financial decisions or preferences. Questions that ask for personal opinions or experiences are acceptable as long as they do not favor a specific group or assume uniform experiences.



  
After final batch of corrections Ensure the test content is impartial and ethical, not favoring unrelated skills or activities. It should not assume similar experiences among test takers or provoke negative emotions. Avoid questions about personal finances, specific hobbies, or activities that may not be universally accessible or common. Personal opinion or experience questions are acceptable if they don't favor a certain group and are not related to sensitive personal information. Also, avoid questions that assume a certain life stage or financial status, such as retirement planning, as not all test takers may have the same experiences or opportunities. Return 'True' if these principles are breached, 'False' otherwise.




§ VALIDATION RESULTS


    Precision, recall, and F1 scores for each model on the validation set can be found in Table <ref>.
    
    

    


§ TEST RESULTS


Precision, recall, and F1 scores for all models on both test sets can be found in Table <ref>.



