text: most similar to the ones used in GPT-2’s training data, i.e. docu- 
  ments linked to from Reddit [25], plus Wikipedia and a collection 
  of books. While this was reportedly effective at filtering out docu- 
  ments that previous work characterized as “unintelligible” [134], 
  what is unmeasured (and thus unknown) is what else it filtered out. 
  The Colossal Clean Crawled Corpus [107], used to train a trillion 
  parameter LM in [43], is cleaned, inter alia, by discarding any page 
  containing one of a list of about 400 “Dirty, Naughty, Obscene or 
  Otherwise Bad Words” [p.6].14 This list is overwhelmingly words 
  related to sex, with a handful of racial slurs and words related to 
  white supremacy (e.g. swastika, white power) included. While possi- 
  bly effective at removing documents containing pornography (and 
  the associated problematic stereotypes encoded in the language of 
  such sites [125]) and certain kinds of hate speech, this approach will 
  also undoubtedly attenuate, by suppressing such words as twink, 
  the influence of online spaces built by and for LGBTQ people.15 If 
  we filter out the discourse of marginalized populations, we fail to 
  provide training data that reclaims slurs and otherwise describes 
  marginalized identities in a positive light. 