title: Java RegEx matcher breaks characters outside the BMP
body: <p>I'm currently writing a util class to <em>sanitize</em> input, that is saved to an xml document. Sanitizing for us means, that all illegal characters (<a href="https://en.wikipedia.org/wiki/Valid_characters_in_XML#XML_1.0" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Valid_characters_in_XML#XML_1.0</a>) are just removed from the string.</p>
  
  <p>I tried to do this by just using some regex, that replaces all invalid characters with an empty string, but for unicode characters outside the BMP, this seems to break the encoding somehow, leaving me with those <code>?</code> characters. It also does not seem to matter which way of replacing by regexp I use (<code>String#replaceAll(String, String)</code>, <code>Pattern#compile(String)</code>, <code>org.apache.commons.lang3.RegExUtil#removeAll(String, String)</code>)</p>
  
  <p>Here's an example implementation with a test (in Spock), that shows the problem:
  XmlStringUtil.java</p>
  
  <pre><code>package com.example.util;
  
  import lombok.NonNull;
  
  import java.util.regex.Pattern;
  
  public class XmlStringUtil {
  
      private static final Pattern XML_10_PATTERN = Pattern.compile(
          "[^\\u0009\\u000A\\u000D\\u0020-\\uD7FF\\uE000-\\uFFFD\\x{10000}-\\x{10FFFF}]"
      );
  
      public static String sanitizeXml10(@NonNull String text) {
          return XML_10_PATTERN.matcher(text).replaceAll("");
      }
  
  }
  </code></pre>
  
  <p>XmlStringUtilSpec.groovy</p>
  
  <pre><code>package com.example.util
  
  import spock.lang.Specification
  
  class XmlStringUtilSpec extends Specification {
  
      def 'sanitize string values for xml version 1.0'() {
          when: 'a string is sanitized'
              def sanitizedString = XmlStringUtil.sanitizeXml10 inputString
  
          then: 'the returned sanitized string matches the expected one'
              sanitizedString == expectedSanitizedString
  
          where:
              inputString                                | expectedSanitizedString
              ''                                         | ''
              '\b'                                       | ''
              '\u0001'                                   | ''
              'Hello World!\0'                           | 'Hello World!'
              'text with emoji \uD83E\uDDD1\uD83C\uDFFB' | 'text with emoji \uD83E\uDDD1\uD83C\uDFFB'
      }
  
  }
  </code></pre>
  
  <p>I have now a solution, where I rebuild the whole string from its single code points, but that does not seem to be the <em>correct</em> solution.</p>
  
  <p>Thanks in advance!</p>