Removing duplicate Unicode characters Motivation

I came across an interesting question on SO: determine-if-string-has-all-unique-characters and thought about providing an extension method to enable duplicate removal given some kind of normalization. 

Description


  The goals:
  
  
  Remove any duplicate characters (keep the first occurence) and return the updated string.
  The method should be able to handle diacritics and extended unicode characters. 
  The method should allow the consumer to control normalization to find duplicates (for instance, case-sensitivity).
  


Questions


Looking for general feedback on C# conventions
Performance feedback
Am I reinventing the wheel?


Code

public static class StringExtension
{
    public static string RemoveDuplicateChars(
        this string text, Func&lt;string, string&gt; normalizer = null)
    {
        var output = new StringBuilder();
        var entropy = new HashSet&lt;string&gt;();
        var iterator = StringInfo.GetTextElementEnumerator(text);

        if (normalizer == null)
        {
            normalizer = x =&gt; x.Normalize();
        }

        while (iterator.MoveNext())
        {
            var character = iterator.GetTextElement();
            if (entropy.Add(normalizer(character)))
            {
                output.Append(character);
            }
        }

        return output.ToString();
    }
}


Unit Tests

Let's test a string that contains variations on the letter A, including the Angstrom sign Å. The Angstrom sign has unicode codepoint u212B, but can also be constructed as the letter A with the diacritic u030A. Both represent the same character.

[TestClass]
public class Fixtures
{
    [TestMethod]
    public void Fixture()
    {
        // ÅÅAaA -&gt; ÅAa
        Assert.AreEqual("ÅAa", "\u212BA\u030AAaA"
            .RemoveDuplicateChars());

        // ÅÅAaA -&gt; ÅA
        // Note that the ToLowerInvariant is used to normalize characters
        // when searching for duplicates, it does not mean the output gets
        // transformed to lower case.
        Assert.AreEqual("ÅA", "\u212BA\u030AAaA"
            .RemoveDuplicateChars(x =&gt; x.Normalize().ToLowerInvariant()));
    }
}
There are not much to say other than the usual missing argument null check:

It is valid to write the following:

  string test = null;
  test.RemoveDuplicateChars();


and RemoveDuplicateChars will be called with a null for the this argument text. Therefore you'll have to test for null:

public static string RemoveDuplicateChars(
    this string text, Func&lt;string, string&gt; normalizer = null)
{
  if (text == null)
    return text;

  ...


or throw an exception...



The default initialization  of normalizer could be a little less verbose:

  normalizer = normalizer ?? ((x) =&gt; x.Normalize());




A minor detail: Angstrom and Å: is also represented by \u00C5, which your code interprets as equal to Angstrom, but MS Word interprets them as different when using its Find-function?