Python regex £ sign
Question: I'm reading from a text file which contains pound signs (£):

    
    
    f = open(file, 'r')
    string = f.read()
    f.close()
    

Along with some other regex operations, I want to remove these pound signs,
and write the string to a new file. The closest I've got to making this work
is the following code:

    
    
    n = re.compile(unichr(163))
    string = n.sub('', string)
    

This seems to find the pound signs correctly, but instead of replacing them
with nothing, the `£` are converted this symbol: `Â`

Anyone have any idea what's going on?


Answer: # Summary:

> In utf8, `£` maps to raw bytes `\xc2\xa3`. The `re` module allows for string
> substitution to occur between unicode and byte encoded strings, which is an
> error.

It's my opinion that J.F. Sebastian's
[answer](http://stackoverflow.com/a/29317102/1213041) is more succint, but
here is a walkthrough.

# Details:

Calls to `read()` return a bytestring.

To illustrate, lets create the following file `durp`:

    
    
    echo -n "£" > durp
    

The next command gets the contents of the file in hex:

    
    
    $ cat durp | xxd  | cut -d " " -f 2
    c2a3
    

**Note:** Visiting this [url](http://cdosborn.github.io/unicode?cp=163/) will
display `£` in multiple encodings.

These are the raw bytes which constitute `£`. What does python do with the
file when its read?

    
    
    $ python
    > f = open("durp")
    > f.read()
    '\xc2\xa3'
    

It doesn't know what the encoding is so it represents the bytes in their
escaped hex form.

Let's import your code:

    
    
    > import re
    > r = re.compile(u'£')
    > u'£'
    u'\xa3'
    

That last line is just to see what we're making a pattern on. This is the
source of the error.

Now we perform the substitution on the contents of the file:

    
    
    > r.sub('', '\xc2\xa3')
    '\xc2'
    

Which is conceivable but wrong. We substituted `'\xa3'` for `''` in
`'\xc2\xa3'` and got `'\xc2'`. This is an error in `re`, because unicode
strings are being mixed with bytestrings. It doesn't not make sense to perform
substitions of characters that have different encodings. This is essentially
substituting bytes rather than characters.

J.F. Sebastian's [answer](http://stackoverflow.com/a/29317102/1213041)
explains how your terminal would interpret `'\xc2'` as `Â`.

