Title: UnicodeDecodeError: Converting type string to unicode
Tags: python;python-2.7;unicode
Question: I am trying to replace text. Unfortunately, the main string is stored as type unicode, but the string which describes the text to be replaced is stored as type string. Below is a reproducible example:

```mystring = u'Bunch of text with non-standard character in the name Rubén'
old = 'Rubén'
new = u'newtext'
mystring.replace(old, new)
```

This throws an error: 

```UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 3: ordinal not in range(128)```

I get the same error when I try to convert ```old``` to unicode with ```unicode(old)```. Several answers solve the problem for specific characters, but I cannot find a generic solution.


Here is the accepted answer: You need to convert the ```old``` value to Unicode with an explicit codec. What that codec is depends entirely on how you sourced ```old```.

If it is a string literal in the source code, use the source code encoding. Python won't accept your source file unless you specified a valid codec at the top in a comment; see PEP 263

Pasting your ```old``` definition into a terminal will use your terminal codec (the terminal sends Python encoded bytes as you paste).

If the data is sourced from anywhere else, you'll need to determine the encoding from that source. For HTTP data, check the ```Content-Type``` header for a ```charset``` parameter, for example.

Then decode:

```old = old.decode(encoding)
```

When you use ```unicode(old)``` without an explicit codec, or try to use the bytestring in ```unicode.replace()```, Python uses the default codec, ASCII.

Demo in my terminal, configured to use UTF-8:

```&gt;&gt;&gt; import sys
&gt;&gt;&gt; sys.stdin.encoding  # reflects the detected terminal codec
'UTF-8'
&gt;&gt;&gt; old = 'Rubén'
&gt;&gt;&gt; old  # shows encoded data in python string literal form
'Rub\xc3\xa9n'
&gt;&gt;&gt; old.decode('utf8')  # unicode string literal form
u'Rub\xe9n'
&gt;&gt;&gt; print old.decode('utf8')  # string value written to the terminal
Rubén
&gt;&gt;&gt; mystring = u'Bunch of text with non-standard character in the name Rubén'
&gt;&gt;&gt; new = u'newtext'
&gt;&gt;&gt; mystring.replace(old, new)
Traceback (most recent call last):
  File "<stdin&gt;", line 1, in <module&gt;
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)
&gt;&gt;&gt; mystring.replace(old.decode('utf8'), new)
u'Bunch of text with non-standard character in the name newtext'
```

Generally speaking, you want to decode early, encode late; make your data flow a Unicode Sandwich. As soon as your receive text, decode it all to Unicode values, and don't encode again until the data is leaving your program.

Comment for this answer: @Michael: People are used to using ASCII only. It is all plain text, really.
Comment for this answer: Thanks! Key piece of info missing from answers to other questions is how to figure out what the codec was `sys.stdin.encoding`. Unfortunately, plain text is not plain text but (in my case) cp1252.


{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 1, 'CommentCount': 0, 'ContentLicense': 'CC BY-SA 3.0', 'CreationDate': '2014:00:06/24/14', 'FavoriteCount': 0.0, 'PostTypeId': 1, 'Tags': '<python><python-2.7><unicode>', 'Title': 'UnicodeDecodeError: Converting type string to unicode', 'ViewCount': 403, 'id': 24391780, 'user_AboutMe': '', 'user_AccountId': 2695709.0, 'user_DisplayName': 'Michael', 'user_Reputation': 12245.0, 'user_Views': 548.0, 'user_id': 2327821.0}}