I'm scraping some website, encoding is unicode:

```
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

```

To do that I use urllib:

```
html = unicode(urllib.openurl(url).read(), 'utf-8')

```

Sometimes some of the unicode characters will be replaced with two 0xFFFD characters at random. I guess that a single character encoded with 4 bytes gets broken, and is replaced with two 0xFFFD for each of the two 2-byte halves. The probability of such replacement is pretty low - maybe one character in 500 000 is replaced, something about that. It seems like only non-ascii characters are being corrupted.

I don't observe any of these replacements when the same page is rendered in browser.

How do I identify which part introduces the problem? Is it the server that sends corrupted bytes or the client that doesn't read them properly?

@John Machin:

1. Python version is 2.7.1
2. It's observable in all browsers as well as in most of http client (libcurl, wget, python)
3. I actually was wrong here - sometimes these errors show up in browsers on the rendered page. The errors appear like two diamond characters with question marks.
4. The errors are random. Sometimes they show up, sometimes no. The larger the page is, the larger the chance is that the error will show up.

So I think that the error is introduced on the web server side. I will just try to ignore them.