Robustly dealing with malformed Unicode files
The answer to your question depends on the version of python being used, and exactly what behavior you want. In Python 3.1 and up, the surrogateescape error handler can adeptly handle corrupted files that are otherwise encoded with UTF-8, and even write them back out again. Just use codecs.open(FILE, MODE, "utf-8", "surrogateescape"). If you want to know when invalid bytes were encountered, you can either scan for the relevant surrogate characters (those in the range U+DC80..U+DCFF), or compare to values you get from the replace error handler.

Through Python 2.7, it appears that the surrogateescape error handler is unavailable. You could contrast the results between the ignore and replace error handlers, but there is no convenient way to process the file in Unicode and write it out byte for byte.

In either version, you can theoretically write your own error handler that performs similarly to surrogateescape, ignore or replace, and counts the number of problems it sees. I'm uncertain whether this count can be scoped to a single file, however, as it appears an error handler only receives knowledge of the substring in question when an error is encountered.



On to a code review. Your code feels almost clunky to me. There's a stray import codecs, a lot of repetition (I can't fathom why re-reads the line with an error) and it doesn't try to leverage iterators. Perhaps you're on a particularly old version of Python (no, you use as in the except line), or perhaps you just haven't worked with iterators much. Probably most of this was from trying to extract the core issue, but I can only review it as is. I apologize for any errors in my code that follows; I haven't tested it, but I am willing to update it.

I would be strongly tempted to replace most of your code with something like this, and that's before trying to leverage anything I talked about above:

unicodeerror = 0
for i, line in enumerate(open("test.xml", "r")):
    try:
        line.decode("utf-8")
    except UnicodeDecodeError:
        print "WARNING: Encoding error in line", i+1
        unicodeerror += 1

if unicodeerror &gt; 5:
    raise StandardError("ERROR: Too many Unicode errors")


If you need to do more, such as actually process the line, then it's reasonable to save the results of the decode, and to try again with an error handler in the except clause:

    try:
        text = line.decode("utf-8")
    except UnicodeDecodeError:
        # error tracking omitted
        text = line.decode("utf-8", "ignore")
    process(text)


If you wanted to integrate a comparison between two error handlers, it would replace the exception, so you could do something like this:

import codecs
unicodeerror = 0
for i, (ignored, replaced) in enumerate(zip(
    codecs.open("test.xml", "r", "utf-8", "ignore"),
    codecs.open("test.xml", "r", "utf-8", "replace"))):
    if ignored != replaced:
        print "WARNING: Encoding error in line", i+1
        unicodeerror += 1
if unicodeeror &gt; 5:
    raise StandardError("ERROR: Too many Unicode errors")


And I'll leave a custom error handler totally up to you. If it works, it would probably be a lot more efficient than this approach, but there's something to be said for the simplicity of the visible code.



As a side comment, while I found this question interesting enough to delve into, the more I did so, the more it felt like a question about the right way to do this than a request to review your code. Keep in mind that it's much more on topic here to review code and make suggestions on how to improve it. So this particular approach isn't necessarily a great fit for codereview.SE.