Delete output image file on error
With a lot of trying and testing I came up with the following solution, which is a bit of a hack but it works so far. 

I did not find any direct way to find out if there was an error in the rendered image. So I am resorting to analyzing the rendered image after it has been generated. 

def PostRender(scene):
    print("Post render: Frame {} of scene {}.".format(scene.frame_current, scene.name))
    if scene != bpy.context.scene:
        # this is not the main scene. skip
        return

    file_invalid = 1
    try:
        print("Retrieving last output file [{0}]".format(last_output_file))
        image_object = bpy.data.images.load(last_output_file)

        num_pixels = len(image_object.pixels)
        width = image_object.size[0]
        height = image_object.size[1]

        # in every 10x10 pixel rectangle, test one pixel. this makes total 1% of the pixels spread all over the image
        for x in range(0, width-1, 10):
            for y in range(0, height-1, 10):
                xoff = int(random.uniform(0, 9))
                yoff = int(random.uniform(0, 9))
                pix_at = 4*(x+xoff + (y+yoff)*width)
                for i2 in range(3):
                    if image_object.pixels[pix_at+i2] != 0.0:
                        file_invalid = 0
                        break                

    except Exception:
        # error accessing data. file is assumed to be invalid
        pass

    if file_invalid == 1:
        print("Image seems to be invalid.")
        # handle the error case here

bpy.app.handlers.render_post.append(PostRender)


By checking only 1% of the pixels, the loop is very fast even for huge images. If you want to be one hundred percent sure, you can add a second loop afterwards for scanning all the pixels if file_invalid is still 1. 

This approach has the obvious shortcoming that it will not work if you have black images in your animation.

Possible options what to do to handle the error:


Leave a log file entry
Optionally delete the file
If you suspect a CUDA error (as in my case) it might be a good idea to reboot the computer
Restart rendering after reboot


I found the last two points especially interesting and implementing that has increased the productivity drastically, since I don't have much downtime anymore. I will not post the implementation here, since it is out of the scope of the question.