A good way to render objects on a photo background that works also with focus-blurred margins I'm trying to render with Cycles on a photo background using Blender Guru's compositor nodes:



[ Here, the foreground objects are obId=1 and are on layer 1 (rendered in RenderLayer); reconstructed background objects are obId=2 and are on both layer 1 and 2. Layer 2 is rendered as "ground". ]

However, in the current frame the main subject is supposed to be out of focus. As a result, both the ObIndex=1 pass (foreground) and the ObIndex=2 pass (reconstructed background objects, i.e. shadow catchers) are blurry. See the first two panels below:



As you can see, the blurriness doesn't consist in a smooth ramp of fractional intensities; it is instead a random cloud of blacks and whites, which will give an unusable result in terms of foreground/background separation (see the third panel). 

I guess this is because the ObIndex pass only contains one integer per pixel, that can be either one of the "Pass Index" values assigned to the objects in the scene, but not anything in between.

So, my question: is there a better way to blend a rendered foreground onto a photo background that works well with blurred objects?



Here's a blend file with the scene described above.
As 3dpointedit pointed out, there is a limitation of the current ID mask system:

Is Z-Render Pass accuracy dependent on sample count? Are others like object id? (Cycles)

As such, these layers are unusable for any post-processing workflow. You need to obtain these layers differently, or build the pipeline in another way.

The correct way of compositing onto a background is detailed here:

Cycles: Shadeless &quot;Shadow-catcher&quot; Material for Rendering onto Footage

In short, you need to put your objects in layer 1, your shadow catchers in layer 2, and the lights in both layers. The render layers will be "object only" (include: 1), "shadows" (include: 2), and "clean background" (include: 2, exclude: 1). Render with transparent background.

This will give you these layers:


"shadows"

 
"clean background"




Dividing the former with the latter you get the shadow:



You multiply this shadow with your backdrop, and you alpha-over the object into your backdrop. You have to render the scene 3 times though.

You can also use the new Shadow catcher feature of Blender 2.79, but it will only produce black shadows.