Render Layers causing pixelation
The index passes in Cycles are only rendered with only 1 sample, unfortunately, so they are not antialiased.
Your setup looks to me as if it should work with a simple Alpha Over too. Replace your Mix node with AlphaOver and just don't use the index pass.
However, for that to really work correctly in every case you have to make sure that the object you want to separate is masked by the occluding geometry from the rest of your scene. 

So put the object that you want to separate on a layer with only that object on it. Then in Renderlayers exclude the layer with the element that should not appear on that layer by using the Exclude Layers, like you already did. Finally set the Mask Layers to all layers with objects on it, that would occlude the separated object. In your case that would be the table, in case the makeup thing would penetrate the table geometry a bit. Though in your scene you might even be lucky enough that it might work without even bothering with Mask Layers.

If you really have to use an index pass you can try to smooth it with the Dilate/Erode node, with mode set to "Feather".