Why does the difference modifier not work on objects with many vertices, but will on a subset of those vertices? I'm trying to create a template for  a rubber stamp mold as in the following figure 

The idea is then to import an SVG curve which will be given a Z-dimension, and placed inside the template as in this figure: 

Then, the SVG mesh will be subtracted from the template using a boolean modifier. However the difference modifier does not seem to work, giving the message "Cannot execute boolean operation". At first I thought this might be due to conflicting normals, but I have checked this. I have also made sure there a no non-manifold faces.

I decided to eliminate the solids in the object one by one, and I discovered that I could delete all of the letters except "INL" and it still wouldn't work. 

However, once any of the remaining letters are deleted, the difference modifier works as expected. Why does it not work on all the letters together?
I solved my problem by simply subsurfing the cylinders I used to make the larger object.

Explanation:

In the original design, you can see that the circular faces are whole (not subdivided into quads or triangles). In search of another solution using the knife projection tool, I discovered that it does not work well on whole faces, and subdividing the face was the solution there. It then occurred to me that subdividing faces might help the boolean modifier approach. In fact it did work: the only other caveat was that the top surface of the cylinders aren't subdivided by the Subdivide command - you have to use the Subdivide Surface command first.