book arts: how to make cut outs
You are using a typical tool by having an Xacto knife in hand. If your results are poor, please consider that the blade may be dull. New blades are sharp enough to result in very clean cuts, while an even slightly dull blade will pull the material being cut.

If you don't wish to exhaust your finances by purchasing new blades at every turn, consider to purchase also a sharpening stone and stone oil (often water based) to hone the blades you have. A newly honed blade will slice through paper like a hot knife through butter, but with far less mess.

There are some references online suggesting to use sandpaper, but for best results, a fine grain honing stone and oil provides best results.

You can improve your results by making use of a cutting mat specifically designed for this purpose. Look for the terms "self-healing" in the ad copy or on the label. The material and construction allow the knife blade to penetrate slightly into the surface, but close up in short order.

Because your reference to art implies freehand shapes, a typical guillotine cutter would not be practical, nor would other shear type cutters or rotary blades.

On the more extreme level, one could make use of a laser cutter. Low power hobby-grade lasers tend to leave burned edges, while the higher power CO2 hobby lasers will provide a much cleaner cut. There is no "freehand" involved in such devices and would require computer generated images, removing much of the art from the craft.

I nearly forgot one option if non-freehand images are acceptable. Less expensive, by far, than any laser cutter and very clean cuts can be had by using a hobby cutter. Known by a couple brand names such as Cricut and Cameo, they may provide an option for your project.

See also this most recent post:

Laser Cut Wedding Invitations