Primer Bulls Eye 123 is peeling off newly flattened walls with AG joint compound
It has been my experience that primer over drywall mud requires a looooooooog dry time. The primer is a water based product that will soften up drywall mud even over 1/8 inch thick. Your using the damp cloth to clean off dust instead of using a vacuum and tack cloths probably lent some to the softening as well. I had to wait a full week before I applied a second coat of the primer.

If you applied painters tape to primer when the underlying mud was even a slight amount damp the paint will come off.