Is there an adhesive I can use with anchors to keep them from pulling out?
You could try some "No Nails" or equivalent, though as you say it shouldn't be necessary.

The size of screw you are using will determine the size of the anchor you use. This in turn will determine the size of hole you drill.

The anchor should fit snugly into the hole and even require a light tap with a small hammer to ensure it fits flush with the surface. This will mean that the grippers on the sides will have a good purchase on the brickwork as you drive the screw home. It's this outwards force that will stop the anchor coming out of the hole.