Will a drywall anchor work fine after it's already open or engaged?
It depends on the type of anchor and the length of the screw, but I think you may have another problem.

Most drywall anchors rely on the screw to expand the anchor and secure it into the drywall.  In these cases, if the screw is not into the anchor to an adequate depth, the anchor will not have sufficient purchase in the sheetrock.

With this said, however, if you are putting up a shelf that you plan to put any sort of weight on (books, etc.), sheetrock anchors are a very poor choice.  You would definitely want the majority of your anchors secured into a stud.

If you must anchor to the sheetrock, then you may want to consider butterfly anchors (toggle bolts) rather than the screw in surface anchors.  They will provide better support.