Can I use any coarse drywall screws when fixing nail pops in the ceiling?
I would not use backer board screws they are designed to cut a bevel, 
The cut will tear the paper and that is your strength in Sheetrock.

You do not want to break the paper, this is even more important on ceilings! 
 use Sheetrock screws or bugle head Sheetrock screws these come in both fine and corse.