What type of register and/or installation method to use for a inside duct mounting holes?
You say &quot;there was no bite behind the plaster when I drilled that pilot hole&quot;. I'm not sure if you mean that the screw didn't bite into something solid, or if, as your next sentence indicates, you were still drilling but hit something hard.
If you tried driving a screw into that and found that it wouldn't grip, that's not surprising for plaster. You'll want to use a molly/rawl plug/anchor that will go into an oversize hole in the plaster, then you drive the screw through the grill and into the molly. This will give your screw something to bite into, and the (usually) plastic molly is designed to hold in the plaster.
If you were still drilling and hit something solid, it may have been brick, or it could be a protective plate over a stud (though not likely that someone tried to run wiring that close to the vent, though it could be left over from a remodel). In that case, I'd still recommend some sort of wall anchor and a shorter screw to hold the grill in place so you don't have to drill any deeper.