Can I put acoustical putty *inside* the electrical box rather than around the outside?
As per the comments, no, it's not a good idea.

The box / soundproofing should have been installed before drywalling.  If you want to do it right, cut a piece of the drywall out to give you room to work and install it properly, then close up the wall again.