How to remove junction box that is nailed on?
The two items that you were thinking are nails are in fact almost 100% sure to be rivets. See yollow circles in the below picture.

Picture Source
These rivets secure some type of metal bracket to this phenolic/bakelite box. The bracket is then either nailed to the adjacent ceiling joist or to a rail that bridges between two joists. Trying to remove this box without entering the area above (if that is even possible) will usually require opening up the ceiling drywall or plaster leading to a much bigger repair job.
It would not be advisable to replace this box with an old work type box that is just secured to the edges of the ceiling drywall. Such box would never been up to handling the installation of a ceiling fan/light fixture. Ceiling lights often get replaced with heavy fixtures like that even if it is not you doing it right now.
Here is a picture of the type of box that you likely have there. The rivets and metal bracket are readily obvious.

Picture Source
Here is yet another almost similar phenolic electrical box with a different style metal bracket riveted to it:

Picture Source