How can I secure this metal 1-gang electrical box without creating a bigger hole?
Before we even get started, there's a bigger issue here.

Before discussing the box and opening, note that the armored cable is not properly connected to the box.  There are boxes with internal clamps for armored cable but I do not recommend using those, I'd get fittings that connect the armored cable through knockouts with a locknut.  If you're not 100% sure of yourself using these fittings with armored cable, call an electrician to make this repair, getting it wrong can be dangerous.  It is extremely important to install these correctly without damaging the insulation of the wires on the sharp edge of the armor.  

The rest of this task is pretty simple but like most things, the devil is in the details;  you have to select a suitable box, and the details are very important.  The details are more complicated than they may seem at first glance.  

You probably can't use the existing hole as-is.

The original question is how to make this right without making a bigger hole.  However, code demands that there be little to no space between the box and the opening in the plaster / drywall.  

You know how with a haircut, it's very easy to make it shorter, but just about impossible to make it longer?  It's kind of the same way with the hole in the wall.  It's easy to enlarge it, but a lot of work to make it smaller.  

So for practical purposes, you either you need to find a box that's just the size of this hole, or find a larger box that works, and enlarge the hole.  I don't think you're going to find a suitable box the exact size of this hole that meets all the requirements.  

What kind of box will work?  

Metal Box

To simplify bonding the cable armor, it's simplest to stick to metal boxes.  A plastic box would be nice because they are a little bigger than metal boxes, but bonding the cable armor presents an additional challenge.  I think a decent solution is possible with a metal box, so I won't go into this more complicated possibility.  

Device Box

You want a device box - one with threaded holes in the face of the box that you can screw devices (switches or receptacles) to.   Device boxes are also known as switch boxes, even though you can use a switch box for receptacles as well - that's why I prefer the less-used term "device box."  You can see that the existing box is a device box - but that isn't all you need.  

Old Work Box

Old work (retrofit) boxes have plaster ears - flanges to keep the box from falling into the wall.  The existing box does not have plaster ears.  The box below has plaster ears on the top and bottom:  



Some old work boxes also have hold fast clamps that swing out and move in when you turn a screw, and hold the box from the inside, to keep the box from falling out of the wall:  

 

but those are optional;  you can also use F-clips to keep the box from pulling out of the wall.  F-clips are shaped like the letter "F" cut out of sheet metal, and are installed when the box is in place to keep it from pulling out of the wall:  



(Edit:  in a subsequent comment the OP adds that the box will be going through a baseboard molding.  In that case, it's actually easier to use the screw holes in the plaster ears to hold the box to the molding rather than hold fast clamps.  So for this application, a box without holdfast clamps is preferred.)   

Same dimensions or larger than the existing opening

As mentioned above, code requires that the opening fit tight to the box, and it's hard to make a hole smaller.  It will be difficult or impossible to find a single gang old work device box to fit the existing opening.  So you'll probably be forced to use a two gang old work device box, and enlarge the existing opening.  

You can install a second duplex receptacle in the same box, or use one receptacle and a half-blank two gang cover.  

Knockouts

To keep things simple, you'll want a box with pre-punched knockouts for your armored cable fittings, and you'll want them in places where the fitting won't interfere with getting the box into the opening. The existing box side knockouts may have made it difficult or impossible to get the box into the opening with armored cable fittings attached.  

If you have to use knockouts on the back of the box, you may need to use right angle fittings unless the wall hollow is very deep.  

With all this considered, you want a two-gang metal old-work device box, with usable knockouts.  If the box doesn't have hold-fast clamps, you need F-clips too.  (Edit:  as mentioned above - with the box set in a wood base molding, hold fast clamps nor F-clips are needed, you can screw the plaster ears / flange into the bade molding and it will hold it securely.)   

This box would check all the boxes

Just as an example, the Garvin Industries MGSB-20W would work:  



It has hold fast clamps on the sides, so you won't need F clips.  (Edit:  since you're going into wood molding, you can actually just pry off the hold fast clamps.) A deeper box would be ideal, but this one will work.  The KO's are probably too close to the face of the box to enter the box on the top or sides, but it's no big deal to buy right angle armored cable fittings, so this box would work.