I do not have power to all my breakers
Oh dear.  This is a foogly mess.

First, you did the right thing by punching that main panel breaker down onto a single.  The problem is with the subpanel; it is very badly misconfigured by a guy who cut a lot of shortcuts.  

First, it is illegal to double-tap neutral bar screws like that, unless the panel's labeling or instructions say they are intended for that, and these aren't.  A few panels will allow 3 grounds on a screw, but always one neutral.  

Second, I see no cable clamps on the Romex cables entering this panel.  Run down to the electrical supply house and grab a handful. 

Separate neutrals and grounds

This sub panel has neutrals and grounds on the same bar. That is improper in a subpanel.  Those neutrals and grounds need to be separated, and the grounds need to be attached to the subpanel chassis/frame.  The ideal way to do this is to install an accessory  ground bar for the panel - however the last guy didn't leave you much length to reach it.  

Option 1: The panel may have factory drilled/tapped screw holes for accessory grounding bars. Fit them and buy a Square D brand grounding bar that lines up on those holes.  Extend the ground wires down to it with wire nuts.  There is a green grounding screw that connects the neutral bar to the box chassis; it appears to already be removed. Remove it. 

Option 2: Squeeze a ground bar above the neutral bar, but it must be physically screwed down to the panel, it can't flop around.  To bolt it down, either a) use any old metal screws to physically mount it, then run an actual #10++ (#8 would be great) ground wire to a suitable place on the chassis.  Or b) mount the ground bar with fine-thread machine screws (bolts) of -32 or finer pitch, and then you can ground the bar through the screws. 

Option 3: Convert the neutral bar to a ground bar only. You'll never find that green grounding screw, so just run a jumper wire from this bar to the panel chassis. Remove all the neutrals, and use wire nuts and a pigtail chain to attach supply neutral to the other neutrals. Inelegant, but effective. 

Supplying the two hot terminals

The black hot wire coming in needs to be split to supply both hot terminals.  (the last guy did a very ugly cheat he should not have done).  For this, get about 2 feet of #8 copper wire, black will do, and make 2 pigtails off the solitary hot wire.  One goes to where the black wire now is.  The other goes to where the ground wire now is.  Ideal's big blue wire nut is about right to join three #8s, crank it down HARD.

This converts the panel to a 120V panel. It is not legal to power this subpanel in 120/240V mode, with only 3 wires, as it was before - that would create a very dangerous situation, where if the neutral wire had a problem, everything that is supposed to be grounded would energize at 120V.  

The panel will only able to supply 40A for all circuits added together, somewhat less than the 2 poles of 40 A that had been cheated in here.  So you are at somewhat higher risk of tripping the supply breaker, but it probably won't be an issue. 

Is it conduit? Add the 4th wire

If the route of the wire between subpanel and main panel is all conduit, a red wire of the same size should be added to the pipe.  Then it can be hooked up like a normal panel.  

While you're at this, maybe replace this panel.

Thinking about all the work needed here - ground bars, cable clamps, etc. - and the fact that this panel is already full - and that you may want to replace with a 4-wire cable which you might as well upsize too for more power...

-- well, given that, I'm thinking this might be a good time to just replace the whole panel with a much larger one.  

For instance a 20 space would be a step in the right direction (ThreePhaseEel and myself would certainly go 30+, because spaces are cheap, and being out of space is expensive).  By "space" I mean actual knockout, ignore that "20 space/40 circuit" baloney, that is a lie since nowadays most breakers must be GFCI or AFCI, and those require a full space each.   (this requirement won't affect you since  you are grandfathered; merely replacing a panel does not trigger the obligation to retrofit AFCI/GFCI). 

If you want to keep the "sunk costs" you already have in those six breakers (we're talking $27 here) then you could stay in Square D Homeline - which is exactly what it says on the tin, a builder-grade cheapie (not unsafe, just not excellent).  The way panels are marketed, there's hardly a price difference to get up into the industrial grade units like Square D's QO line, Eaton CH, GE or Siemens. Also, QO and CH are physically smaller. 

So you might shop for a panel  that will fit, has plenty of extra spaces, and where the neutral and ground wires will reach the bar without extending (we already must split the hot wire). Any type will do; a main breaker is unnecessary here.  Some panels come with "bonus breakers".