How to repair holes in wall and properly mount a clothes pole?
Jack is right.  Take down the old mounting piece.  Buy a new longer piece of plywood that will extend over to the nearest studs.  You can take the old wood piece into the store to use for color matching a pint of paint.  You could paint it outside before attaching it to the wall.
Then center the new piece along the wall for aesthetics and use heavy screws to screw it into the studs.  This would hide the old torn out drywall holes.  You can then mount the old hardware with L brackets.
Alternatively, you could try to mount a taller piece of wood that extends up to the ceiling and screw diagonally up to the wall's frame.
Just a thought... You might be able to see the studs from above.  I see the attic access right there.