How to patch siding half covered by electrical panel
That looks like T1-11 you should be able to clean the caulk out and install a new piece. I don’t like leaving voids because wood destroying insects may find a way in and make it home.
They probably put the panel there because it feeds the inside panel using the same opening.
Some lumber stores have partial sheets of T1-11. I would get a piece and match the pattern, if that is a seam just to the right I would cut that 1” piece out. Slip the new one in and caulk top and bottom. I would expect screws to be the best attachment method.
So clean out the caulk, remove the small strip, install new section and caulk the box and seams. Prime and paint to match.