Electrical subpanel losing voltage
You most likely have a bad connection of the breaker to the sub-panel bus bar or a loose terminal somewhere. It could POSSIBLY be a bad pole on the 50A breaker, but that's fairly rare in comparison. You MUST address this immediately by the way, because it's a fire waiting to happen.