My rear wheel keeps getting side to side wobble issues. Can it be too much pannier weight?
If the rim is aluminum made, truing it after a serious bend can result in fatigue of the material leading to repeated subsequent failures.

I'd try a sturdier replacement wheel instead of repeatedly truing it, especially if it's a job you pay for and not doing it yourself. 

If you are happy with the hub  you can keep it and get just a rim. The cost of building a wheel can be pretty high though, depending on where you live. 

The weight of the pannier should not count too much, unless you are a heavy rider and the total weight on the bike is near the specced limit of the wheel. This is usually the case for low-weight, racing, higher-end wheels (some have a 80-120kg limit) or for cheapo wheels with no specs and poor materials used.

The unbalanced weight should also not account for the issues you are having, given the fact the bike should be able to resist great side forces when turning, especially at high speeds.