Arduino - ServoMotor Control and SoftwareSerial
When SoftwareSerial enters an interrupt each time it receives a character and busy-waits (delays) to time in each data sample as the bits in the character come in.  No other interrupt service routines can run while this is happening.  It also turns off interrupts when it sends.
The servo library for the AVR uses timer interrupts to generate the servo position signals.
When characters are being received, SoftwareSerial will interfere with the generation of the servo position signal because only one interrupt service routine can run at a time.
You may be able to mitigate the problem adding significant delay between characters sent, reducing the total number of characters sent.  This would limit the interference and give servo time to get back to its correct position from however far it managed to deviate over the course of an inbound serial character.
There are also alternative libraries that use the AVR's Input Capture feature on an ICP pin to do some of the work of serial in hardware, and that may work for you.  Here's one example library.
@ocrdu has rightly prompted me to add that there are servo libraries that will do the servo signal generation in hardware.