Adafruit GPS module with SoftwareSerial affecting servo signals
The bad news is that you are using the absolute worst libraries (SoftwareSerial and Adafruit_GPS), and your program structure is based on the awful Adafruit_GPS example.

The good news is that there are much better alternatives: NeoSWSerial and NeoGPS.  Still, there's no doubt that you are near the limits of this processor, especially with Wire and timer-based Servos.  It can be an "interesting challenge" or a "beginner's nightmare."  

Suggestions:

1) Consider using the HardwareSerial object, Serial, for the GPS interface.  Your excerpt does not show what Serial is currently used for, but if it's debug print statements... don't.  Instead, use a software serial port for debug statements (use a serial-to-USB adapter on those pins to connect it to your PC).  This would have a tremendous impact.

2) If you really really can't use Serial for the GPS (are you sure???), consider using an AltSoftSerial instance on pins 8 &amp; 9, as this is much better than the dreaded SoftwareSerial.  Next best is NeoSWSerial on any pins, including 4&amp;3 that you are using now.  There is absolutely no reason to shoot yourself in the foot with SoftwareSerial.

3) As other's have commented, use HW PWM... let the machine do the work for you.

4) Consider using the faster and smaller NeoGPS library.  And restructure your program to look like the NeoGPS example programs, because their sequencing is much more efficient.  The Adafruit_GPS example uses a timer interrupt... to read a character from the input buffer... that was stored by a SoftwareSerial interrupt... that was on for the entire character time of ~1ms.  What a phenomenal waste!

5) Use interrupt-driven GPS parsing.  All three Neo__Serial libraries (HW, IC and SW) allow NeoGPS to handle the GPS characters during the rx interrupt, avoiding character loss because the port was not read in time (input buffer overflow).  And because just a little work is performed on each character, there is no big chunk of processing when a sentence is complete.  This also uses half the interrupts and saves copying the characters multiple times, in and out of the serial input buffer... into the Adafruit double-buffered sentences, where it gets searched multiple times with strcmp and strchr.  Ugh.  No, I meant to say UGH!

6) NeoGPS will also free up hundreds of bytes of RAM, which you may be able to trade for some more speed, elsewhere.

It's definitely a tight squeeze.  Either reallocate your pins and use better libraries, or get a bigger hammer. &nbsp;&nbsp;&nbsp;&nbsp; :-)