On start servos rotates to 180 degree
You will need to open up the VarSpeedServo library source code and read through it to understand how the library actually works with your sketch. 

Consider that you have not setup an initial position before issuing your first "slow moving" command.  It may be by setting an initial position before assigning a pin to the servo, you may prevent any unexpected movement.

Try something (for your code) like "myServo.wirte(0,0)" where the first argument is the position and second is the speed.