Does the voltage difference have an effect on the electrons' speed?
tl; dr: yes, voltage affects electron speed, but not in the way you think.
With no voltage applied, electrons in a conductor rattle around at their Fermi velocity, which while high in its own right (0.81 x 106 m/s for copper), doesn't result in a net electron motion in one direction or another. That is, the vector sum of all the electron velocities is zero.
Applying a voltage to the conductor changes this: the electric field influences the Fermi-speed random motion to have a directional bias from negative to positive. This bias creates a net flow. The speed of that net flow, called drift velocity or \$v_d\$, depends on the material's charge density and the current.
That is,

From here:
http://hyperphysics.phy-astr.gsu.edu/hbase/electric/miccur.html#c1
We can see that for a given conductor, \$ne\$ (charge density) and \$A\$ (cross-sectional area) are constant, while \$I\$ variable. It follows that \$v_d\$ is a variable proportional to current.
That is,

\$v_d = \frac{I}{neA}\$

And by Ohm's law, we can relate voltage to current, and ultimately, \$v_d\$:

\$v_d = \frac{E}{RneA}\$

This makes sense: the electrons are being accelerated in a net negative-to-positive flow by the applied electric field, creating the current flow. The greater the the field, the greater the acceleration, and thus the greater the current.
Finally, don't confuse drift velocity with signal propagation. Drift velocity is fairly slow, on the order of cm per second. Signal propagation speed in the other hand is a significant a fraction of the speed of light (0.8C for coax cable, for example). Why? Signals propagate as electromagnetic waves, not as moving electrons.
Related answer: https://physics.stackexchange.com/questions/376452/why-is-current-slowed-down-by-resistance
And, about Fermi velocity: https://physics.stackexchange.com/questions/150015/how-to-calculate-the-speed-of-electrons-in-a-metal