Does speed or angle of attack generally have the greatest impact on total induced drag?
The change in angle of attack --as an end result of changes in speed-- causes the greatest difference in induced drag.

Induced drag drops with increases in speed because the faster-flying wing needs to deflect air by less in order to create the same lift.

Please read this answer which tries to explain induced drag much more intuitively than what you probably have read until now. With more air flowing past the wing per unit of time at higher speed, less deflection is needed, so the backward tilt of the lift vector is smaller. Since the weight of the aircraft should not change with speed, the same aerodynamic force has a smaller backward component at higher speed or in denser air.