Cuing a location in space produces a short-lived advantage in reaction time to targets at that location. This early advantage, however, switches to a reaction time cost and has been termed inhibition of return (IOR). IOR behaves differently for different response modalities, suggesting that it may not be a unified effect. This letter presents new data from two experiments testing the gradient of IOR with random, continuous cue-target Euclidean distance and cue-target onset asynchrony. These data were then used to train multiple diffusion models of saccadic and manual reaction time for these cuing experiments. Diffusion models can generate accurate distributions of reaction time data by modeling a response as a buildup of evidence toward a response threshold. If saccadic and attentional IOR are based on similar processes, then differences in distribution will be best explained by adjusting parameter values such as signal and noise within the same model structure. Although experimental data show differences in the timing of IOR across modality, best-fit models are shown to have similar model parameters for the gradient of IOR, suggesting similar underlying mechanisms for saccadic and manual IOR.