Confusion about the SN1 and SN2 Mechanism
Your reasoning is correct. It's the relative rates for a given alkyl halide that matter, and it's difficult to draw conclusions about those, because the relative rates are not fixed, as the $\mathrm{S_N2}$ rate is dependent also on the nucleophile concentration. 

We can write the rate of the $\mathrm{S_N1}$ as $k_1[\text{alkyl halide}]$ and the rate of the $\mathrm{S_N2}$ as $k_2[\text{alkyl halide}][\text{nucleophile}]$, so the relative rate at a fixed concentration of the alkyl halide is given by

$$\frac{\mathrm{S_N2}}{\mathrm{S_N1}} = \frac{k_2[\text{nucleophile}]}{k_1}.$$

The identity of the nucleophile will be a factor as well, since that will affect $k_2$.

So all in all, you can't make a general conclusion that one reaction type will always go faster than the other, although it is likely that there are alkyl halides for which one reaction type is faster in a large majority of cases.