Why do we consider the order of growth when analyzing algorithms?
I think the key point in time complexity analysis is it can give us a metric to measure the time independent of any hardware or software dependency directly. 

Hence, we can talk about the time complexity based on inputs and outputs of the problem without considering more sophisticated or even complex cases by mentioning that we are talking about the hardware or software specifications.  

However, hardware usage is mentioned in some real applications too, somewhere the time complexity can't distinguish the solutions in its abstract asymptotic definition. These kind of analysis are common in AI algorithms which have the same time complexity but have some real differences in real applications and hardware usage. For example, in real applications "Quick Sort" is better than "Merge Sort" because we have a large constant factor in "Merge Sort" time complexity, however, on the paper, in time complexity, "Merge Sort" is better than "Quick Sort" in worst cases.

In sum, time complexity base on the growth can be proper to compare algorithms independent of any hardware or software specifications, although hardware usage in real applications makes sense more!