Are hardware specs relevant in software performance comparisons?
Yes, hardware specs are relevant.  No, relative performance differences are not hardware-agnostic.

Suppose I find that editing a certain movie on my laptop takes twice as long as printing a certain Word document on my laptop.  Does this mean if I do those two tasks on my desktop computer, the movie editing task will also take twice as long as printing the Word document?  No, not necessarily.

Maybe my desktop computer has a great GPU (graphics card) but my laptop doesn't have a GPU.  The GPU can be expected to speed up the movie-editing task, but it won't speed up printing from Word.  So, on my desktop it's even possible that the movie-editing task might be faster than the printing task.

This is one reason why hardware specs are relevant.