Awards when applying for faculty jobs
Best Paper Awards — especially at top conferences — count for a lot in computer science and can certainly help you stand out on the job market. I've seen people introduced multiple times in computer science venues as having won "many best papers awards" or "multiple best paper awards." People notice and people care.

Lots of other things matter as well and will matter more. A best paper awards at a conference nobody has heard of is unlikely to help much. In that sense, I don't think that hiring committees are "watching out for" best paper awards in any systematic ways. But I think it's absolutely normal to note your award winning papers as such in your CV and I think you definitely should. Having your work recognized as among the best at a conference will only help so there's no reason not to mention it.