What techniques do high level SET players use when looking at a layout of cards?
For purposes of describing things, I'm going to divide sets in "easy" sets -- those with 2 or 3 matching characteristics and only 1 or 2 non-matching ones, and "hard" sets -- those with NO matching characteristics.  I think this corresponds to how hard the sets are to spot for most people -- the easy sets will get spotted right away by most people, while the hard sets are hard to see even when they're pointed out.

So when the cards are first revealed, I glance at them without concentrating on any particular card to see if there's an easy set.  If there is, it will generally get spotted by someone immediately.  If I don't see an easy set after a second or two, I start looking for a hard set.  First step there is to figure out which attributes are rare in the array -- if there's only a single red card or only a single solid card, any hard set will necessarily include that card.  Any attribute that appears on only two or three cards means that one of those cards will be in any hard sets.  This generally means you can narrow things down to 5-10 pairs of cards that differ in all attributes, and any possible hard set must include one of those pairs.  Then for each of those pairs, figure out what the third matching card is, and see if its on the table.

The process above will generally only take around 10 seconds or so.  By that point, if noone has seen a set there probably isn't one, but you can start looking for sets with 1 matching characteristic -- Look for the single most common characteristic on the table, and see if any three of those form a set.