What do Arora and Barak mean by $x|_S$ in their definition of certificate complexity?
It is a typo. $S$ indeed should be a subset of $[n] = \{1,2,\ldots,n\}$. On page 263, in the second sentence of their proof of Theorem 12.5, they list a 0-certificate (or 1-certificate) as a subset of $[n]$.

Because $S \subseteq [n]$, it is easy to understand $x|_S$ (as is stated in the question). This is just the usual notation for the restriction of a function on a subset of its domain. Note that an $x \in \{0,1\}^n$ can be thought of as a function from $[n]$ to $\{0,1\}$.