what does "\" symbol mean?
The \ symbol is often used in mathematics for set difference, besides dash or minus symbol (-)

Since tables and relations are very similar to sets and the operations of union, intersection and difference are similarly defined (see Wikipedia: Relational algebra, set operators), the \ symbol is also sometimes used for difference. 

If the Q = R\S expression is found in a database theory book and Q, R and S are relations, it means that Q is a relation (table) that has the same attributes (columns) as R and S and also has those tuples (rows) that are in relation R but not in relation S.