Does publishing a book while employed by an academic institution grant them legal claim on the royalties?
In general, no. (This is based on my experience in the US.) One's employing institution does not have any claim on royalties from books written while a student or faculty member. However, it's possible that there might be exceptions: for example, sometimes a university may help financially in the publication of a book, and this might be reflected in the publication contract. Of course, in such circumstances the book is not expected to make any money, which is why the university is helping out in the first place. 

A related and very common phenomenon is for the publisher to hold the copyright on an academic title. My own book is like this. I am entitled to royalties, but the press holds the copyright for some defined period. Again, this is due to the terrible economics of publishing academic monographs. 

There have been cases where universities have tried to assert very broad rights over the intellectual property of their faculty employees (e.g., lecture notes as well as books, etc), but I think these have generally failed. It does still happen: here's an example where the University of Louisiana is trying to broadly claim rights to scholarly output, including royalties from books. These sweeping assertions of rights have sometimes been motivated by the desire of administrators to claim a share of some of the genuinely lucrative things now produced by some university researchers, such as patentable biotechnologies, with books caught up in the net but not really directly targeted. Patents are an area with real money at stake, where the university's investment (in laboratory space and so on) is much higher, and where university claims on income from work done while employed are strongly and successfully asserted. Books, not so much.