how does word variety depend on total words?
The simple answer is to use a sample of equal size from every book, or even better to randomly extract several samples of equal size from every book and then use the mean across samples.


  I tried simply dividing the number of unique words in each book by the number of total words in each book


This is known as the type/token ratio, the simplest way to measure lexical density. I think it makes perfect sense in the case you describe, as far as I know usually it's not too biased.