Rectangle detection in "real life" images
I think you would use a 2D Matched Filter. You would convolve your image with a series of rectangles. The peaks in the resulting images would be the location of your books. You could do this quickly in by Fourier transforming your image and using the known function for a rectange in 2D Fourier space (its two sinc functions, multiplied).