What is the difference between hypothesis space and representational capacity?
A hypothesis space/class is the set of functions that the learning algorithm considers when picking one function to minimize some risk/loss functional.
The capacity of a hypothesis space is a number or bound that quantifies the size (or richness) of the hypothesis space, i.e. the number (and type) of functions that can be represented by the hypothesis space. So a hypothesis space has a capacity. The two most famous measures of capacity are VC dimension and Rademacher complexity.
In other words, the hypothesis class is the object and the capacity is a property (that can be measured or quantified) of this object, but there is not a big difference between hypothesis class and its capacity, in the sense that a hypothesis class naturally defines a capacity, but two (different) hypothesis classes could have the same capacity.
Note that representational capacity (not capacity, which is common!) is not a standard term in computational learning theory, while hypothesis space/class is commonly used. For example, this famous book on machine learning and learning theory uses the term hypothesis class in many places, but it never uses the term representational capacity.
Your book's definition of representational capacity is bad, in my opinion, if representational capacity is supposed to be a synonym for capacity, given that that definition also coincides with the definition of hypothesis class, so your confusion is understandable.