Systematic Approach to Finding Number of Isomers
To my knowledge, there is no direct method or formula to find out how many isomers for given framework of an organic compound. Interestingly, this could be a good question for a mathematician. The only formula of this kind is the one to find theoretical optically active stereoisomers of a compound if the number of stereocenters (chiral centers) are given.

For instance, the given framework has 4 stereo centers, hence theoretically, you have $2^4 = 16$ stereoisomers for that compound alone. If your book says it has 64 stereoisomers, it probably has counted geometric isomers of conjugated double bonds (E/Z isomers) as well. All these 64 isomers are illustrated as compound 1 in the scheme below where each red asterisk represent a chiral center. Note that, since the ring has three different substitution groups, none would lead to symmetry in the molecule. Thus, all theoretical amount is equal to the actual number of isomers. Now, question is whether you need to count the stereoisomers for this framework only or not. If it is only for this framework, then the number is 64. But, if you want to change the framework by moving the substitution around, that would be a different story.



Now, for example, if you just move the carbonyl group around three other available carbons (say, move to the carbon between two hydroxy groups as shown in the compound 2 in the diagram), then you have additional 64 stereoisomers by that single move (compound 2). There are other 4 movable groups in the ring and all of them create multiple combinations such as 2,3,4,5-substituted cycloocta-1-one. Also note that chloro group and one of the hydroxyl groups can switch their positions to give another option. Or, the chloro group can just occupy the ring as well, creating another chiral center (e.g., 6-alkenyl-3-chloro-1,4-dihydroxy-8-methylcycloocta-1-one) or move to side chain. Upon all, there are changeable positions for the two double bond on heptyl chain as well (see compound 3). Double bond(s) in the side chain and an alkyl radical(s) can also move to ring (as methy, ethyl, etc.) are also possibilities. Thus, you are going to have tons of stereo isomer as any moving to the ring creating more stereo centers. Good luck on that, if you are not a mathematician.