Would Z and E isomers show different optical activity?
Not necessarily. Although the change you suggest would not alter the absolute stereochemistry (S in both cases) of the asymmetric center starred, the two molecules are diastereomers and therefore will have different physical and optical properties. Even if they did both rotate light the same way, they probably won’t have the same degree of rotation for the same concentration (specific rotation).

The problem boils down to whether we can predict the direction (or degree) of rotation from a drawn absolute stereochemical configuration. The answer to this is still “no”. We don’t currently have a theory that will allow us to say that, for a certain molecule, R gives us d or l, etc.

Some studies have been able to come up with some rules of thumb for small molecules, but there is no comprehensive theory to make this prediction in general. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2704476/