Difference in continuing and episodic cases in Sutton and Barto - Introduction to RL, exercise 3.5
Is it just about final states? So for $s \in S$ when S is not final? 


You are thinking the right way, but to represent what you mean you don't need to write out "when $s$ is not final" - although that would be fine (and is used in some places), there is a more concise way of saying that given to you by the book.

As this is a formal exercise from the book, I don't want to write out an answer that could be cut&amp;paste for all students.

Instead I suggest you take a look at the notations section at the beginning of the book, and find how Sutton &amp; Barto use different set labels for all states including terminal states, and all states excluding terminal states. Also, check carefully which of those sets needs to be summed over.