Difference in continuing and episodic cases in Sutton and Barto - Introduction to RL, exercise 3.5 Excercise 3.5 The equastions in Section 3.1 are for the continuing
  case and need to be modified (very slightly) to apply to episodic
  tasks. Show that you know the modifications needed by giving the
  modified version of (3.3).


$\displaystyle\sum_{s^{\prime} \in S} \displaystyle\sum_{r \in R} = p(s^{\prime}, r 
| s,a) = 1$ , for all $s\in S, a \in A(s)$    (3.3)

Is it just about final states? So for $s \in S$ when S is not final?
Is it just about final states? So for $s \in S$ when S is not final? 


You are thinking the right way, but to represent what you mean you don't need to write out "when $s$ is not final" - although that would be fine (and is used in some places), there is a more concise way of saying that given to you by the book.

As this is a formal exercise from the book, I don't want to write out an answer that could be cut&amp;paste for all students.

Instead I suggest you take a look at the notations section at the beginning of the book, and find how Sutton &amp; Barto use different set labels for all states including terminal states, and all states excluding terminal states. Also, check carefully which of those sets needs to be summed over.