How to determine what the states should be in this state diagram
Define your initial condition in the 1st bubble with input X and output, Y

With 3 sequences in time one bit can present 8 different patterns thus 8 bubbles are the maximum required. But since only 1 combination produces an output, you can simplify or share bubbles that do not lead to a possible output in the next state one or two states and return to a shared state. 

In your case Y=1 only if past inputs were 110.  


Thus in bubble S0, if X=0 it stays in S0 and goes to S1 if X=1
Then in S1, X=0, it goes back to S0 or else X=1 it goes to S2 bubble.


And so forth..