What should I expect the output of my line scanner to look like on an oscilloscope?
I expected the output to be a set of square waves the represented the
  pixels being outputted but isntead I just see a really shaky line


Pin 3 is called "analogue output". That should be a clue to tell you that the output is not digital. Here are some more clues in red: -