Hough lines and Convex Hull methods give jagged lines
Let's go step by step.


First of all you can remove perspective distortion without camera calibration. e.g. 


  Robust Radial Distortion from a Single Image, Faisal Bukhari and Matthew N. Dailey
  
  Robust Line Based Calibration of Radial Distortion from a Single View, Thorsten Thormahlen, Hellward Broszio, Ingolf Wassermann

For any given image, Hough transform might find multiple lines, many of those would be an indicator of the same edge. But luckily you could easily merge the lines which overlap. To detect the overlap you could well use the angle between the lines and the distance of lines. This way, you obtain a clustering of the lines. In any case, finding a short line on each edge would be sufficient for you to recover the projective quad.
If you are always sure that the orientation of this quad is more or less fronto parallel, then you could as well use directed gradients (X or Y) to emphasize the vertical or horizontal lines, during the edge detection step. 


I agree with the other comments on Gaussian smoothing and anti aliasing. I would not recommend the use of convex hulls though, as they are very noise sensitive.