Why is the line from the camera to vanishing point parallel to the other parallel lines? When trying to find the vanishing point of an image, why is it that the vector from the center of the camera to the vanishing point is parallel to the parallel lines of the image? 

What's the mathematical reasoning behind this?
FYI every set of parallel lines will have their own vanishing point. The reason why there is usually only a single vanishing point (the most obvious one) talked about in an image is due to aesthetic choice.

A perspective projection can be seen as a point projection of a 3D scene onto a 2D plane. The way to determine where a point in the 3D scene ends up on the 2D plane is to draw a line between the point and the camera (the sight line) and see where it intersects the projection target plane.

To find the vanishing point of a set of parallel lines you can take a the limit of traveling on that line infinitely far. At the limit the sight line will be parallel to the line.