A transmission line with continuously varying impedance, how would reflection occur in this case?
What you're asking about is called a transmission line taper. 

In general, there's no analytical solution to describe the reflections. The link in Chris L's answer (if you follow through to Klopfenstein's paper) gives some examples of specific taper shapes where something close to an analytical answer has been found.

The basic way to study it is to imagine breaking up the continuous taper into several   segments, each with a slightly different Z0 value. You calculate the reflections at each discontinuity and how they add up to give the overall reflection and transmission characteristics.

Then you split up the taper into finer and finer steps (with smaller and smaller discontinuities in Z0) until you have a good enough approximation to the continuous taper. You could try to calculate the results by hand but it's much easier to just get a computer program to do do it. Luckily, this kind of program is pretty easy to find --- it's called a finite element simulation program.