How to use Common Table Expression properly - Example where it seems to be very slow in TSQL
To expand a bit more on Jonathan's comment, each time you reference a CTE, the query within the CTE will be called.

Each UNION you are using is creating a separate execution of the CTE, thereby executing the table value function each time as well. By putting the results into a temp table, you are only then reading the static data in the temp table for each UNION and avoid recalculating the data.

If you check the execution plan for each of the queries, you'll be able to see a visual confirmation of this fact. 

In your case, it makes much more sense to use a temp table as illustrated by the vastly different execution times. I personally find non-recursive CTEs rarely offer much by way of performance and are usually more helpful for readability  and logical separation of query contents.