How to find Sum of dynamically generated time attribute in sql server 2012
Use your current query as a CTE (Common Table Expression) and select form that with aggregate functions, like:

WITH TimeDiffs AS (&lt;your query&gt;)
SELECT   taskid
       , SUM(diff)
FROM     TimeDiffs
GROUP BY taskid


(assuming you want the total per task, your question is not very clear on the details).

You could also use the existing query as a derived table:

SELECT taskid
     , SUM(diff)
FROM   (&lt;your query&gt;) AS TimeDiffs
GROUP BY taskid


which will produce the same result, though the CTE arrangement usually ends up being more readable and so easier to maintain going forward.