TPL Thread Leak and Memory Leak
There are several issues in your code that make it error-prone:


you are using a single byte array to feed all memory streams, thus the same data may be sent in different chunks. Same stands for count.
the order of chunks in blockids can be different to the one you've sent so resulting BLOB will have incorrect order of blocks.
there is a potential thread contention on blockids as different threads may add a new element at the same time causing all sorts of problems like missing blocks in resulting list, incorrect block IDs etc.
use Task.Factory instead of new TaskFactory()
if you create a task and immediately wait for it then you don't need that task, so you can replace the last BeginPutBlockList with synchronous version


Other than that I don't see memory leak from the graphs you provided. Threads may still be alive in a thread pool, and memory may not be collected by GC yet.

Here is what I've got as a result (note that it may consume more memory as you need to store all the chunks that are currently being sent in memory):

    private static Task PutBlockAsync(CloudBlockBlob blob, string id, Stream stream, RetryPolicy policy)
    {
        Func&lt;Task&gt; uploadTaskFunc = () =&gt; Task.Factory
            .FromAsync(
                (asyncCallback, state) =&gt; blob.BeginPutBlock(id, stream, null, null, null, null, asyncCallback, state)
                , blob.EndPutBlock
                , null
            );
        return policy.ExecuteAsync(uploadTaskFunc);
    }

    public static Task ChunkedUploadStreamAsync(CloudBlockBlob blob, Stream source, BlobRequestOptions options, int chunkSize, RetryPolicy policy)
    {
        var blockids = new List&lt;string&gt;();
        var blockid = 0;

        int count;

        // first create a list of TPL Tasks for uploading blocks asynchronously
        var tasks = new List&lt;Task&gt;();

        var bytes = new byte[chunkSize];
        while ((count = source.Read(bytes, 0, bytes.Length)) != 0)
        {
            var id = Convert.ToBase64String(BitConverter.GetBytes(++blockid));
            blockids.Add(id);
            tasks.Add(PutBlockAsync(blob, id, new MemoryStream(bytes, 0, count), policy));
            bytes = new byte[chunkSize]; //need a new buffer to avoid overriding previous one
        }

        return Task.Factory.ContinueWhenAll(
            tasks.ToArray(),
            array =&gt;
            {
                // propagate exceptions and make all faulted Tasks as observed
                Task.WaitAll(array);
                policy.ExecuteAction(() =&gt; blob.PutBlockList(blockids));
            });
    }