Web crawler that uses Task Parallel Library This is the first time using the Task Parallel Library and it seems to be performing much slower than I was expecting. The application is simply a web crawler that returns all URLs for a given URL.

Here is my method that, depending on the level param, will either download single page links (getSinglePageLinks()) or it will make use of my threaded method (getManyPageLinks()):

public static IEnumerable&lt;string&gt; getLinks(string url, bool hostMatch=true, bool validatePages=true, int level=0)
    {
        string formattedUrl = urlFormatValidator(url);
        if (string.IsNullOrEmpty(formattedUrl)) return Enumerable.Empty&lt;string&gt;();
        //download root url's
        IEnumerable&lt;string&gt; rootUrls = getSinglePageLinks(formattedUrl, hostMatch, validatePages);
        //download url's for each level
        for (int i=0; i&lt;level; i++)
        {
            rootUrls = rootUrls.Union(getManyPageLinks(rootUrls, hostMatch, validatePages));
        }
        return rootUrls;
    }


getSinglePageLinks() makes use of the HTML agility pack, and it simply downloads and parses the given URL:

private static IEnumerable&lt;string&gt; getSinglePageLinks(string formattedUrl, bool hostMatch = true, bool validatePages = true)
    {
        try
        {
            HtmlDocument doc = new HtmlWeb().Load(formattedUrl);
            var linkedPages = doc.DocumentNode.Descendants("a")
                                              .Select(a =&gt; a.GetAttributeValue("href", null))
                                              .Where(u =&gt; !String.IsNullOrEmpty(u))
                                              .Distinct();

             //hostMatch and validatePages left out
             return linkedPages;
        }catch(...){...}


And my getManyPageLinks():

   private static IEnumerable&lt;string&gt; getManyPageLinks(IEnumerable&lt;string&gt; rootUrls, bool hostMatch, bool validatePages)
    {
        List&lt;Task&gt; tasks = new List&lt;Task&gt;();
        List&lt;List&lt;string&gt;&gt; allLinks = new List&lt;List&lt;string&gt;&gt;();

        foreach (string rootUrl in rootUrls)
        { 
            string rootUrlCopy = rootUrl; //required
            var task = Task.Factory.StartNew(() =&gt;
            {
                IEnumerable&lt;string&gt; taskResult = getSinglePageLinks(rootUrlCopy, hostMatch, validatePages);
                return taskResult;
            });

            tasks.Add(task);
            allLinks.Add(task.Result.ToList());
        }

        Task.WaitAll(tasks.ToArray());
        return allLinks.SelectMany(x =&gt; x).Distinct();            
    }


The app works OK if the level is set to 0, but if I set the level to 1 so it gets all links for all of the roots URL, the CPU/Network usage does not go above 1-3%. How can I improve performance?
From MSDN on Task.Wait:


  If the current task has not started execution, the Wait method
  attempts to remove the task from the scheduler and execute it inline
  on the current thread.


I have a feeling that something similar might be happening with WaitAll, killing performances. I'll have a look and see if I can find documentation about it. As you are using StartNew some of the tasks might be started already so they won't be in-lined.

I would refactor it using async/await so you are sure you are not using and blocking only one thread from the pool:

async static Task&lt;IEnumerable&lt;string&gt;&gt; GetAllPagesLinks(IEnumerable&lt;string&gt; rootUrls, bool hostMatch, bool validatePages)
{
    var result = await Task.WhenAll(rootUrls.Select(url =&gt; GetPageLinks(url, hostMatch, validatePages)));

    return result.SelectMany(x =&gt; x).Distinct();
}

static async Task&lt;IEnumerable&lt;string&gt;&gt; GetPageLinks(string formattedUrl, bool hostMatch = true, bool validatePages = true)
{
    var htmlDocument = new HtmlDocument();

    try
    {
        using (var client = new HttpClient())
            htmlDocument.Load(await client.GetStringAsync(formattedUrl));

        return htmlDocument.DocumentNode
                           .Descendants("a")
                           .Select(a =&gt; a.GetAttributeValue("href", null))
                           .Where(u =&gt; !string.IsNullOrEmpty(u))
                           .Distinct();
    }
    catch
    {
        return Enumerable.Empty&lt;string&gt;();
    }
}

async static Task&lt;IEnumerable&lt;string&gt;&gt; GetLinks(string url, bool hostMatch = true, bool validatePages = true, int level = 0)
{
    if (level &lt; 0)
        throw new ArgumentOutOfRangeException(nameof(level));

    string formattedUrl = FormatAndValidateUrl(url);

    if (string.IsNullOrEmpty(formattedUrl))
        return Enumerable.Empty&lt;string&gt;();

    var rootUrls = await GetPageLinks(formattedUrl, hostMatch, validatePages);

    if (level == 0)
        return rootUrls;

    var links = await GetAllPagesLinks(rootUrls, hostMatch, validatePages);

    var tasks = await Task.WhenAll(links.Select(link =&gt; GetLinks(link, hostMatch, validatePages, --level)));

    return tasks.SelectMany(l =&gt; l);
}


Haven't had a chance to test it, but just get the gist.