Performance of an OLX crawler I'm learning concurrency and asynchronous programming in C#. For a better grasp on the matter I'm trying to create an OLX Crawler. My issue here is the performance: It takes roughly 159 minutes to crawl 17.901 links.

The method which crawls each item is this:

public async Task FetchItemsData()
    {
        var itemsTask = new List&lt;Task&lt;string&gt;&gt;();
        string itemUrl = string.Empty;
        var semaphore = new SemaphoreSlim(150);

        while (this.queue.TryDequeue(out itemUrl))
        {
            await semaphore.WaitAsync();
            try
            {
                var source =  Task.Run( () =&gt; this.GetSource(itemUrl) );

                itemsTask.Add(source);

                await Task.Delay(1);
            }
            finally
            {
                if (itemsTask.Count == 150)
                {
                    await Task.WhenAll(itemsTask);
                    ItemSourceParser itemParser = new ItemSourceParser();

                    foreach (var task in itemsTask)
                    {
                        await task.ContinueWith(async _ =&gt;
                        {
                            itemParser.InitializeNewItem(_.Result);
                            itemParser.TryToParse();
                            Console.WriteLine(itemParser.data["Id"]);
                            this.database.InsertItem(itemParser.data);
                            itemParser.RefreshData();
                            await Task.Delay(1);
                        });
                    }

                    itemsTask.Clear();
                }
                semaphore.Release();
            }
        }

       /* FURTHER PROCESSING */
    }


The queue is a ConcurrentQueue which holds the item string. In the finally block I'm trying to await 150 concurrent tasks to not overload the server.

The GetSource method is here:

public async Task&lt;string&gt; GetSource(string url)
    {
        HttpResponseMessage responseMessage = new HttpResponseMessage();
        try
        {
            this.ProgressUpdater.ProgressBarUpdater.Report(1);
            responseMessage.Version = HttpVersion.Version10;
            responseMessage = await this.client.GetAsync(url);
        }
        catch (Exception e)
        {
            this.ProgressUpdater.LogUpdater.Report("Connection timeout on: " + url + ". Retrying after 0.5 seconds");
            await Task.Delay(1500);
            this.ProgressUpdater.ProgressBarUpdater.Report(-1);
            return await this.GetSource(url);
        }
        return await responseMessage.Content.ReadAsStringAsync();
    }


I'm guessing that something is not right. Is there any redundant code that needs to be removed or something that I should have done so that the crawler has a better performance?
You don't have to use a SemaphoreSlim because a Task is not equivalent to a Thread. It is a more abstract kind of "working item" that will be spreaded to real threads by the TPL. If you start 1000 tasks it does not mean that you create 1000 threads. When starting a task like you did (without option TaskCreationOptions.LongRunning), it will be executed on threads from the TreadPool. Therfore, the Threadpool is managing the number of threads for you and you can just start 17.901 tasks at once.



What is await Task.Delay(1) for?

If there is no reason for it, it just produces unnecessary overhead



var source =  Task.Run( () =&gt; this.GetSource(itemUrl) );


You don't need to wrap the returning taks in a new task. The following has less overhead and is functional equivalent:

var source =  this.GetSource(itemUrl);




Move the code from the finally block to the call of GetSourceAsync. There is no need to attach it via ContinueWith in the finally block.



All the points above applied yields to something like:

async Task FetchItemsDataAsync()
{
    var itemsTask = new List&lt;Task&lt;string&gt;&gt;();
    var itemParser = new ItemSourceParser();

    while (this.queue.TryDequeue(out itemUrl))
    {
        var task = Task.Run(async() =&gt;
        {
            var result = await this.GetSourceAsync(itemUrl);
            itemParser.InitializeNewItem(result);
            itemParser.TryToParse();
            Console.WriteLine(itemParser.data["Id"]);
            this.database.InsertItem(itemParser.data);
            itemParser.RefreshData();
        }
        itemsTask.Add(task);
    }

    return Task.WhenAll(itemsTask);
}