title: How to implement an async task queue with multiple concurrent workers (async) in dart
body: <p>My goal is to create a kind of web-crawler in dart. For this I want to maintain an task queue where the elements are stored that need to be crawled (e.g URLs). The elements are crawled within the crawl function which returns a List of more elements that need to be processed. Thus these elements are added to the queue. Example code:</p>
  <pre class="lang-dart prettyprint-override"><code>import &quot;dart:collection&quot;;
  final queue = Queue&lt;String&gt;();
  main() async{
    queue
      ..add(&quot;...&quot;)
      ..add(&quot;...&quot;)
      ..add(&quot;...&quot;);
    while (queue.isNotEmpty) {
      results = await crawl(queue.removeFirst());
      queue.addAll(results);
    }
  }
  
  Future&lt;List&lt;String&gt;&gt; crawl(String x) async {
    ...
    res = await http.get(x)
    ...
    return results;
  }
  </code></pre>
  <p>This code of coarse only processes one element at a time. However I want to have a pool of workers (for example 5) that take elements out of the queue and process them at the same time, and add the results back to the queue. Since the bottleneck is the HTTP Request I think a Future.wait() call with multiple workers could speed up the execution. However I dont want to overload the servers and thus I also want to limit the amount of workers.</p>
  <p>Can this be realized with basic async primitives and semaphores? I would like to avoid isolates if possible in order to keep the solution as easy as possible.</p>