title: How do you wrap synchronous network I/O trivially with Tokio?
body: <p>There is an evident lapse in my understanding on concurrent development in Rust unfortunately. This question stems from weeks repeated struggles to solve a seemingly &quot;trivial&quot; problem.</p>
  <hr />
  <h1>Problem Domain</h1>
  <p>Developing a Rust library, named <a href="https://github.com/JuxhinDB/twistrs" rel="nofollow noreferrer">twistrs</a> that is a domain name permutation and enumeration library. The aim and objective of the library, is to be provide a root domain (e.g. <code>google.com</code>) and generate permutations of that domain (e.g. <code>guugle.com</code>) and enrichment that permutation (e.g. it resolves to <code>123.123.123.123</code>).</p>
  <p>One of its objectives, is to perform substantially faster than its <a href="https://github.com/elceef/dnstwist" rel="nofollow noreferrer">Python counterpart</a>. Most notably, network calls such as, but not limited to, DNS lookups.</p>
  <h1>Currently Design Proposal</h1>
  <p>The idea behind the library (apart from being a learning ground) is to develop a very trivial security library that can be implemented to meet various requirements. You (as a client) can choose to interact directly to the <a href="https://github.com/JuxhinDB/twistrs/blob/master/twistrs/src/permutate.rs" rel="nofollow noreferrer">permutation</a> or <a href="https://github.com/JuxhinDB/twistrs/blob/master/twistrs/src/enrich.rs" rel="nofollow noreferrer">enrichment</a> modules internally, or use the library provided async/concurrent implementation.</p>
  <p><a href="https://i.stack.imgur.com/XSJr4.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/XSJr4.png" alt="Twistrs proposed architecture" /></a></p>
  <p>Note that there is <em>no</em> shared state internally. This is probably very inefficient, but somewhat meaningless for the time being as it prevents a lot of issues.</p>
  <h1>Current Problem</h1>
  <p>Internally the DNS lookup is done synchronously and blocks by nature. I'm having trouble turning this into concurrent code. The closest I could get was to use tokio mpsc channels, and perform spawn a single tokio task:</p>
  <pre class="lang-rust prettyprint-override"><code>use twistrs::enrich::{Result, DomainMetadata};
  use twistrs::permutate::Domain;
  
  use tokio::sync::mpsc;
  
  #[tokio::main]
  async fn main() {
      let domain = Domain::new(&quot;google.com&quot;).unwrap();
  
      let _permutations = domain.all().unwrap().collect::&lt;Vec&lt;String&gt;&gt;();
  
      let (mut tx, mut rx) = mpsc::channel(1000);
  
      tokio::spawn(async move {
          for (i, v) in _permutations.into_iter().enumerate() {
              let domain_metadata = DomainMetadata::new(v.clone());
  
              let dns_resolution = domain_metadata.dns_resolvable();
  
              if let Err(_) = tx.send((i, dns_resolution)).await {
                  println!(&quot;receiver dropped&quot;);
                  return;
              }
          }
      });
  
      while let Some(i) = rx.recv().await {
          println!(&quot;got: {:?}&quot;, i);
      }
  }
  </code></pre>
  <p>That said, an astute reader will immediately notice that this blocks, and effectively runs the DNS lookups synchronously either way.</p>
  <p>Trying to spawn a Tokio task within the for-loop is not possible, due to <code>move</code> being done on the <code>tx</code> (and <code>tx</code> not impl <code>Copy</code>):</p>
  <pre class="lang-rust prettyprint-override"><code>for (i, v) in _permutations.into_iter().enumerate() {
      tokio::spawn(async move {
          let domain_metadata = DomainMetadata::new(v.clone());
  
          let dns_resolution = domain_metadata.dns_resolvable();
  
          if let Err(_) = tx.send((i, dns_resolution)).await {
              println!(&quot;receiver dropped&quot;);
              return;
          }
      });
  }
  </code></pre>
  <p>Removing the <code>await</code> ofcourse will result in nothing happening, as the spawned task needs to be polled. How would I effectively wrap all those synchronous calls into async tasks, that can run independently and eventually converge into a collection?</p>
  <p>A similar Rust project I came across was <a href="https://github.com/mersinvald/batch_resolve" rel="nofollow noreferrer">batch_resolve</a>, which does a tremendous job at this (!). However, I found the implementation to be exceptionally complicated for what I'm looking to achieve (maybe I'm wrong). Any help or insight to achieve this is greatly appreciated.</p>
  <p>If you want a quick way to reproduce this, you can simply clone the project and update the <code>examples/twistrs-cli/main.rs</code> using the first code snippet in this post.</p>