title: Check if an IP address is in a IPNetwork with Pyspark
body: <p>With Pyspark, I would like to join/merge if an IP address in the dataframe A is in a IP network range or hits the same IP address in the dataframe B.</p>
  
  <p>The dataframe A contains IP addresses only and the other one has IP addresses or IP addresses with a CIDR. Here is an example.</p>
  
  <pre><code>Dataframe A
  +---------------+
  |     ip_address|
  +---------------+
  |      192.0.2.2|
  |   164.42.155.5|
  |    52.95.245.0|
  |  66.42.224.235|
  |            ...|
  +---------------+
  
  Dataframe B
  +---------------+
  |     ip_address|
  +---------------+
  | 123.122.213.34|
  |    41.32.241.2|
  |  66.42.224.235|
  |   192.0.2.0/23|
  |            ...|
  +---------------+
  </code></pre>
  
  <p>then an expected output is something like below</p>
  
  <pre><code>+---------------+--------+
  |     ip_address| is_in_b|
  +---------------+--------+
  |      192.0.2.2|    true|  -&gt; This is in the same network range as 192.0.2.0/23
  |   164.42.155.5|   false|
  |    52.95.245.0|   false|
  |  66.42.224.235|    true|  -&gt; This is in B
  |            ...|     ...|
  +---------------+--------+
  </code></pre>
  
  <p>The idea I first wanted to try is using a udf comparing one by one and checking an IP range when a CIDR comes up but it seems udfs don't multiple dataframes. I also tried to convert the df B to a list and then compare. However, it is very inefficient and takes a long time as the A row number*the B row number is over 100 million. Is there any efficient solution?</p>
  
  <p>Edited:
  For more detailed information, I used the following code to check without pyspark and using any library.</p>
  
  <pre><code>def cidr_to_netmask(c):
      cidr = int(c)
      mask = (0xffffffff &gt;&gt; (32 - cidr)) &lt;&lt; (32 - cidr)
  
      return (str((0xff000000 &amp; mask) &gt;&gt; 24) + '.' + str((0x00ff0000 &amp; mask) &gt;&gt; 16) + '.' + str((0x0000ff00 &amp; mask) &gt;&gt; 8) + '.' + str((0x000000ff &amp; mask)))
  
  def ip_to_numeric(ip):
      ip_num = 0
      for i, octet in enumerate(ip.split('.')):
          ip_num += int(octet) &lt;&lt; (24 - (8 * i))
  
      return ip_num
  
  def is_in_ip_network(ip, network_addr):
      if len(network_addr.split('/')) &lt; 2:
          return ip == network_addr.split('/')[0]
      else:
          network_ip, cidr = network_addr.split('/')
          subnet = cidr_to_netmask(cidr)
          return (ip_to_numeric(ip) &amp; ip_to_numeric(subnet)) == (ip_to_numeric(network_ip) &amp; ip_to_numeric(subnet))
  </code></pre>