title: IPSec on Linux with strongSwan: received netlink error: No such file or directory (2)
body: <p>I have set up an IPSec system on two hosts with the following basic information:
  - Linux kernel: 4.4.135
  - IKE: strongSwan 5.6.1</p>
  
  <p>LAN1 --->| WAN1: 192.168.100.121 | &lt;---->| WAN2: 192.168.100.122 | &lt;---- LAN2</p>
  
  <p>I am testing a new block  algorithm named BC512 with a block size of 128 bits, a key size of 512 bits (the key is larger than the usual algorithms like AES or Camellia, but this is a test algorithm I need to run the test).</p>
  
  <p>The general configuration of the /etc/ipsec.conf file is as follows:</p>
  
  <pre><code>    # ipsec.conf - Configuration file for IPSec
  
      config setup
              cachecrls=no
              strictcrlpolicy=no
              uniqueids=no
              charondebug="cfg 4, dmn 4, ike 4, net 4, lib 4, knl 4"
  
      conn %default
              leftfirewall=yes
              type=tunnel
              authby=pubkey
              auto=start
              closeaction=clear
              compress=yes
              dpdaction=clear
              dpddelay=60s
              dpdtimeout=300s
              inactivity=300s
              installpolicy=yes
              keyexchange=ikev2
              keyingtries=5
              lifetime=8h
              leftauth=ike:rsa/pss-sha256
              rightauth=ike:rsa/pss-sha256
  
      conn VM-0121_VM-0122
              left=192.168.100.221
              leftsubnet=172.16.121.0/24
              leftcert=VM.2019.0121_IPSEC_RSA_2048.PEM.CRT
              leftid="C=VN, ST=Ha Noi, O=ABC.NET, OU=Dev Team, CN=ipsec121, E=ipsec121@abc.net"
              leftfirewall=yes
              right=192.168.100.222
              rightsubnet=172.16.122.0/24
              rightcert=VM.2019.0122_IPSEC_RSA_2048.PEM.CRT
              rightid="C=VN, ST=Ha Noi, O=ABC.NET, OU=Dev Team, CN=ipsec122, E=ipsec122@abc.net"
              rightfirewall=yes
              esp=aes256ctr-sha512-modp2048!
              ike=aes256ctr-sha512-modp2048!
              type=tunnel
              authby=pubkey
  </code></pre>
  
  <p>When I configure strongSwan in the /etc/ipsec.conf file using ESP/IKE parameters, the following results are given:</p>
  
  <pre><code>    1)  esp=aes256ctr-sha1-modp2048!
          ike=aes256ctr-sha512-modp2048!
  
      Result: OK
  
      2)  esp=aes256ctr-sha256-modp2048!
          ike=aes256ctr-sha512-modp2048!
  
      Result: OK  
  
      3)  esp=aes256ctr-sha512-modp2048!
          ike=aes256ctr-sha512-modp2048!
  
      Result: OK
  
      4)  esp=aes256ctr-sha1-modp2048!
          ike=bc512ctr-sha512-modp2048!
  
      Result: OK
  
      5)  esp=aes256ctr-sha256-modp2048!
          ike=bc512ctr-sha512-modp2048!
  
      Result: OK  
  
      6)  esp=aes256ctr-sha512-modp2048!
          ike=bc512ctr-sha512-modp2048!
  
      Result: OK
  
      7)  esp=bc512ctr-sha1-modp2048!
          ike=bc512ctr-sha512-modp2048!
  
      Result: OK
  
      8)  esp=bc512ctr-sha256-modp2048!
          ike=bc512ctr-sha512-modp2048!
  
      Result: Error (as described later)
  
      9)  esp=bc512ctr-sha512-modp2048!
          ike=bc512ctr-sha512-modp2048!
  
      Result: Error (as described later)
  </code></pre>
  
  <p>Error while running ipsec start --nofork:
  On Host1:</p>
  
  <pre><code>    ...
      ...
      ...
      11[IKE] IKE_SA VM-0121_VM-0122[1] state change: CONNECTING =&gt; ESTABLISHED
      03[NET] waiting for data on sockets
      11[IKE] scheduling reauthentication in 10005s
      11[IKE] maximum IKE_SA lifetime 10545s
      11[IKE] received NO_PROPOSAL_CHOSEN notify, no CHILD_SA built
      11[CFG] configured proposals: ESP:BC512_CTR_512/HMAC_SHA2_512_256/MODP_2048/NO_EXT_SEQ
      11[IKE] failed to establish CHILD_SA, keeping IKE_SA
      11[KNL] deleting SAD entry with SPI 00003ac0
      ...
      ...
      ...
  </code></pre>
  
  <p>On Host2:</p>
  
  <pre><code>    ...
      ...
      ...
      06[KNL] adding SAD entry with SPI cccad04c and reqid {1}
      06[KNL]   using encryption algorithm BC512_CTR with key size 544
      06[KNL]   using integrity algorithm HMAC_SHA2_512_256 with key size 512
      06[KNL]   using replay window of 0 packets
      06[KNL] sending XFRM_MSG_NEWSA 212: =&gt; 516 bytes @ 0x7f3e1248b4c0
      ...
      ...
      ...
      06[KNL] received netlink error: No such file or directory (2)
      06[KNL] unable to add SAD entry with SPI cccad04c (FAILED)
      06[IKE] unable to install inbound and outbound IPsec SA (SAD) in kernel
      06[IKE] failed to establish CHILD_SA, keeping IKE_SA
      06[KNL] deleting policy 172.16.121.0/24 === 172.16.122.0/24 in
      06[KNL] deleting policy 172.16.121.0/24 === 172.16.122.0/24 in failed, not found
      06[KNL] deleting policy 172.16.121.0/24 === 172.16.122.0/24 fwd
      06[KNL] deleting policy 172.16.121.0/24 === 172.16.122.0/24 fwd failed, not found
      06[KNL] deleting SAD entry with SPI 00008d4e
      ...
      ...
      ...
  </code></pre>
  
  <p>I think that integrating the BC512 algorithm source code into strongSwan is not a problem because IKE can be configured without any error.</p>
  
  <p>It is possible that an error occurs due to the integration of the BC512 algorithm into the Linux kernel. Could the reason be that the BC512 algorithm has a very large key length (512 bits)?</p>
  
  <p>Some other information:
  1. Run the command cat /proc/crypto</p>
  
  <pre><code>    cat /proc/crypto | grep aes
      name         : __ctr-aes-aesni
      driver       : cryptd(__driver-ctr-aes-aesni)
      name         : seqiv(authenc(hmac(sha512),rfc3686(ctr(aes))))
      driver       : seqiv(authenc(hmac(sha512-generic),rfc3686(ctr-aes-aesni)))
      name         : authenc(hmac(sha512),rfc3686(ctr(aes)))
      driver       : authenc(hmac(sha512-generic),rfc3686(ctr-aes-aesni))
      name         : rfc3686(ctr(aes))
      driver       : rfc3686(ctr-aes-aesni)
      name         : rfc3686(ctr(aes))
      driver       : rfc3686(ctr-aes-aesni)
      name         : ctr(aes)
      driver       : ctr-aes-aesni
      ...
      ...
      ...
  
      name         : aes
      driver       : aes-aesni
      module       : aesni_intel
      name         : aes
      driver       : aes-asm
      module       : aes_x86_64
      name         : aes
      driver       : aes-generic
  </code></pre>
  
  <pre><code>    cat /proc/crypto | grep bc512
      name         : seqiv(authenc(hmac(sha1),rfc3686(ctr(bc512))))
      driver       : seqiv(authenc(hmac(sha1-generic),rfc3686(ctr(bc512-generic))))
      name         : authenc(hmac(sha1),rfc3686(ctr(bc512)))
      driver       : authenc(hmac(sha1-generic),rfc3686(ctr(bc512-generic)))
      name         : authenc(hmac(sha256),rfc3686(ctr(bc512)))
      driver       : authenc(hmac(sha256-generic),rfc3686(ctr(bc512-generic)))
      name         : authenc(hmac(sha512),rfc3686(ctr(bc512)))
      driver       : authenc(hmac(sha512-generic),rfc3686(ctr(bc512-generic)))
      name         : rfc3686(ctr(bc512))
      driver       : rfc3686(ctr(bc512-generic))
      name         : rfc3686(ctr(bc512))
      driver       : rfc3686(ctr(bc512-generic))
      name         : ctr(bc512)
      driver       : ctr(bc512-generic)
      name         : ctr(bc512)
      driver       : ctr(bc512-generic)
      name         : bc512
      driver       : bc512-generic
      module       : bc512
  </code></pre>
  
  <p>As you can see, the kernel cannot initialize conversions like:</p>
  
  <pre><code>    name         : seqiv(authenc(hmac(sha256),rfc3686(ctr(bc512))))
      driver       : seqiv(authenc(hmac(sha256-generic),rfc3686(ctr(bc512-generic))))
      or
      name         : seqiv(authenc(hmac(sha512),rfc3686(ctr(bc512))))
      driver       : seqiv(authenc(hmac(sha512-generic),rfc3686(ctr(bc512-generic))))
  </code></pre>
  
  <ol start="2">
  <li>Debugging information from the Linux kernel:</li>
  </ol>
  
  <pre><code>[ 4158.398344] ------------[ cut here ]------------
  [ 4158.398346] WARNING: CPU: 0 PID: 18942 at lib/iomap.c:43 bad_io_access+0x38/0x40()
  [ 4158.398347] Bad IO access at port 0x3c (outl(val,port))
  
  [ 4158.398509] Call Trace:
  [ 4158.398511]  [&lt;ffffffff813c7e1c&gt;] dump_stack+0x63/0x87
  [ 4158.398513]  [&lt;ffffffff810802e6&gt;] warn_slowpath_common+0x86/0xc0
  [ 4158.398515]  [&lt;ffffffff8108036c&gt;] warn_slowpath_fmt+0x4c/0x50
  [ 4158.398517]  [&lt;ffffffff8138246d&gt;] ? shash_default_export+0x1d/0x30
  [ 4158.398518]  [&lt;ffffffff81384450&gt;] ? hmac_setkey+0x1a0/0x1b0
  [ 4158.398520]  [&lt;ffffffff813e0b88&gt;] bad_io_access+0x38/0x40
  [ 4158.398522]  [&lt;ffffffff813e0cff&gt;] iowrite32+0x2f/0x40
  [ 4158.398524]  [&lt;ffffffffc046a035&gt;] write_key_to_hardware+0x35/0x50 [bc512]
  [ 4158.398526]  [&lt;ffffffffc046a06b&gt;] bc512_set_key_32bit+0x1b/0x20 [bc512]
  [ 4158.398527]  [&lt;ffffffff8137bf2d&gt;] setkey+0x5d/0x110
  [ 4158.398529]  [&lt;ffffffffc0465034&gt;] crypto_ctr_setkey+0x34/0x50 [ctr]
  [ 4158.398531]  [&lt;ffffffff8137ff3d&gt;] setkey+0x4d/0x100
  [ 4158.398532]  [&lt;ffffffff8137fffe&gt;] async_setkey+0xe/0x10
  [ 4158.398534]  [&lt;ffffffffc04651ca&gt;] crypto_rfc3686_setkey+0x4a/0x60 [ctr]
  [ 4158.398536]  [&lt;ffffffffc047936e&gt;] crypto_authenc_setkey+0x9e/0xe0 [authenc]
  [ 4158.398537]  [&lt;ffffffff8137e565&gt;] crypto_aead_setkey+0x35/0xd0
  [ 4158.398539]  [&lt;ffffffff8137bcde&gt;] ? crypto_create_tfm+0x4e/0xc0
  [ 4158.398541]  [&lt;ffffffff8137e612&gt;] aead_geniv_setkey+0x12/0x20
  [ 4158.398542]  [&lt;ffffffff8137e565&gt;] crypto_aead_setkey+0x35/0xd0
  [ 4158.398544]  [&lt;ffffffffc041e62e&gt;] esp_init_state+0x36e/0x450 [esp4]
  [ 4158.398546]  [&lt;ffffffff81788116&gt;] ? xfrm_get_mode+0x96/0xb0
  [ 4158.398548]  [&lt;ffffffff81788307&gt;] __xfrm_init_state+0x1d7/0x230
  [ 4158.398550]  [&lt;ffffffffc044d7ee&gt;] xfrm_add_sa+0x67e/0xaa0 [xfrm_user]
  [ 4158.398552]  [&lt;ffffffff813f3b63&gt;] ? nla_parse+0xa3/0x100
  [ 4158.398554]  [&lt;ffffffffc0449d29&gt;] xfrm_user_rcv_msg+0x1a9/0x1d0 [xfrm_user]
  [ 4158.398556]  [&lt;ffffffffc0449b80&gt;] ? xfrm_dump_sa_done+0x30/0x30 [xfrm_user]
  [ 4158.398558]  [&lt;ffffffff81720809&gt;] netlink_rcv_skb+0xa9/0xc0
  [ 4158.398560]  [&lt;ffffffffc0449805&gt;] xfrm_netlink_rcv+0x35/0x50 [xfrm_user]
  [ 4158.398562]  [&lt;ffffffff817201c3&gt;] netlink_unicast+0x163/0x230
  [ 4158.398563]  [&lt;ffffffff817205b2&gt;] netlink_sendmsg+0x322/0x3a0
  [ 4158.398565]  [&lt;ffffffff816d1ade&gt;] sock_sendmsg+0x3e/0x50
  [ 4158.398566]  [&lt;ffffffff816d1f82&gt;] SYSC_sendto+0x102/0x190
  [ 4158.398568]  [&lt;ffffffff817f0ab6&gt;] ? __schedule+0x2d6/0x820
  [ 4158.398570]  [&lt;ffffffff8121740d&gt;] ? SyS_select+0xbd/0xf0
  [ 4158.398572]  [&lt;ffffffff816d2ace&gt;] SyS_sendto+0xe/0x10
  [ 4158.398574]  [&lt;ffffffff817f48e5&gt;] entry_SYSCALL_64_fastpath+0x22/0x9d
  [ 4158.398575] ---[ end trace 01669c0f1d468642 ]---
  [ 4158.422826] ip_tables: (C) 2000-2006 Netfilter Core Team
  </code></pre>
  
  <p>There was an error after the procedure hmac_setkey.</p>
  
  <p>I debugged and searched weekly but could not fix the problem.</p>
  
  <p>I think the Crypto API of the Linux kernel has been limited to not be able to initialize the block cipher algorithm (BC512) running in CTR mode with the authentication code HMAC-SHA256 or HMAC-SHA512.</p>
  
  <p>In addition, when using other block cipher such as AES (key size 128, 192 and 256), Blowfish (key size 128), Camellia (key size 128, 192 and 256) can be operating in CTR mode and HMAC-SHA256 or HMAC-SHA512.</p>
  
  <p>When configuring BC512 in CBC, CFB, OFB, GCM, and CCM modes; HMAC-SHA1, HMAC-SHA256 and HMAC-SHA512 can be used.</p>
  
  <p>Only BC512 with CTR mode cannot operate with HMAC-SHA256 and HMAC-SHA512.</p>
  
  <p>Please help me, thank you in advance.</p>