title: SSH slow log in on Centos 6
body: <p>We have a cluster of servers, some Centos 5.9 and some 6.5</p>
  
  <p>If I ssh from an 5.9 machine to a 5.9 the connection is quick</p>
  
  <pre><code>[user@cn10procms00 ~]$ time ssh cn10proadbe00 "exit;"
  real    0m0.147s
  user    0m0.005s
  sys     0m0.004s
  </code></pre>
  
  <p>If I ssh from a 5.9 machine to a 6.5 the connection is quick</p>
  
  <pre><code>[user@cn10procms00 ~]$ time ssh cn10prowidx01 "exit;"
  real    0m0.113s
  user    0m0.005s
  sys     0m0.003s
  </code></pre>
  
  <p>However if I ssh from a 6.5 machine to a 6.5 machine the connection is slow (the same is true of ssh connections from 6.5 to 5.9 machine)</p>
  
  <pre><code>[user@cn10prowidx00 ~]$ time ssh cn10prowidx01 "exit;"
  real    0m1.112s
  user    0m0.008s
  sys     0m0.004s
  </code></pre>
  
  <p>The same is show with the client running in verbose mode</p>
  
  <pre><code>[user@cn10prowidx00 ~]$ time ssh -vv cn10prowidx01 "exit;"
  OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
  debug1: Reading configuration data /etc/ssh/ssh_config
  debug1: Applying options for *
  debug2: ssh_connect: needpriv 0
  debug1: Connecting to cn10prowidx01 [xx.xx.xx.xx] port 22.
  debug1: Connection established.
  debug1: identity file /home/steve.cherry/.ssh/identity type -1
  debug1: identity file /home/steve.cherry/.ssh/identity-cert type -1
  debug1: identity file /home/steve.cherry/.ssh/id_rsa type -1
  debug1: identity file /home/steve.cherry/.ssh/id_rsa-cert type -1
  debug1: identity file /home/steve.cherry/.ssh/id_dsa type -1
  debug1: identity file /home/steve.cherry/.ssh/id_dsa-cert type -1
  debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
  debug1: match: OpenSSH_5.3 pat OpenSSH*
  debug1: Enabling compatibility mode for protocol 2.0
  debug1: Local version string SSH-2.0-OpenSSH_5.3
  debug2: fd 3 setting O_NONBLOCK
  debug1: SSH2_MSG_KEXINIT sent
  debug1: SSH2_MSG_KEXINIT received
  debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
  debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss
  debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
  debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
  debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
  debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
  debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
  debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
  debug2: kex_parse_kexinit: 
  debug2: kex_parse_kexinit: 
  debug2: kex_parse_kexinit: first_kex_follows 0 
  debug2: kex_parse_kexinit: reserved 0 
  debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
  debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
  debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
  debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
  debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
  debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
  debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
  debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
  debug2: kex_parse_kexinit: 
  debug2: kex_parse_kexinit: 
  debug2: kex_parse_kexinit: first_kex_follows 0 
  debug2: kex_parse_kexinit: reserved 0 
  debug2: mac_setup: found hmac-md5
  debug1: kex: server-&gt;client aes128-ctr hmac-md5 none
  debug2: mac_setup: found hmac-md5
  debug1: kex: client-&gt;server aes128-ctr hmac-md5 none
  debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024&lt;1024&lt;8192) sent
  debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
  debug2: dh_gen_key: priv key bits set: 144/256
  debug2: bits set: 510/1024
  debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
  debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
  debug1: Host 'cn10prowidx01' is known and matches the RSA host key.
  debug1: Found key in /home/steve.cherry/.ssh/known_hosts:1
  debug2: bits set: 506/1024
  debug1: ssh_rsa_verify: signature correct
  debug2: kex_derive_keys
  debug2: set_newkeys: mode 1
  debug1: SSH2_MSG_NEWKEYS sent
  debug1: expecting SSH2_MSG_NEWKEYS
  debug2: set_newkeys: mode 0
  debug1: SSH2_MSG_NEWKEYS received
  debug1: SSH2_MSG_SERVICE_REQUEST sent
  debug2: service_accept: ssh-userauth
  debug1: SSH2_MSG_SERVICE_ACCEPT received
  debug2: key: /home/steve.cherry/id_rsa (0x7fc093834d90)
  debug2: key: /home/steve.cherry/.ssh/identity ((nil))
  debug2: key: /home/steve.cherry/.ssh/id_rsa ((nil))
  debug2: key: /home/steve.cherry/.ssh/id_dsa ((nil))
  debug1: Authentications that can continue: publickey
  debug1: Next authentication method: publickey
  debug1: Offering public key: /home/steve.cherry/id_rsa
  debug2: we sent a publickey packet, wait for reply
  debug1: Server accepts key: pkalg ssh-rsa blen 279
  debug2: input_userauth_pk_ok: SHA1 fp f1:0a:91:09:e0:61:88:a0:06:5f:b4:ee:12:b9:7d:03:73:c9:42:7d
  debug1: Authentication succeeded (publickey).
  debug1: channel 0: new [client-session]
  debug2: channel 0: send open
  debug1: Requesting no-more-sessions@openssh.com
  debug1: Entering interactive session.
  debug2: callback start
  debug2: client_session2_setup: id 0
  debug1: Sending environment.
  debug1: Sending env LANG = en_US.UTF-8
  debug2: channel 0: request env confirm 0
  debug1: Sending command: exit;
  debug2: channel 0: request exec confirm 1
  debug2: fd 3 setting TCP_NODELAY
  debug2: callback done
  debug2: channel 0: open confirm rwindow 0 rmax 32768
  debug2: channel 0: rcvd adjust 2097152
  debug2: channel_input_status_confirm: type 99 id 0
  debug2: exec request accepted on channel 0
  debug2: channel 0: rcvd eof
  debug2: channel 0: output open -&gt; drain
  debug2: channel 0: obuf empty
  debug2: channel 0: close_write
  debug2: channel 0: output drain -&gt; closed
  debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
  debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
  debug2: channel 0: rcvd eow
  debug2: channel 0: close_read
  debug2: channel 0: input open -&gt; closed
  debug2: channel 0: rcvd close
  debug2: channel 0: almost dead
  debug2: channel 0: gc: notify user
  debug2: channel 0: gc: user detached
  debug2: channel 0: send close
  debug2: channel 0: is dead
  debug2: channel 0: garbage collecting
  debug1: channel 0: free: client-session, nchannels 1
  Transferred: sent 2472, received 2312 bytes, in 0.0 seconds
  Bytes per second: sent 61830.6, received 57828.7
  debug1: Exit status 0
  
  real    0m1.120s
  user    0m0.008s
  sys     0m0.005s
  </code></pre>
  
  <p>The ssh client config /etc/ssh/ssh_config is shown below</p>
  
  <pre><code>#       $OpenBSD: ssh_config,v 1.25 2009/02/17 01:28:32 djm Exp $
  
  # This is the ssh client system-wide configuration file.  See
  # ssh_config(5) for more information.  This file provides defaults for
  # users, and the values can be changed in per-user configuration files
  # or on the command line.
  
  # Configuration data is parsed as follows:
  #  1. command line options
  #  2. user-specific file
  #  3. system-wide file
  # Any configuration value is only changed the first time it is set.
  # Thus, host-specific definitions should be at the beginning of the
  # configuration file, and defaults at the end.
  
  # Site-wide defaults for some commonly used options.  For a comprehensive
  # list of available options, their meanings and defaults, please see the
  # ssh_config(5) man page.
  
  # Host *
  #   ForwardAgent no
  #   ForwardX11 no
  #   RhostsRSAAuthentication no
  #   RSAAuthentication yes
  #   PasswordAuthentication yes
  #   HostbasedAuthentication no
  #   GSSAPIAuthentication no
  #   GSSAPIDelegateCredentials no
  #   GSSAPIKeyExchange no
  #   GSSAPITrustDNS no
  #   BatchMode no
  #   CheckHostIP yes
  #   AddressFamily any
  #   ConnectTimeout 0
  #   StrictHostKeyChecking ask
  #   IdentityFile ~/.ssh/identity
  #   IdentityFile ~/.ssh/id_rsa
  #   IdentityFile ~/.ssh/id_dsa
  #   Port 22
  #   Protocol 2,1
  #   Cipher 3des
  #   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc
  #   MACs hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160
  #   EscapeChar ~
  #   Tunnel no
  #   TunnelDevice any:any
  #   PermitLocalCommand no
  #   VisualHostKey no
  Host *
          GSSAPIAuthentication yes
  # If this option is set to yes then remote X11 clients will have full access
  # to the original X11 display. As virtually no X11 client supports the untrusted
  # mode correctly we set this to yes.
          ForwardX11Trusted yes
  # Send locale-related environment variables
          SendEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
          SendEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
          SendEnv LC_IDENTIFICATION LC_ALL LANGUAGE
  </code></pre>
  
  <p>Has anyone seen this before, can anyone help?</p>
  
  <p>Thanks
  Steve</p>