title: What is etcd looking for in 127.0.0.1:4001?
body: <p>I'm trying to set up a test cluster using etcd 2.3.7 installed from CentOS RPM on CentOS 7.1.  On the <strong>Loader 1</strong> I executed:</p>
  
  <pre><code>etcdctl member add loader2 http://10.11.51.231:2380
  </code></pre>
  
  <p>And received response which confirmed the operation completed successfully.</p>
  
  <p>Similarly:</p>
  
  <pre><code>etcdctl member add loader3 http://10.11.51.231:2380
  </code></pre>
  
  <p>with all default settings, and here's what I see:</p>
  
  <h1>Loader 1 10.11.51.166</h1>
  
  <pre><code>    systemctl status etcd -ln1
      etcd.service - Etcd Server
         Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled)
         Active: active (running) since Sun 2017-02-19 14:33:18 IST; 28min ago
       Main PID: 19009 (etcd)
         CGroup: /system.slice/etcd.service
                 └─19009 /usr/bin/etcd --name=default --data-dir=/var/lib/etcd/default.etcd --listen-client-urls=http://localhost:2379
  
      Feb 19 15:02:03 loader3 etcd[19009]: cannot get the version of member a4803061db803edc (Get http://10.11.51.166:2380/version: dial tcp 10.11.51.166:2380: getsockopt: connection refused)
  </code></pre>
  
  <p>Tried to see cluster health:</p>
  
  <pre><code>    etcdctl --debug cluster-health
      Cluster-Endpoints: http://127.0.0.1:4001, http://127.0.0.1:2379
      cURL Command: curl -X GET http://127.0.0.1:4001/v2/members
      cURL Command: curl -X GET http://127.0.0.1:2379/v2/members
      member ce2a822cea30bfca is unhealthy: got unhealthy result from http://localhost:2379
      member da05b63349d818dc is unreachable: no available published client urls
      cluster is unhealthy
  </code></pre>
  
  <p>Note how this ignores the two nodes added previously, but sends requests to random port on localhost...</p>
  
  <h1>Loader 2 10.11.51.174</h1>
  
  <p>At first this machine started OK, but after I saw there was something wrong with <strong>Loader 1</strong>, I tried adding <strong>Loader 1</strong> as a member from this machine, and now I see the same picture on this machine too. I.e. it tries to query this 4001 port, where nobody responds.  On all machines:</p>
  
  <pre><code>    netstat -tupln | grep etcd
      tcp        0      0 127.0.0.1:7001          0.0.0.0:*               LISTEN      4507/etcd           
      tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      4507/etcd           
      tcp        0      0 127.0.0.1:2380          0.0.0.0:*               LISTEN      4507/etcd           
  </code></pre>
  
  <p>Nobody listens on 4001...</p>
  
  <h1>Loader 3 10.11.51.231</h1>
  
  <p>On this loader I didn't try to add new members. So it looks like this:</p>
  
  <pre><code>    etcdctl --debug cluster-health
      Cluster-Endpoints: http://127.0.0.1:4001, http://127.0.0.1:2379
      cURL Command: curl -X GET http://127.0.0.1:4001/v2/members
      cURL Command: curl -X GET http://127.0.0.1:2379/v2/members
      member ce2a822cea30bfca is healthy: got healthy result from http://localhost:2379
      cluster is healthy
  </code></pre>
  
  <p>In other words it still sends requests to random port, but this time it isn't bothered by the fact that nobody replied...</p>
  
  <hr>
  
  <p>Below is the contents of the configuration files:</p>
  
  <pre><code>cat /usr/lib/systemd/system/etcd.service
  [Unit]
  Description=Etcd Server
  After=network.target
  After=network-online.target
  Wants=network-online.target
  
  [Service]
  Type=notify
  WorkingDirectory=/var/lib/etcd/
  EnvironmentFile=-/etc/etcd/etcd.conf
  User=etcd
  # set GOMAXPROCS to number of processors
  ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd --name=\"${ETCD_NAME}\" --data-dir=\"${ETCD_DATA_DIR}\" --listen-client-urls=\"${ETCD_LISTEN_CLIENT_URLS}\""
  Restart=on-failure
  LimitNOFILE=65536
  
  [Install]
  WantedBy=multi-user.target
  </code></pre>
  
  <p>And:</p>
  
  <pre><code>cat /etc/etcd/etcd.conf
  # [member]
  ETCD_NAME=default
  ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
  #ETCD_WAL_DIR=""
  #ETCD_SNAPSHOT_COUNT="10000"
  #ETCD_HEARTBEAT_INTERVAL="100"
  #ETCD_ELECTION_TIMEOUT="1000"
  #ETCD_LISTEN_PEER_URLS="http://localhost:2380"
  ETCD_LISTEN_CLIENT_URLS="http://localhost:2379"
  #ETCD_MAX_SNAPSHOTS="5"
  #ETCD_MAX_WALS="5"
  #ETCD_CORS=""
  #
  #[cluster]
  #ETCD_INITIAL_ADVERTISE_PEER_URLS="http://localhost:2380"
  # if you use different ETCD_NAME (e.g. test), set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."
  #ETCD_INITIAL_CLUSTER="default=http://localhost:2380"
  #ETCD_INITIAL_CLUSTER_STATE="new"
  #ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
  ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379"
  #ETCD_DISCOVERY=""
  #ETCD_DISCOVERY_SRV=""
  #ETCD_DISCOVERY_FALLBACK="proxy"
  #ETCD_DISCOVERY_PROXY=""
  #ETCD_STRICT_RECONFIG_CHECK="false"
  #
  #[proxy]
  #ETCD_PROXY="off"
  #ETCD_PROXY_FAILURE_WAIT="5000"
  #ETCD_PROXY_REFRESH_INTERVAL="30000"
  #ETCD_PROXY_DIAL_TIMEOUT="1000"
  #ETCD_PROXY_WRITE_TIMEOUT="5000"
  #ETCD_PROXY_READ_TIMEOUT="0"
  #
  #[security]
  #ETCD_CERT_FILE=""
  #ETCD_KEY_FILE=""
  #ETCD_CLIENT_CERT_AUTH="false"
  #ETCD_TRUSTED_CA_FILE=""
  #ETCD_PEER_CERT_FILE=""
  #ETCD_PEER_KEY_FILE=""
  #ETCD_PEER_CLIENT_CERT_AUTH="false"
  #ETCD_PEER_TRUSTED_CA_FILE=""
  #
  #[logging]
  #ETCD_DEBUG="false"
  # examples for -log-package-levels etcdserver=WARNING,security=DEBUG
  #ETCD_LOG_PACKAGE_LEVELS=""
  #
  #[profiling]
  #ETCD_ENABLE_PPROF="false"
  </code></pre>
  
  <p>So... what is going on? The error messages given by <code>etcd</code> are the typical mindless nonsense produced by Go built-ins. The HTTP server that <code>etcd</code> uses is again, the Go built-in junk, that produces non-standard and absolutely worthless replies. So I cannot understand what was (if at all) misconfigured / missing.</p>