title: Getting "rpc error: code = Unavailable desc = error reading from server: EOF" when trying to create a new etcdv3 client
body: <p>I'm trying to access my ETCD database from a K8s controller, but getting rpc error/EOF when trying to open ETCD client.</p>
  <p>My setup:</p>
  <ul>
  <li>ETCD service is deployed in my K8s cluster and included in my Istio service mesh (its DNS record: <code>my-etcd-cluster.my-etcd-namespace.svc.cluster.local</code>)</li>
  <li>I have a custom K8s controller developed with use of Kubebuilder framework and deployed in the same cluster, different namespace, but configured to be a part of the same Istio service mesh</li>
  <li>I'm trying to connect to ETCD database from the controller, using Go client SDK library for ETCD</li>
  </ul>
  <p>Here's my affected Go code:</p>
  <pre class="lang-golang prettyprint-override"><code>cli, err := clientv3.New(clientv3.Config{
      Endpoints:   []string{&quot;http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379&quot;},
      DialTimeout: 5 * time.Second,
      Username:    username,
      Password:    password,
  })
  
  if err != nil {
      return nil, fmt.Errorf(&quot;opening ETCD client failed: %v&quot;, err)
  }
  </code></pre>
  <p>And here's an error I'm getting when <code>clientv3.New(...)</code> gets executed:</p>
  <pre><code>{&quot;level&quot;:&quot;warn&quot;,&quot;ts&quot;:&quot;2022-03-16T23:37:42.174Z&quot;,&quot;logger&quot;:&quot;etcd-client&quot;,&quot;caller&quot;:&quot;v3@v3.5.0/retry_interceptor.go:62&quot;,&quot;msg&quot;:&quot;retrying of unary invoker failed&quot;,
  &quot;target&quot;:&quot;etcd-endpoints://0xc00057f500/#initially=[http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379]&quot;,&quot;attempt&quot;:0,
  &quot;error&quot;:&quot;rpc error: code = Unavailable desc = error reading from server: EOF&quot;}
  ...
  1.647473862175209e+09   INFO    controller.etcdclient   Finish reconcile loop for some-service/test-svc-client  {&quot;reconciler group&quot;: &quot;my-controller.something.io&quot;, &quot;reconciler kind&quot;: &quot;ETCDClient&quot;, &quot;name&quot;: &quot;test-svc-client&quot;, &quot;namespace&quot;: &quot;some-service&quot;, &quot;reconcile-etcd-client&quot;: &quot;some-service/test-svc-client&quot;}
  1.6474738621752858e+09  ERROR   controller.etcdclient   Reconciler error        {&quot;reconciler group&quot;: &quot;my-controller.something.io&quot;, &quot;reconciler kind&quot;: &quot;ETCDClient&quot;, &quot;name&quot;: &quot;test-svc-client&quot;, &quot;namespace&quot;: &quot;some-service&quot;, &quot;error&quot;: &quot;opening ETCD client failed: rpc error: code = Unavailable desc = error reading from server: EOF&quot;}
  sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
          /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
  sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
          /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
  </code></pre>
  <p>The same error happens when I'm passing some dummy, invalid credentials.</p>
  <p>However, when I tried to access the database in a HTTP API manner:</p>
  <pre class="lang-golang prettyprint-override"><code>postBody, _ := json.Marshal(map[string]string{
      &quot;name&quot;:     username,
      &quot;password&quot;: password,
  })
  responseBody := bytes.NewBuffer(postBody)
  
  resp, err := http.Post(&quot;http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379/v3/auth/authenticate&quot;, &quot;application/json&quot;, responseBody)
  if err != nil {
      return ctrl.Result{}, fmt.Errorf(&quot;an error occured %w&quot;, err)
  }
  l.Info(fmt.Sprintf(&quot;code: %d&quot;, resp.StatusCode))
  defer resp.Body.Close()
  </code></pre>
  <p>...I got 200 OK and a proper token (which is expected), so I believe my Istio configuration is ok and my controller should be able to see the ETCD db service. I have no clue why this doesn't work when following the client SDK approach.</p>
  <p>When I'm using port-forwarding of the ETCD service and accessing it locally, <code>clientv3.New()</code> and other client SDK methods work like a charm. What am I missing? I'd really appreciate any suggestions.</p>
  <p>EDIT:
  I've also added a simple pod to try accessing my etcd db via etcdctl:</p>
  <pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
  kind: Pod
  metadata:
    name: test-pod
    namespace: my-controller-namespace
  spec:
    containers:
    - name: etcdctl
      image: bitnami/etcd
      command:
      - sleep
      - infinity
  </code></pre>
  <p>When logged into the container via <code>kubectl exec</code>, I was able to access my db:</p>
  <pre><code>$ etcdctl --endpoints=my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379 --user=&quot;user&quot; --password=&quot;password&quot; put foo bob
  OK
  </code></pre>
  <p>I guess the problem is somewhere in the SDK?</p>