title: Logstash Elasticsearch output gives 401 error
body: <p>I'm trying to deploy and ELK stack on AKS that will take messages from RabbitMQ and ultimately end up in Kibana. To do this I'm using the Elastic operator via</p>
  <pre><code>kubectl apply -f https://download.elastic.co/downloads/eck/1.3.0/all-in-one.yaml
  </code></pre>
  <p>Everything is working except the connection between Logstash and Elasticsearch. I can log in to Kibana, I can get the default Elasticsearch message in the browser, all the logs look fine so I think the issue lies in the logstash configuration. My configuration is at the end of the question, you can see I'm using secrets to get the various passwords and access the public certificates to make the https work.</p>
  <p>Most confusingly, I can bash into the running logstash pod and with the exact same certificate  run</p>
  <pre><code>curl --cacert /etc/logstash/certificates/tls.crt -u elastic:&lt;redacted-password&gt; https://rabt-db-es-http:9200
  </code></pre>
  <p>This gives me the response:</p>
  <pre><code>{
    &quot;name&quot; : &quot;rabt-db-es-default-0&quot;,
    &quot;cluster_name&quot; : &quot;rabt-db&quot;,
    &quot;cluster_uuid&quot; : &quot;9YoWLsnMTwq5Yor1ak2JGw&quot;,
    &quot;version&quot; : {
      &quot;number&quot; : &quot;7.10.0&quot;,
      &quot;build_flavor&quot; : &quot;default&quot;,
      &quot;build_type&quot; : &quot;docker&quot;,
      &quot;build_hash&quot; : &quot;51e9d6f22758d0374a0f3f5c6e8f3a7997850f96&quot;,
      &quot;build_date&quot; : &quot;2020-11-09T21:30:33.964949Z&quot;,
      &quot;build_snapshot&quot; : false,
      &quot;lucene_version&quot; : &quot;8.7.0&quot;,
      &quot;minimum_wire_compatibility_version&quot; : &quot;6.8.0&quot;,
      &quot;minimum_index_compatibility_version&quot; : &quot;6.0.0-beta1&quot;
    },
    &quot;tagline&quot; : &quot;You Know, for Search&quot;
  }
  </code></pre>
  <p>To me, this verifies that I the pod can communicate with the database, and has the correct user, password and certificates in place to do it securely. Why then does this fail using a logstash conf file?</p>
  <p>The error from the logstash end is</p>
  <pre><code>[WARN ] 2021-01-14 15:24:38.360 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.6.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:241] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=&gt;&quot;https://rabt-db-es-http:9200/&quot;, :error_type=&gt;LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=&gt;&quot;Got response code '401' contacting Elasticsearch at URL 'https://rabt-db-es-http:9200/'&quot;}
  </code></pre>
  <p>From Elasticsearch I can see the failed requests as</p>
  <pre><code>{&quot;type&quot;: &quot;server&quot;, &quot;timestamp&quot;: &quot;2021-01-14T15:36:13,725Z&quot;, &quot;level&quot;: &quot;WARN&quot;, &quot;component&quot;: &quot;o.e.x.s.t.n.SecurityNetty4HttpServerTransport&quot;, &quot;cluster.name&quot;: &quot;rabt-db&quot;, &quot;node.name&quot;: &quot;rabt-db-es-default-0&quot;, &quot;message&quot;: &quot;http client did not trust this server's certificate, closing connection Netty4HttpChannel{localAddress=/10.244.0.30:9200, remoteAddress=/10.244.0.15:37766}&quot;, &quot;cluster.uuid&quot;: &quot;9YoWLsnMTwq5Yor1ak2JGw&quot;, &quot;node.id&quot;: &quot;9w3fXZBZQGeBMeFYGqYUXw&quot;  }
  </code></pre>
  <pre><code>---
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: logstash-config
    labels:
      app.kubernetes.io/name: rabt
      app.kubernetes.io/component: logstash
  data:
    logstash.yml: |
      http.host: &quot;0.0.0.0&quot;
      path.config: /usr/share/logstash/pipeline
  ---
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: logstash-pipeline
    labels:
      app.kubernetes.io/name: rabt
      app.kubernetes.io/component: logstash
  data:
    logstash.conf: |
      input {
  
        rabbitmq {
          host =&gt; &quot;rabt-mq&quot;
          port =&gt; 5672
          durable =&gt; true
          queue =&gt; &quot;rabt-rainfall-queue&quot;
          exchange =&gt; &quot;rabt-rainfall-exchange&quot;
          exchange_type =&gt; &quot;direct&quot;
          heartbeat =&gt; 30
          durable =&gt; true
          user =&gt; &quot;${RMQ_USERNAME}&quot;
          password =&gt; &quot;${RMQ_PASSWORD}&quot;
        }
  
        file {
          path =&gt; &quot;/usr/share/logstash/config/intensity.csv&quot;
          start_position =&gt; &quot;beginning&quot;
          codec =&gt; plain {
              charset =&gt; &quot;ISO-8859-1&quot;
          }
          type =&gt; &quot;intensity&quot;
        }
      }
  
      filter {
        csv {
          separator =&gt; &quot;,&quot;
          columns =&gt; [&quot;Duration&quot;, &quot;Intensity&quot;]
        }
      }
  
      output {
        if [type] == &quot;rainfall&quot; {
          elasticsearch {
            hosts =&gt; [ &quot;${ES_HOSTS}&quot; ]
            ssl =&gt; true
            cacert =&gt; &quot;/etc/logstash/certificates/tls.crt&quot;
            index =&gt; &quot;rabt-rainfall-%{+YYYY.MM.dd}&quot;
          }   
        }
        else if[type] == &quot;intensity&quot;{
          elasticsearch {
            hosts =&gt; [ &quot;${ES_HOSTS}&quot; ]
            ssl =&gt; true
            cacert =&gt; &quot;/etc/logstash/certificates/tls.crt&quot;
            index =&gt; &quot;intensity-%{+YYYY.MM.dd}&quot;
          }   
        }
      }
  ---
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: rainfall-intensity-threshold
    labels:
      app.kubernetes.io/name: rabt
      app.kubernetes.io/component: logstash
  data:
    intensity.csv: |
      Duration,Intensity
      0.1,7.18941593
      0.2,6.34611898
      0.3,5.89945352
      0.4,5.60173824
      0.5,5.38119846
      0.6,5.20746530
      0.7,5.06495933
      0.8,4.94467113
      0.9,4.84094288
      1,4.75000000
      2,4.19283923
      3,3.89773029
      4,3.70103175
      5,3.55532256
      6,3.44053820
      7,3.34638544
      8,3.26691182
      9,3.19837924
      10,3.13829388
      20,2.77018141
      30,2.57520486
      40,2.44524743
      50,2.34897832
      60,2.27314105
      70,2.21093494
      80,2.15842723
      90,2.11314821
      100,2.07345020 
  ---
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: rabt-logstash
    labels:
      app.kubernetes.io/name: rabt
      app.kubernetes.io/component: logstash
  spec:
    replicas: 1
    selector:
      matchLabels:
        app.kubernetes.io/name: rabt
        app.kubernetes.io/component: logstash
    template:
      metadata:
        labels:
          app.kubernetes.io/name: rabt
          app.kubernetes.io/component: logstash
      spec:
        containers:
          - name: logstash
            image: docker.elastic.co/logstash/logstash:7.9.2
            ports:
              - name: &quot;tcp-beats&quot;
                containerPort: 5044
            env:
              - name: ES_HOSTS
                value: &quot;https://rabt-db-es-http:9200&quot;
              - name: ES_USER
                value: &quot;elastic&quot;
              - name: ES_PASSWORD
                valueFrom:
                  secretKeyRef:
                    name: rabt-db-es-elastic-user
                    key: elastic
              - name: RMQ_USERNAME
                valueFrom:
                  secretKeyRef:
                    name: rabt-mq-default-user
                    key: username
              - name: RMQ_PASSWORD
                valueFrom:
                  secretKeyRef:
                    name: rabt-mq-default-user
                    key: password
            volumeMounts:
              - name: config-volume
                mountPath: /usr/share/logstash/config
              - name: pipeline-volume
                mountPath: /usr/share/logstash/pipeline
              - name: ca-certs
                mountPath: /etc/logstash/certificates
                readOnly: true
        volumes:
          - name: config-volume
            projected:
              sources:
              - configMap:
                  name: logstash-config
              - configMap:
                  name: rainfall-intensity-threshold
          - name: pipeline-volume
            configMap:
              name: logstash-pipeline
          - name: ca-certs
            secret:
              secretName: rabt-db-es-http-certs-public
  ---
  apiVersion: v1
  kind: Service
  metadata:
    name: rabt-logstash
    labels:
      app.kubernetes.io/name: rabt
      app.kubernetes.io/component: logstash
  spec:
    ports:
      - name: &quot;tcp-beats&quot;
        port: 5044
        targetPort: 5044
    selector:
      app.kubernetes.io/name: rabt
      app.kubernetes.io/component: logstash
  </code></pre>