title: Contacting loki from fluentbit output plugin
body: <p>I am collecting logs from a kubernetes cluster using fluentbit, having an output that connect to loki to send them there.</p>
  <p>This is my loki configuration at fluentbit configmap file</p>
  <p>Since loki is deployed at <code>loki</code> namespace, and fluentbit at <code>fluentbit</code> namespace I am using to contact loki: <code>host loki.loki.svc.cluster.local </code></p>
  <pre><code>apiVersion: v1
  data:
    custom_parsers.conf: |
      [PARSER]
          Name docker
          Format json
          Time_Key time
          Time_Format %Y-%m-%dT%H:%M:%S %z
    fluent-bit.conf: |
      [SERVICE]
          Daemon Off
          Flush 1
          Log_Level info
          Parsers_File parsers.conf
          Parsers_File custom_parsers.conf
          HTTP_Server On
          HTTP_Listen 0.0.0.0
          HTTP_Port 2020
          Health_Check On
      [INPUT]
          Name tail
          Path /var/log/containers/*.log
          multiline.parser docker, cri
          Tag kube.*
          Mem_Buf_Limit 100MB
          Skip_Long_Lines On
      [INPUT]
          Name systemd
          Tag host.*
          Systemd_Filter _SYSTEMD_UNIT=kubelet.service
          Read_From_Tail On
      [FILTER]
          Name kubernetes
          Match kube.*
          Merge_Log On
          Keep_Log Off
          K8S-Logging.Parser On
          K8S-Logging.Exclude On
      [OUTPUT]
          Name stdout
          Match kube.*
          Format json
          Json_date_key timestamp
          Json_date_format iso8601
      [OUTPUT]
          Name loki
          Match kube.*
          host loki.loki.svc.cluster.local 
          port 3100
          tenant_id &quot;&quot;
          Labels {job=&quot;fluent-bit&quot;}
          auto_kubernetes_labels false
          line_format json
  kind: ConfigMap
  metadata:
    annotations:
      meta.helm.sh/release-name: fluent-bit
      meta.helm.sh/release-namespace: fluent-bit
    creationTimestamp: &quot;2021-10-21T13:53:14Z&quot;
    labels:
      app.kubernetes.io/instance: fluent-bit
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: fluent-bit
      app.kubernetes.io/version: 1.8.8
      helm.sh/chart: fluent-bit-0.19.1
    name: fluent-bit
    namespace: fluent-bit 
  </code></pre>
  <p>But I got this error in my fluentbit logs.</p>
  <pre><code>[2021/10/21 14:59:59] [error] [output:loki:loki.1] loki.loki.svc.cluster.local:3100, HTTP status=400 Not retrying.
  1:2: parse error: unexpected left brace '{'
  </code></pre>
  <p>Looks like that is not the correct format, and sometimes I got this another message with the same configuration (weird):</p>
  <pre><code>[2021/10/21 14:59:59] [error] [output:loki:loki.1] loki.loki.svc.cluster.local:3100, HTTP status=400 Not retrying.
  1:2: parse error: unexpected left brace '{'
  </code></pre>
  <p>Like I would have to explicitly specify the POST endpoint on loki to push logs there, <a href="https://grafana.com/docs/loki/latest/api/#post-apiprompush" rel="nofollow noreferrer">this one</a> <code>/loki/api/v1/push</code></p>
  <p>But in general terms, I am getting the most the <code>400</code> bad syntax error.
  How can I contact loki from fluentbit configuration?</p>