title: Prometheus configuration and deployment in Terraform
body: <p>I have an IaaC project that use Terraform and Helm charts to deploy Prometheus (along with Grafana, Kubernetes and the platform app). I do have Kubernetes metrics on Prometheus. However, I realize that the configuration on deployed Prometheus doesn't really come from the Prometheus config file in the project. I am not sure if it's the problem with configMap or other configuration. Here are the files concerned in the project:</p>
  <p><code>/terraform/kubernetes/files/prometheus_config_map.yaml</code></p>
  <pre><code>global:
    scrape_interval: 15s
  scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
    - role: pod
    relabel_configs:
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
    - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
      action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      target_label: __address__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_pod_label_component]
      action: replace
      target_label: job
    - source_labels: [__meta_kubernetes_pod_name]
      action: replace
      target_label: kubernetes_pod_name
  </code></pre>
  <p><code>/terraform/kubernetes/prometheus.tf</code></p>
  <pre><code>resource &quot;kubernetes_namespace&quot; &quot;prometheus&quot; {
    metadata {
      name = &quot;prometheus&quot;
    }
  }
  
  resource &quot;kubernetes_config_map&quot; &quot;prometheus_config&quot; {
    metadata {
      name      = &quot;prometheus-config&quot;
      namespace = &quot;prometheus&quot;
    }
  
    data = {
      &quot;prometheus.yml&quot; = file(&quot;${path.module}/files/prometheus_config_map.yaml&quot;)
    }
    depends_on = [
      kubernetes_namespace.prometheus
    ]
  }
  
  # Values documentation: https://github.com/bitnami/charts/blob/master/bitnami/kube-prometheus/values.yaml
  resource &quot;helm_release&quot; &quot;prometheus&quot; {
    name        = &quot;prometheus&quot;
    repository  = local.helm_repositories.bitnami
    chart       = &quot;kube-prometheus&quot;
    version     = &quot;3.4.0&quot;
    namespace   = &quot;prometheus&quot;
    atomic      = true
    max_history = 5
  
    values = [
      file(&quot;${path.module}/helm_values/security.yaml.tpl&quot;),
      file(&quot;${path.module}/helm_values/prometheus.yaml&quot;)
    ]
  
    depends_on = [
      kubernetes_config_map.prometheus_config
    ]
  }
  </code></pre>
  <p><code>/terraform/kubernetes/helm_values/prometheus.yaml</code></p>
  <pre><code>prometheus:
    podMetadata:
      annotations:
        container.apparmor.security.beta.kubernetes.io/prometheus-operator: runtime/default
        seccomp.security.alpha.kubernetes.io/pod: runtime/default
  
  nodeAffinityPreset:
    ## Node affinity type
    ## Allowed values: soft, hard
    ##
    type: &quot;hard&quot;
    ## Node label key to match
    ## E.g.
    ## key: &quot;kubernetes.io/e2e-az-name&quot;
    ##
    key: &quot;cloud.google.com/gke-nodepool&quot;
    ## Node label values to match
    ## E.g.
    ## values:
    ##   - e2e-az1
    ##   - e2e-az2
    ##
    values: [
      &quot;project-primary-pool&quot;
    ]
  
  prometheus:  
    configMaps:
      - prometheus-config
  </code></pre>
  <p>In this file <code>/terraform/kubernetes/helm_values/prometheus.yaml</code>, I tried to remove the last <code>prometheus:</code> and moved <code>configMaps:</code> to the root as per this <a href="https://github.com/bitnami/charts/blob/master/bitnami/kube-prometheus/values.yaml" rel="nofollow noreferrer">documentation</a> but it broke the Prometheus configuration.
  As you can see in the file <code>/terraform/kubernetes/files/prometheus_config_map.yaml</code>, the <code>scrape_interval</code> is <code>15s</code> but when I check on Prometheus UI config, all of the scrape_interval is 30s, therefore, for sure the configuration of deployed Prometheus doesn't come from this file. Therefore, I have no way to change the configuration such as scrape_interval</p>
  <p><a href="https://i.stack.imgur.com/Fsm9v.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Fsm9v.png" alt="promConfig1" /></a></p>
  <p>I also notice in this <a href="https://github.com/bitnami/charts/blob/master/bitnami/kube-prometheus/values.yaml" rel="nofollow noreferrer">documentation</a> that I am missing many configuration in the file <code>/terraform/kubernetes/helm_values/prometheus.yaml</code> but I am not certain of what to add. Could you please let me know how I can resolve this? Thank you in advance.</p>
  <p><a href="https://i.stack.imgur.com/y9sNE.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/y9sNE.png" alt="promCONFIGmount" /></a></p>