title: Logging .net Core with Elastic stack
body: <p>Trying to set up simple logging with Filebeats, Logstash and be able to view logs in Kibana. Running a simple mvc .net core app with log4net as logger. log4net FileAppender appending logs to C:\Logs\Debug.log just fine. However not able to push those to Kibana.</p>
  
  <p>Based on this artice <a href="https://i.stack.imgur.com/fS3wG.png" rel="nofollow noreferrer">here</a>, I would set up filebeats, then transform log via logstash and be able to view my logs in Kibana.</p>
  
  <ol>
  <li>logstash.yml</li>
  </ol>
  
  <p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
  <div class="snippet-code">
  <pre class="snippet-code-html lang-html prettyprint-override"><code>- module: logstash
    # logs
    log:
      enabled: true
      #var.paths: ["C:/Logs/Debug.log"] - THIS CAUSES ERRROS - should this be UNCOMMENTED?
      
      # Convert the timestamp to UTC. Requires Elasticsearch &gt;= 6.1.
      #var.convert_timezone: false
  
    # Slow logs
    slowlog:
     enabled: true
      #var.paths: ["C:/Logs/Debug.log"]
  
      # Convert the timestamp to UTC. Requires Elasticsearch &gt;= 6.1.
      #var.convert_timezone: false</code></pre>
  </div>
  </div>
  </p>
  
  <ol start="2">
  <li>logstash-sample.conf</li>
  </ol>
  
  <p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
  <div class="snippet-code">
  <pre class="snippet-code-html lang-html prettyprint-override"><code># Sample Logstash configuration for creating a simple
  # Beats -&gt; Logstash -&gt; Elasticsearch pipeline.
  
  input { 
      file {
          path =&gt; "C:\Logs\Debug.log"
   type =&gt; "log4net"
          codec =&gt; multiline {
                      pattern =&gt; "^(DEBUG|WARN|ERROR|INFO|FATAL)"
                      negate =&gt; true
                      what =&gt; previous
                  }
        }
  }
  filter {
    if [type] == "log4net" {
      grok {
        match =&gt; [ "message", "(?m)%{LOGLEVEL:level} %{TIMESTAMP_ISO8601:sourceTimestamp} %{DATA:logger} \[%{NUMBER:threadId}\]  \[%{IPORHOST:tempHost}\] %{GREEDYDATA:tempMessage}" ]
      }
      mutate {
          replace =&gt; [ "message" , "%{tempMessage}" ]
          replace =&gt; [ "host" , "%{tempHost}" ]
          remove_field =&gt; [ "tempMessage" ]
          remove_field =&gt; [ "tempHost" ]
      }
    }
  }
  output {
    elasticsearch {
      host =&gt; localhost
      index =&gt; "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      #user =&gt; "elastic"
      #password =&gt; "changeme"
    }
  }</code></pre>
  </div>
  </div>
  </p>
  
  <p>Running logstash with config-sample output:
  <a href="https://i.stack.imgur.com/483Us.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/483Us.png" alt="enter image description here"></a></p>
  
  <ol start="3">
  <li>Filebeat.yml</li>
  </ol>
  
  <p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
  <div class="snippet-code">
  <pre class="snippet-code-html lang-html prettyprint-override"><code>###################### Filebeat Configuration Example #########################
  
  # This file is an example configuration file highlighting only the most common
  # options. The filebeat.reference.yml file from the same directory contains all the
  # supported options with more comments. You can use it as a reference.
  #
  # You can find the full configuration reference here:
  # https://www.elastic.co/guide/en/beats/filebeat/index.html
  
  # For more available modules and options, please see the filebeat.reference.yml sample
  # configuration file.
  
  #=========================== Filebeat inputs =============================
  
  filebeat.inputs:
  
  # Each - is an input. Most options can be set at the input level, so
  # you can use different inputs for various configurations.
  # Below are the input specific configurations.
  
  - type: log
  
    # Change to true to enable this input configuration.
    enabled: false
  
    # Paths that should be crawled and fetched. Glob based paths.
    paths:
      #- /var/log/*.log
      - c:\Logs\*.log
  
    # Exclude lines. A list of regular expressions to match. It drops the lines that are
    # matching any regular expression from the list.
    #exclude_lines: ['^DBG']
  
    # Include lines. A list of regular expressions to match. It exports the lines that are
    # matching any regular expression from the list.
    #include_lines: ['^ERR', '^WARN']
  
    # Exclude files. A list of regular expressions to match. Filebeat drops the files that
    # are matching any regular expression from the list. By default, no files are dropped.
    #exclude_files: ['.gz$']
  
    # Optional additional fields. These fields can be freely picked
    # to add additional information to the crawled log files for filtering
    #fields:
    #  level: debug
    #  review: 1
  
    ### Multiline options
  
    # Multiline can be used for log messages spanning multiple lines. This is common
    # for Java Stack Traces or C-Line Continuation
  
    # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
    #multiline.pattern: ^\[
  
    # Defines if the pattern set under pattern should be negated or not. Default is false.
    #multiline.negate: false
  
    # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
    # that was (not) matched before or after or as long as a pattern is not matched based on negate.
    # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
    #multiline.match: after
  
  
  #============================= Filebeat modules ===============================
  
  filebeat.config.modules:
    # Glob pattern for configuration loading
    path: ${path.config}/modules.d/*.yml
  
    # Set to true to enable config reloading
    reload.enabled: false
  
    # Period on which files under path should be checked for changes
    #reload.period: 10s
  
  #==================== Elasticsearch template setting ==========================
  
  setup.template.settings:
    index.number_of_shards: 3
    #index.codec: best_compression
    #_source.enabled: false
  
  #================================ General =====================================
  
  # The name of the shipper that publishes the network data. It can be used to group
  # all the transactions sent by a single shipper in the web interface.
  #name:
  
  # The tags of the shipper are included in their own field with each
  # transaction published.
  #tags: ["service-X", "web-tier"]
  
  # Optional fields that you can specify to add additional information to the
  # output.
  #fields:
  #  env: staging
  
  
  #============================== Dashboards =====================================
  # These settings control loading the sample dashboards to the Kibana index. Loading
  # the dashboards is disabled by default and can be enabled either by setting the
  # options here, or by using the `-setup` CLI flag or the `setup` command.
  #setup.dashboards.enabled: false
  
  # The URL from where to download the dashboards archive. By default this URL
  # has a value which is computed based on the Beat name and version. For released
  # versions, this URL points to the dashboard archive on the artifacts.elastic.co
  # website.
  #setup.dashboards.url:
  
  #============================== Kibana =====================================
  
  # Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
  # This requires a Kibana endpoint configuration.
  setup.kibana:
    host: "localhost:5601"
  
    # Kibana Host
    # Scheme and port can be left out and will be set to the default (http and 5601)
    # In case you specify and additional path, the scheme is required: http://localhost:5601/path
    # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
    #host: "localhost:5601"
  
    # Kibana Space ID
    # ID of the Kibana Space into which the dashboards should be loaded. By default,
    # the Default Space will be used.
    #space.id:
  
  #============================= Elastic Cloud ==================================
  
  # These settings simplify using filebeat with the Elastic Cloud (https://cloud.elastic.co/).
  
  # The cloud.id setting overwrites the `output.elasticsearch.hosts` and
  # `setup.kibana.host` options.
  # You can find the `cloud.id` in the Elastic Cloud web UI.
  #cloud.id:
  
  # The cloud.auth setting overwrites the `output.elasticsearch.username` and
  # `output.elasticsearch.password` settings. The format is `&lt;user&gt;:&lt;pass&gt;`.
  #cloud.auth:
  
  #================================ Outputs =====================================
  
  # Configure what output to use when sending the data collected by the beat.
  
  #-------------------------- Elasticsearch output ------------------------------
  #output.elasticsearch:
    # Array of hosts to connect to.
   # hosts: ["localhost:9200"]
  
    # Enabled ilm (beta) to use index lifecycle management instead daily indices.
    #ilm.enabled: false
  
    # Optional protocol and basic auth credentials.
    #protocol: "https"
    #username: "elastic"
    #password: "changeme"
  #----------------------------- Logstash output --------------------------------
  output.logstash:
    # The Logstash hosts
    hosts: ["localhost:5044"]
  
    # Optional SSL. By default is off.
    # List of root certificates for HTTPS server verifications
    #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
  
    # Certificate for SSL client authentication
    #ssl.certificate: "/etc/pki/client/cert.pem"
  
    # Client Certificate Key
    #ssl.key: "/etc/pki/client/cert.key"
  
  #================================ Processors =====================================
  
  # Configure processors to enhance or manipulate events generated by the beat.
  
  processors:
    - add_host_metadata: ~
    - add_cloud_metadata: ~
  
  #================================ Logging =====================================
  
  # Sets log level. The default log level is info.
  # Available log levels are: error, warning, info, debug
  #logging.level: debug
  
  # At debug level, you can selectively enable logging only for some components.
  # To enable all selectors use ["*"]. Examples of other selectors are "beat",
  # "publish", "service".
  #logging.selectors: ["*"]
  
  #============================== Xpack Monitoring ===============================
  # filebeat can export internal metrics to a central Elasticsearch monitoring
  # cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
  # reporting is disabled by default.
  
  # Set to true to enable the monitoring reporter.
  #xpack.monitoring.enabled: false
  
  # Uncomment to send the metrics to Elasticsearch. Most settings from the
  # Elasticsearch output are accepted here as well. Any setting that is not set is
  # automatically inherited from the Elasticsearch output configuration, so if you
  # have the Elasticsearch output configured, you can simply uncomment the
  # following line.
  #xpack.monitoring.elasticsearch:</code></pre>
  </div>
  </div>
  </p>
  
  <p>Output from my Browser windows:
  <a href="https://i.stack.imgur.com/fS3wG.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/fS3wG.png" alt="enter image description here"></a>
  <a href="https://i.stack.imgur.com/v7JKE.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/v7JKE.png" alt="enter image description here"></a></p>
  
  <p>I see my mvc app logging just fine (log4net) logs in C:\Logs\Debug.log, however not able to set it up so that these show up in Kibana.
  How would I set it up so that I would see my logs in Kibana?</p>
  
  <p><strong>EDIT 1:</strong></p>
  
  <ol>
  <li><p>logstash.config
  <div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
  <div class="snippet-code">
  <pre class="snippet-code-html lang-html prettyprint-override"><code># Sample Logstash configuration for creating a simple
  # Beats -&gt; Logstash -&gt; Elasticsearch pipeline.
  
  input {
    beats {
      port =&gt; 5044
    }
  }
  filter {
    grok {      
        match =&gt; { "message" =&gt; "(?m)^%{TIMESTAMP_ISO8601:timestamp}~~\[%{DATA:thread}\]~~\[%{DATA:user}\]~~\[%{DATA:requestId}\]~~\[%{DATA:userHost}\]~~\[%{DATA:requestUrl}\]~~%{DATA:level}~~%{DATA:logger}~~%{DATA:logmessage}~~%{DATA:exception}\|\|" }
        add_field =&gt; { 
          "received_at" =&gt; "%{@timestamp}" 
          "received_from" =&gt; "%{host}"
        }
        remove_field =&gt; ["message"]      
      }
    date {
      match =&gt; [ "timestamp", "yyyy-MM-dd HH:mm:ss:SSS" ]
    }
  }
  output {
    elasticsearch {
   hosts =&gt; ["http://localhost:9200"]
   sniffing =&gt; true
      index =&gt; "%{app_name}_%{app_env}_%{type}-%{+YYYY.MM.dd}"
   document_type =&gt; "%{[@metadata][type]}"
      #user =&gt; "elastic"
      #password =&gt; "changeme"
    }
    stdout { codec =&gt; rubydebug }
  }</code></pre>
  </div>
  </div>
  </p></li>
  <li><p>filebeat.yml</p></li>
  </ol>
  
  <p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
  <div class="snippet-code">
  <pre class="snippet-code-html lang-html prettyprint-override"><code>filebeat.inputs:
  
  # Each - is an input. Most options can be set at the input level, so
  # you can use different inputs for various configurations.
  # Below are the input specific configurations.
  
  - type: log
  
    # Change to true to enable this input configuration.
    enabled: true
  
    # Paths that should be crawled and fetched. Glob based paths.
    paths:
      #- /var/log/*.log
      - c:\Logs\*.log
      
     .....
     
  #-------------------------- Elasticsearch output ------------------------------
  #output.elasticsearch:
    # Array of hosts to connect to.
    #hosts: ["localhost:9200"]
  
    # Enabled ilm (beta) to use index lifecycle management instead daily indices.
    #ilm.enabled: false
  
    # Optional protocol and basic auth credentials.
    #protocol: "https"
    #username: "elastic"
    #password: "changeme"
  #----------------------------- Logstash output --------------------------------
  output.logstash:
    # The Logstash hosts
    hosts: ["localhost:5044"]</code></pre>
  </div>
  </div>
  </p>
  
  <p>I have filebeats enabled/running as service. Also logstash running (see powershell window below). When I change anything in Debug.log file and save, i see those changes being output to console right away.</p>
  
  <p><a href="https://i.stack.imgur.com/FTuD3.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/FTuD3.png" alt="enter image description here"></a></p>
  
  <p>However, when I go to dashboard I do not see any logs still. What am I doing wrong?
  <a href="https://i.stack.imgur.com/Zd9lj.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Zd9lj.png" alt="enter image description here"></a></p>