title: Kibana don't store the right data into the right fiels
body: <p>Ok so now I've my mapping into kibana.
  Here's my mapping: </p>
  
  <pre><code>PUT logstash-2019.05.09
  {
    "mappings": {
      "doc": {
        "properties": {
          "index": {
            "_index": {
              "type": "keyword"
            },
            "_type": {
              "type": "text"
            }
          },
          "@timestamp": {
            "type": "date"
          },
          "ip": {
            "type": "ip"
          },
          "extension": {
            "type": "text"
          },
          "response": {
            "type": "text"
          },
          "geo": {
            "coordinates": {
              "type": "geo_point"
            },
            "src": {
              "type": "text"
            },
            "dest": {
              "type": "text"
            },
            "srcdest": {
              "type": "text"
            }
          },
          "tags": {
            "type": "text"
          },
          "utc_time": {
            "type": "date"
          },
          "referer": {
            "type": "text"
          },
          "agent": {
            "type": "text"
          },
          "clientip": {
            "type": "ip"
          },
          "bytes": {
            "type": "integer"
          },
          "host": {
            "type": "text"
          },
          "request": {
            "type": "text"
          },
          "url": {
            "type": "text"
          },
          "@message": {
            "type": "text"
          },
          "spaces": {
            "type": "text"
          },
          "xss": {
            "type": "text"
          },
          "links": {
            "type": "text"
          },
          "relatedContent": {
            "url": {
              "type": "text"
            },
            "og:type": {
              "type": "text"
            },
            "og:title": {
              "type": "text"
            },
            "og:description": {
              "type": ""
            },
            "og:url": {
              "type": ""
            },
            "article:published_time": {
              "type": "date"
            },
            "article:modified_time": {
              "type": "date"
            },
            "article:section": {
              "type": "keyword"
            },
            "article:tag": {
              "type": "text"
            },
            "og:image": {
              "type": "text"
            },
            "og:image:height": {
              "type": "integer"
            },
            "og:image:width": {
              "type": "integer"
            },
            "og:site_name": {
              "type": "text"
            },
            "twitter:title": {
              "type": "text"
            },
            "twitter:description": {
              "type": "text"
            },
            "twitter:card": {
              "type": "keyword"
            },
            "twitter:image": {
              "type": "text"
            },
            "twitter:site": {
              "type": "keyword"
            }
          },
          "machine": {
            "os": {
              "type": "text"
            },
            "ram": {
              "type": "integer"
            }
          },
          "@version": {
            "type": "integer"
          }
        }
      }
    }
  }
  </code></pre>
  
  <p>But I don't know why, Kibana don't store the right information into the right field. He just put all the information into a message field. I think it's because I've a dynamic mapping by default, I'm not really sure. Here's the result :</p>
  
  <p><a href="https://i.stack.imgur.com/j4Qab.png" rel="nofollow noreferrer">Result (table)</a>
  result (json):</p>
  
  <pre><code>    {
    "_index": "logstash-2019.05.09",
    "_type": "doc",
    "_id": "9zfam2oBWngGU4Wy3Id5",
    "_version": 1,
    "_score": null,
    "_source": {
      "@version": "1",
      "@timestamp": "2019-05-09T09:09:32.167Z",
      "path": "/home/secunix/logs/TestLogPourMapping_09_05.json",
      "message": "{\"@timestamp\":\"2019-05-07T09:56:33.996Z\",\"ip\":\"181.144.250.19\",\"extension\":\"jpg\",\"response\":\"200\",\"geo\":{\"coordinates\":{\"lat\":44.12326,\"lon\":-123.2186856},\"src\":\"IN\",\"dest\":\"CN\",\"srcdest\":\"IN:CN\"},\"@tags\":[\"success\",\"info\"],\"utc_time\":\"2019-05-07T09:56:33.996Z\",\"referer\":\"http://www.slate.com/success/thomas-marshburn\",\"agent\":\"Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.50 Safari/534.24\",\"clientip\":\"181.144.250.19\",\"bytes\":2553,\"host\":\"media-for-the-masses.theacademyofperformingartsandscience.org\",\"request\":\"/uploads/fyodor-yurchikhin.jpg\",\"url\":\"https://media-for-the-masses.theacademyofperformingartsandscience.org/uploads/fyodor-yurchikhin.jpg\",\"@message\":\"181.144.250.19 - - [2019-05-07T09:56:33.996Z] \\\"GET /uploads/fyodor-yurchikhin.jpg HTTP/1.1\\\" 200 2553 \\\"-\\\" \\\"Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.50 Safari/534.24\\\"\",\"spaces\":\"this   is   a   thing    with lots of     spaces       wwwwoooooo\",\"xss\":\"&lt;script&gt;console.log(\\\"xss\\\")&lt;/script&gt;\",\"headings\":[\"&lt;h3&gt;ulrich-walter&lt;/h5&gt;\",\"http://www.slate.com/success/susan-still-kilrain\"],\"links\":[\"viktor-m-afanasyev@twitter.com\",\"http://twitter.com/security/stephen-oswald\",\"www.twitter.com\"],\"relatedContent\":[],\"machine\":{\"os\":\"win xp\",\"ram\":6442450944},\"@version\":\"1\"}\r",
      "host": "qvisbcld0051"
    },
    "fields": {
      "@timestamp": [
        "2019-05-09T09:09:32.167Z"
      ]
    },
    "sort": [
      1557392972167
    ]
  }
  </code></pre>
  
  <p>And that's what I have when I check my mapping:</p>
  
  <pre><code>{
    "mapping": {
      "doc": {
        "dynamic_templates": [
          {
            "message_field": {
              "path_match": "message",
              "match_mapping_type": "string",
              "mapping": {
                "norms": false,
                "type": "text"
              }
            }
          },
          {
            "string_fields": {
              "match": "*",
              "match_mapping_type": "string",
              "mapping": {
                "fields": {
                  "keyword": {
                    "ignore_above": 256,
                    "type": "keyword"
                  }
                },
                "norms": false,
                "type": "text"
              }
            }
          }
        ],
        "properties": {
          "@message": {
            "type": "text"
          },
          "@timestamp": {
            "type": "date"
          },
          "@version": {
            "type": "integer"
          },
          "agent": {
            "type": "text"
          },
          "bytes": {
            "type": "integer"
          },
          "clientip": {
            "type": "ip"
          },
          "extension": {
            "type": "text"
          },
          "geo": {
            "properties": {
              "coordinates": {
                "type": "geo_point"
              },
              "dest": {
                "type": "text"
              },
              "src": {
                "type": "text"
              },
              "srcdest": {
                "type": "text"
              }
            }
          },
          "geoip": {
            "dynamic": "true",
            "properties": {
              "ip": {
                "type": "ip"
              },
              "latitude": {
                "type": "half_float"
              },
              "location": {
                "type": "geo_point"
              },
              "longitude": {
                "type": "half_float"
              }
            }
          },
          "host": {
            "type": "text"
          },
          "ip": {
            "type": "ip"
          },
          "links": {
            "type": "text"
          },
          "machine": {
            "properties": {
              "os": {
                "type": "text"
              },
              "ram": {
                "type": "integer"
              }
            }
          },
          "message": {
            "type": "text",
            "norms": false
          },
          "path": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "referer": {
            "type": "text"
          },
          "relatedContent": {
            "properties": {
              "article:modified_time": {
                "type": "date"
              },
              "article:published_time": {
                "type": "date"
              },
              "article:section": {
                "type": "keyword"
              },
              "article:tag": {
                "type": "text"
              },
              "og:description": {
                "type": "text"
              },
              "og:image": {
                "type": "text"
              },
              "og:image:height": {
                "type": "integer"
              },
              "og:image:width": {
                "type": "integer"
              },
              "og:site_name": {
                "type": "text"
              },
              "og:title": {
                "type": "text"
              },
              "og:type": {
                "type": "text"
              },
              "og:url": {
                "type": "text"
              },
              "twitter:card": {
                "type": "keyword"
              },
              "twitter:description": {
                "type": "text"
              },
              "twitter:image": {
                "type": "text"
              },
              "twitter:site": {
                "type": "keyword"
              },
              "twitter:title": {
                "type": "text"
              },
              "url": {
                "type": "text"
              }
            }
          },
          "request": {
            "type": "text"
          },
          "response": {
            "type": "text"
          },
          "spaces": {
            "type": "text"
          },
          "tags": {
            "type": "text"
          },
          "url": {
            "type": "text"
          },
          "utc_time": {
            "type": "date"
          },
          "xss": {
            "type": "text"
          }
        }
      },
      "_default_": {
        "dynamic_templates": [
          {
            "message_field": {
              "path_match": "message",
              "match_mapping_type": "string",
              "mapping": {
                "norms": false,
                "type": "text"
              }
            }
          },
          {
            "string_fields": {
              "match": "*",
              "match_mapping_type": "string",
              "mapping": {
                "fields": {
                  "keyword": {
                    "ignore_above": 256,
                    "type": "keyword"
                  }
                },
                "norms": false,
                "type": "text"
              }
            }
          }
        ],
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "@version": {
            "type": "keyword"
          },
          "geoip": {
            "dynamic": "true",
            "properties": {
              "ip": {
                "type": "ip"
              },
              "latitude": {
                "type": "half_float"
              },
              "location": {
                "type": "geo_point"
              },
              "longitude": {
                "type": "half_float"
              }
            }
          }
        }
      }
    }
  }
  </code></pre>
  
  <p>I send my data thanks to logstash, so here's the conf of the input:</p>
  
  <pre><code>input {
    beats {
       port =&gt; 5044
       tags =&gt; "fromBeats"
    }
    file {
      path =&gt; [
      "/home/secunix/logs/*",
      "/tech/*"
      ]
      start_position =&gt; "beginning"
  
      sincedb_path =&gt; "/dev/null"
    }
    tcp {
      port =&gt; 5514
      type =&gt; "syslog"
      tags =&gt; "from Syslog-ng"
    } 
  }
  
  filter {
   if [type] == "syslog"{
      grok {
        match =&gt; ["message", "&lt;(?&lt;sys_priority&gt;\d+?)&gt;(?&lt;syslog_timestamp&gt;%{CISCOTIMESTAMP})\s(?&lt;logsource&gt;%{URIHOST})(\s(?:(?&lt;application&gt;.*?)(%(?&lt;project&gt;.*?))?))?:(?:\s)?(?&lt;logmessage&gt;.*$)"]
      }
      if [logmessage] {
        mutate {
          replace =&gt; [ "message", "%{logmessage}" ]
          remove_field =&gt; [ "logmessage" ]
        }
      }
      if [project]  {
        mutate {
          replace =&gt; [ "type", "%{project}" ]
          remove_field =&gt; [ "project" ]
        }
      }else if [application] {
        mutate {
          lowercase =&gt; [ "application" ]
        }
        mutate {
          gsub =&gt; [ "application", " ", "_" ]
        }
        mutate {
          replace =&gt; [ "type", "%{application}" ]
        }
      }else {
        mutate {
          replace =&gt; [ "type", "uknapp" ]
          add_field =&gt; { "application" =&gt; "uknapp" }
        }
      }
    }
  }
  </code></pre>
  
  <p>and the output:</p>
  
  <pre><code>output {
     elasticsearch {
          hosts =&gt; ["localhost:9200"]
          index =&gt; "logstash-%{+YYYY.MM.dd}"
          #+++ sa Added by scr-sop-af-config-elksandbox. Do not remove this line.
  user =&gt; "logstash"
  #--- sa Added by scr-sop-af-config-elksandbox. Do not remove this line.
          #+++ sa Added by scr-sop-af-config-elksandbox. Do not remove this line.
  password =&gt; "logstash"
  #--- sa Added by scr-sop-af-config-elksandbox. Do not remove this line.
  
     }
  }
  
  </code></pre>
  
  <p>Can someone tell me how I can fix this please ?</p>