问:
Need to shorten Grok Pattern
I am using a Grok pattern to parse firewall logs and the Grok pattern works when using Grok Debugger. The log data changes and I've created patterns to match each change. My issue is that ELK is producing multiple duplicate fields for the parsed data. I am sure there is a way to shorten my grok pattern but at this point I am not able to figure out how. So any assistance would be awesome. See below for examples:

Sample logs:

```<code>Nov 15 12:18:31 removed_ip 2017:11:15-12:18:31 sophie ulogd[23109]: id="2001" severity="info" sys="SecureNet" sub="packetfilter" name="Packet dropped" action="drop" fwrule="60001" initf="eth3" srcmac="removed_mac" dstmac="removed_mac" srcip="removed_ip" dstip="removed_ip" proto="6" length="40" tos="0x00" prec="0x00" ttl="247" srcport="58261" dstport="5315" tcpflags="SYN"
Nov 15 12:33:01 removed_ip 2017:11:15-12:33:01 sophie ulogd[23109]: id="2001" severity="info" sys="SecureNet" sub="packetfilter" name="Packet dropped" action="drop" fwrule="60003" outitf="wlan1" srcmac="removed_mac" srcip="removed_ip" dstip="removed_ip" proto="6" length="40" tos="0x00" prec="0x00" ttl="64" srcport="443" dstport="49824" tcpflags="RST"
Nov 15 12:20:29 removed_ip 2017:11:15-12:20:29 sophie httpproxy[6835]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="GET" srcip="removed_ip" dstip="removed_ip" user="" group="" ad_domain="" statuscode="200" cached="0" profile="REF_DefaultHTTPProfile (Default Web Filter Profile)" filteraction="REF_DefaultHTTPCFFAction (Default content filter action)" size="371" request="0xd3a9ac00" url="http://removed_ip/icingaweb2/monitoring/tactical?view=compact" referer="http://removed_ip/icingaweb2/dashboard" error="" authtime="0" dnstime="218" cattime="0" avscantime="2584" fullreqtime="15393756" device="0" auth="0" ua="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36" exceptions="" overridecategory="1" overridereputation="1" category="105" reputation="trusted" categoryname="Business" country="United States" sandbox="-" content-type="text/xml"
Nov 15 12:30:33 removed_ip 2017:11:15-12:30:33 sophie httpproxy[6835]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="CONNECT" srcip="removed_ip" dstip="removed_ip" user="" group="" ad_domain="" statuscode="200" cached="0" profile="REF_DefaultHTTPProfile (Default Web Filter Profile)" filteraction="REF_DefaultHTTPCFFAction (Default content filter action)" size="11571" request="0xd21c1800" url="https://www.google.com/" referer="" error="" authtime="0" dnstime="1" cattime="97" avscantime="0" fullreqtime="361956728" device="0" auth="0" ua="" exceptions="" category="145" reputation="neutral" categoryname="Search Engines" country="United States" application="google" app-id="182"
</code>```

Sample Grok Pattern:

```<code>filter {
if [type] == "utm"{
grok {
  match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" fwrule=\"%{INT:fwrule}\" initf=\"%{NOTSPACE:initf}\" outitf=\"%{NOTSPACE:outitf}\" mark=\"%{WORD:mark}\" app=\"%{WORD:app}\" srcmac=\"%{MAC:srcmac}\" dstmac=\"%{MAC:dstmac}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" proto=\"%{WORD:protocol}\" length=\"%{INT:length}\" tos=\"%{DATA:tos}\" prec=\"%{DATA:prec}\" ttl=\"%{INT:ttl}\" srcport=\"%{INT:srcport}\" dstport=\"%{INT:dstport}\" "}
}
grok {
  match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" fwrule=\"%{INT:fwrule}\" initf=\"%{NOTSPACE:initf}\" outitf=\"%{NOTSPACE:outitf}\" srcmac=\"%{MAC:srcmac}\" dstmac=\"%{MAC:dstmac}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" proto=\"%{WORD:protocol}\" length=\"%{INT:length}\" tos=\"%{DATA:tos}\" prec=\"%{DATA:prec}\" ttl=\"%{INT:ttl}\" srcport=\"%{INT:srcport}\" dstport=\"%{INT:dstport}\" "}
 }
grok {
  match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" fwrule=\"%{INT:fwrule}\" initf=\"%{NOTSPACE:initf}\" outitf=\"%{NOTSPACE:outitf}\" srcmac=\"%{MAC:srcmac}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" proto=\"%{WORD:protocol}\" length=\"%{INT:length}\" tos=\"%{DATA:tos}\" prec=\"%{DATA:prec}\" ttl=\"%{INT:ttl}\" srcport=\"%{INT:srcport}\" dstport=\"%{INT:dstport}\" "}
 }
grok {
  match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" fwrule=\"%{INT:fwrule}\" outitf=\"%{NOTSPACE:outitf}\" mark=\"%{DATA:mark}\" app=\"%{DATA:app}\" srcmac=\"%{MAC:srcmac}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" proto=\"%{WORD:protocol}\" length=\"%{INT:length}\" tos=\"%{DATA:tos}\" prec=\"%{DATA:prec}\" ttl=\"%{INT:ttl}\" srcport=\"%{INT:srcport}\" dstport=\"%{INT:dstport}\" "}
 }
grok {
  match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" fwrule=\"%{INT:fwrule}\" initf=\"%{NOTSPACE:initf}\" srcmac=\"%{MAC:srcmac}\" dstmac=\"%{MAC:dstmac}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" proto=\"%{WORD:protocol}\" length=\"%{INT:length}\" tos=\"%{DATA:tos}\" prec=\"%{DATA:prec}\" ttl=\"%{INT:ttl}\" srcport=\"%{INT:srcport}\" dstport=\"%{INT:dstport}\" "}
 }
grok {
  match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" fwrule=\"%{INT:fwrule}\" outitf=\"%{NOTSPACE:outitf}\" srcmac=\"%{MAC:srcmac}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" proto=\"%{WORD:protocol}\" length=\"%{INT:length}\" tos=\"%{DATA:tos}\" prec=\"%{DATA:prec}\" ttl=\"%{INT:ttl}\" srcport=\"%{INT:srcport}\" dstport=\"%{INT:dstport}\" "}
 }
grok {
  match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" fwrule=\"%{INT:fwrule}\" initf=\"%{NOTSPACE:initf}\" outitf=\"%{NOTSPACE:outitf}\" srcmac=\"%{MAC:srcmac}\" dstmac=\"%{MAC:dstmac}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" proto=\"%{WORD:protocol}\" length=\"%{INT:length}\" tos=\"%{DATA:tos}\" prec=\"%{DATA:prec}\" ttl=\"%{INT:ttl}\" srcport=\"%{INT:srcport}\" dstport=\"%{INT:dstport}\" tcpflags=\"%{DATA:tcpflags}\" \"(,)\" "}
}
grok {
 match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" fwrule=\"%{INT:fwrule}\" initf=\"%{NOTSPACE:initf}\" srcmac=\"%{MAC:srcmac}\" dstmac=\"%{MAC:dstmac}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" proto=\"%{WORD:protocol}\" length=\"%{INT:length}\" tos=\"%{DATA:tos}\" prec=\"%{DATA:prec}\" ttl=\"%{INT:ttl}\" srcport=\"%{INT:srcport}\" dstport=\"%{INT:dstport}\" tcpflags=\"%{DATA:tcpflags}\" "}
}
grok {
 match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" fwrule=\"%{INT:fwrule}\" outitf=\"%{NOTSPACE:outitf}\" srcmac=\"%{MAC:srcmac}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" proto=\"%{WORD:protocol}\" length=\"%{INT:length}\" tos=\"%{DATA:tos}\" prec=\"%{DATA:prec}\" ttl=\"%{INT:ttl}\" srcport=\"%{INT:srcport}\" dstport=\"%{INT:dstport}\" tcpflags=\"%{DATA:tcpflags}\" "}
}
grok {
 match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" fwrule=\"%{INT:fwrule}\" initf=\"%{NOTSPACE:initf}\" outitf=\"%{NOTSPACE:outitf}\" mark=\"%{WORD:mark}\" app=\"%{WORD:app}\" srcmac=\"%{MAC:srcmac}\" dstmac=\"%{MAC:dstmac}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" proto=\"%{WORD:protocol}\" length=\"%{INT:length}\" tos=\"%{DATA:tos}\" prec=\"%{DATA:prec}\" ttl=\"%{INT:ttl}\" srcport=\"%{INT:srcport}\" dstport=\"%{INT:dstport}\" tcpflags=\"%{DATA:tcpflags}\" "}
}
grok {
 match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" fwrule=\"%{INT:fwrule}\" initf=\"%{NOTSPACE:initf}\" outitf=\"%{NOTSPACE:outitf}\" mark=\"%{WORD:mark}\" srcmac=\"%{MAC:srcmac}\" dstmac=\"%{MAC:dstmac}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" proto=\"%{WORD:protocol}\" length=\"%{INT:length}\" tos=\"%{DATA:tos}\" prec=\"%{DATA:prec}\" ttl=\"%{INT:ttl}\" srcport=\"%{INT:srcport}\" dstport=\"%{INT:dstport}\" tcpflags=\"%{DATA:tcpflags}\" "}
}
}
if "httpproxy" in [message]{
grok {
match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" method=\"%{WORD:method}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" user=\"%{DATA:user}\" group=\"%{DATA:group}\" ad_domain=\"%{DATA:ad_domain}\" statuscode=\"%{INT:statuscode}\" cached=\"%{INT:cached}\" profile=\"%{DATA:profile}\" filteraction=\"%{DATA:filteraction}\" size=\"%{INT:size}\" request=\"%{BASE16FLOAT:request}\" url=\"%{URI:url}\" referer=\"%{DATA:referer}\" error=\"%{DATA:error}\" authtime=\"%{INT:authtime}\" dnstime=\"%{INT:dnstime}\" cattime=\"%{INT:cattime}\" avscantime=\"%{INT:avscantime}\" fullreqtime=\"%{INT:fullreqtime}\" device=\"%{INT:device}\" auth=\"%{INT:auth}\" ua=\"%{DATA:ua}\" exceptions=\"%{DATA:exceptions}\" category=\"%{INT:category}\" reputation=\"%{WORD:reputation}\" categoryname=\"%{DATA:categoryname}\" country=\"%{DATA:country}\" application=\"%{WORD:application}\" app-id=\"%{INT:app-id}\" sandbox=\"%{DATA:sandbox}\" content-type=\"%{DATA:content-type}\" "}
}
grok {
match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" method=\"%{WORD:method}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" user=\"%{DATA:user}\" group=\"%{DATA:group}\" ad_domain=\"%{DATA:ad_domain}\" statuscode=\"%{INT:statuscode}\" cached=\"%{INT:cached}\" profile=\"%{DATA:profile}\" filteraction=\"%{DATA:filteraction}\" size=\"%{INT:size}\" request=\"%{BASE16FLOAT:request}\" url=\"%{URI:url}\" referer=\"%{DATA:referer}\" error=\"%{DATA:error}\" authtime=\"%{INT:authtime}\" dnstime=\"%{INT:dnstime}\" cattime=\"%{INT:cattime}\" avscantime=\"%{INT:avscantime}\" fullreqtime=\"%{INT:fullreqtime}\" device=\"%{INT:device}\" auth=\"%{INT:auth}\" ua=\"%{DATA:ua}\" exceptions=\"%{DATA:exceptions}\" overridecategory=\"%{INT:overridecategory}\" overridereputation=\"%{INT:overridereputation}\" category=\"%{INT:category}\" reputation=\"%{WORD:reputation}\" categoryname=\"%{DATA:categoryname}\" country=\"%{DATA:country}\" sandbox=\"%{DATA:sandbox}\" content-type=\"%{DATA:content-type}\" "}
}
grok {
 match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" method=\"%{WORD:method}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" user=\"%{DATA:user}\" group=\"%{DATA:group}\" ad_domain=\"%{DATA:ad_domain}\" statuscode=\"%{INT:statuscode}\" cached=\"%{INT:cached}\" profile=\"%{DATA:profile}\" filteraction=\"%{DATA:filteraction}\" size=\"%{INT:size}\" request=\"%{BASE16FLOAT:request}\" url=\"%{URI:url}\" referer=\"%{DATA:referer}\" error=\"%{DATA:error}\" authtime=\"%{INT:authtime}\" dnstime=\"%{INT:dnstime}\" cattime=\"%{INT:cattime}\" avscantime=\"%{INT:avscantime}\" fullreqtime=\"%{INT:fullreqtime}\" device=\"%{INT:device}\" auth=\"%{INT:auth}\" ua=\"%{DATA:ua}\" exceptions=\"%{DATA:exceptions}\" category=\"%{INT:category}\" reputation=\"%{WORD:reputation}\" categoryname=\"%{DATA:categoryname}\" sandbox=\"%{DATA:sandbox}\" content-type=\"%{DATA:content-type}\" "}
}
grok {
match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" method=\"%{WORD:method}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" user=\"%{DATA:user}\" group=\"%{DATA:group}\" ad_domain=\"%{DATA:ad_domain}\" statuscode=\"%{INT:statuscode}\" cached=\"%{INT:cached}\" profile=\"%{DATA:profile}\" filteraction=\"%{DATA:filteraction}\" size=\"%{INT:size}\" request=\"%{BASE16FLOAT:request}\" url=\"%{URI:url}\" referer=\"%{DATA:referer}\" error=\"%{DATA:error}\" authtime=\"%{INT:authtime}\" dnstime=\"%{INT:dnstime}\" cattime=\"%{INT:cattime}\" avscantime=\"%{INT:avscantime}\" fullreqtime=\"%{INT:fullreqtime}\" device=\"%{INT:device}\" auth=\"%{INT:auth}\" ua=\"%{DATA:ua}\" exceptions=\"%{DATA:exceptions}\" "}
}
grok {
match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" method=\"%{WORD:method}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" user=\"%{DATA:user}\" group=\"%{DATA:group}\" ad_domain=\"%{DATA:ad_domain}\" statuscode=\"%{INT:statuscode}\" cached=\"%{INT:cached}\" profile=\"%{DATA:profile}\" filteraction=\"%{DATA:filteraction}\" size=\"%{INT:size}\" request=\"%{BASE16FLOAT:request}\" url=\"%{URI:url}\" referer=\"%{DATA:referer}\" error=\"%{DATA:error}\" authtime=\"%{INT:authtime}\" dnstime=\"%{INT:dnstime}\" cattime=\"%{INT:cattime}\" avscantime=\"%{INT:avscantime}\" fullreqtime=\"%{INT:fullreqtime}\" device=\"%{INT:device}\" auth=\"%{INT:auth}\" ua=\"%{DATA:ua}\" exceptions=\"%{DATA:exceptions}\" overridecategory=\"%{INT:overridecategory}\" overridereputation=\"%{INT:overridereputation}\" country=\"%{DATA:country}\" "}
}
grok {
match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" method=\"%{WORD:method}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" user=\"%{DATA:user}\" group=\"%{DATA:group}\" ad_domain=\"%{DATA:ad_domain}\" statuscode=\"%{INT:statuscode}\" cached=\"%{INT:cached}\" profile=\"%{DATA:profile}\" filteraction=\"%{DATA:filteraction}\" size=\"%{INT:size}\" request=\"%{BASE16FLOAT:request}\" url=\"%{URI:url}\" referer=\"%{DATA:referer}\" error=\"%{DATA:error}\" authtime=\"%{INT:authtime}\" dnstime=\"%{INT:dnstime}\" cattime=\"%{INT:cattime}\" avscantime=\"%{INT:avscantime}\" fullreqtime=\"%{INT:fullreqtime}\" device=\"%{INT:device}\" auth=\"%{INT:auth}\" ua=\"%{DATA:ua}\" exceptions=\"%{DATA:exceptions}\" overridecategory=\"%{INT:overridecategory}\" overridereputation=\"%{INT:overridereputation}\" category=\"%{INT:category}\" reputation=\"%{WORD:reputation}\" categoryname=\"%{DATA:categoryname}\" country=\"%{DATA:country}\" "}
}
grok {
match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" method=\"%{WORD:method}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" user=\"%{DATA:user}\" group=\"%{DATA:group}\" ad_domain=\"%{DATA:ad_domain}\" statuscode=\"%{INT:statuscode}\" cached=\"%{INT:cached}\" profile=\"%{DATA:profile}\" filteraction=\"%{DATA:filteraction}\" size=\"%{INT:size}\" request=\"%{BASE16FLOAT:request}\" url=\"%{URI:url}\" referer=\"%{DATA:referer}\" error=\"%{DATA:error}\" authtime=\"%{INT:authtime}\" dnstime=\"%{INT:dnstime}\" cattime=\"%{INT:cattime}\" avscantime=\"%{INT:avscantime}\" fullreqtime=\"%{INT:fullreqtime}\" device=\"%{INT:device}\" auth=\"%{INT:auth}\" ua=\"%{DATA:ua}\" exceptions=\"%{DATA:exceptions}\" category=\"%{INT:category}\" reputation=\"%{WORD:reputation}\" categoryname=\"%{DATA:categoryname}\" country=\"%{DATA:country}\" "}
}
grok {
match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" method=\"%{WORD:method}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" user=\"%{DATA:user}\" group=\"%{DATA:group}\" ad_domain=\"%{DATA:ad_domain}\" statuscode=\"%{INT:statuscode}\" cached=\"%{INT:cached}\" profile=\"%{DATA:profile}\" filteraction=\"%{DATA:filteraction}\" size=\"%{INT:size}\" request=\"%{BASE16FLOAT:request}\" url=\"%{URI:url}\" referer=\"%{DATA:referer}\" error=\"%{DATA:error}\" authtime=\"%{INT:authtime}\" dnstime=\"%{INT:dnstime}\" cattime=\"%{INT:cattime}\" avscantime=\"%{INT:avscantime}\" fullreqtime=\"%{INT:fullreqtime}\" device=\"%{INT:device}\" auth=\"%{INT:auth}\" ua=\"%{DATA:ua}\" exceptions=\"%{DATA:exceptions}\" category=\"%{INT:category}\" reputation=\"%{WORD:reputation}\" categoryname=\"%{DATA:categoryname}\" country=\"%{DATA:country}\" application=\"%{WORD:application}\" app-id=\"%{INT:app-id}\" "}
}
}
}
</code>```

As you can see I spent sometime trying to account for each variation of log entries the firewall will produce. For the most part the data is the same but after a certain point it changes. When logstash parse the data it produce multiple data for one field. Here is a pic of the duplicate fields screenshot from ELK

My assumption is that my grok patterns are too similar. I originally tried break_on_match => false but that produced the same duplicates.

答:
Like @Phonolog said, since you have multiple grok filter, your message will be matched once with each filter. And if more than one filter successfully match its pattern against your message, you'll have duplicates of the field.

Instead of trying to match each possible combination of key-value pair at the end of your message, you can put the part with the key-value pairs in one field and then use the kv filter on it.

It would look like this:

```<code>grok {
  match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: %{GREEDYDATA:kvdata}"}
}

kv {
    source => "kvdata"
    trim_value => "\""
 }
}
</code>```

Title: Need to shorten Grok Pattern

Body:
I am using a Grok pattern to parse firewall logs and the Grok pattern works when using Grok Debugger. The log data changes and I've created patterns to match each change. My issue is that ELK is producing multiple duplicate fields for the parsed data. I am sure there is a way to shorten my grok pattern but at this point I am not able to figure out how. So any assistance would be awesome. See below for examples:

Sample logs:

Nov 15 12:18:31 removed_ip 2017:11:15-12:18:31 sophie ulogd[23109]: id="2001" severity="info" sys="SecureNet" sub="packetfilter" name="Packet dropped" action="drop" fwrule="60001" initf="eth3" srcmac="removed_mac" dstmac="removed_mac" srcip="removed_ip" dstip="removed_ip" proto="6" length="40" tos="0x00" prec="0x00" ttl="247" srcport="58261" dstport="5315" tcpflags="SYN"
Nov 15 12:33:01 removed_ip 2017:11:15-12:33:01 sophie ulogd[23109]: id="2001" severity="info" sys="SecureNet" sub="packetfilter" name="Packet dropped" action="drop" fwrule="60003" outitf="wlan1" srcmac="removed_mac" srcip="removed_ip" dstip="removed_ip" proto="6" length="40" tos="0x00" prec="0x00" ttl="64" srcport="443" dstport="49824" tcpflags="RST"
Nov 15 12:20:29 removed_ip 2017:11:15-12:20:29 sophie httpproxy[6835]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="GET" srcip="removed_ip" dstip="removed_ip" user="" group="" ad_domain="" statuscode="200" cached="0" profile="REF_DefaultHTTPProfile (Default Web Filter Profile)" filteraction="REF_DefaultHTTPCFFAction (Default content filter action)" size="371" request="0xd3a9ac00" url="http://removed_ip/icingaweb2/monitoring/tactical?view=compact" referer="http://removed_ip/icingaweb2/dashboard" error="" authtime="0" dnstime="218" cattime="0" avscantime="2584" fullreqtime="15393756" device="0" auth="0" ua="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36" exceptions="" overridecategory="1" overridereputation="1" category="105" reputation="trusted" categoryname="Business" country="United States" sandbox="-" content-type="text/xml"
Nov 15 12:30:33 removed_ip 2017:11:15-12:30:33 sophie httpproxy[6835]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="CONNECT" srcip="removed_ip" dstip="removed_ip" user="" group="" ad_domain="" statuscode="200" cached="0" profile="REF_DefaultHTTPProfile (Default Web Filter Profile)" filteraction="REF_DefaultHTTPCFFAction (Default content filter action)" size="11571" request="0xd21c1800" url="https://www.google.com/" referer="" error="" authtime="0" dnstime="1" cattime="97" avscantime="0" fullreqtime="361956728" device="0" auth="0" ua="" exceptions="" category="145" reputation="neutral" categoryname="Search Engines" country="United States" application="google" app-id="182"

Sample Grok Pattern:

filter {
if [type] == "utm"{
grok {
  match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" fwrule=\"%{INT:fwrule}\" initf=\"%{NOTSPACE:initf}\" outitf=\"%{NOTSPACE:outitf}\" mark=\"%{WORD:mark}\" app=\"%{WORD:app}\" srcmac=\"%{MAC:srcmac}\" dstmac=\"%{MAC:dstmac}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" proto=\"%{WORD:protocol}\" length=\"%{INT:length}\" tos=\"%{DATA:tos}\" prec=\"%{DATA:prec}\" ttl=\"%{INT:ttl}\" srcport=\"%{INT:srcport}\" dstport=\"%{INT:dstport}\" "}
}
grok {
  match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" fwrule=\"%{INT:fwrule}\" initf=\"%{NOTSPACE:initf}\" outitf=\"%{NOTSPACE:outitf}\" srcmac=\"%{MAC:srcmac}\" dstmac=\"%{MAC:dstmac}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" proto=\"%{WORD:protocol}\" length=\"%{INT:length}\" tos=\"%{DATA:tos}\" prec=\"%{DATA:prec}\" ttl=\"%{INT:ttl}\" srcport=\"%{INT:srcport}\" dstport=\"%{INT:dstport}\" "}
 }
grok {
  match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" fwrule=\"%{INT:fwrule}\" initf=\"%{NOTSPACE:initf}\" outitf=\"%{NOTSPACE:outitf}\" srcmac=\"%{MAC:srcmac}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" proto=\"%{WORD:protocol}\" length=\"%{INT:length}\" tos=\"%{DATA:tos}\" prec=\"%{DATA:prec}\" ttl=\"%{INT:ttl}\" srcport=\"%{INT:srcport}\" dstport=\"%{INT:dstport}\" "}
 }
grok {
  match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" fwrule=\"%{INT:fwrule}\" outitf=\"%{NOTSPACE:outitf}\" mark=\"%{DATA:mark}\" app=\"%{DATA:app}\" srcmac=\"%{MAC:srcmac}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" proto=\"%{WORD:protocol}\" length=\"%{INT:length}\" tos=\"%{DATA:tos}\" prec=\"%{DATA:prec}\" ttl=\"%{INT:ttl}\" srcport=\"%{INT:srcport}\" dstport=\"%{INT:dstport}\" "}
 }
grok {
  match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" fwrule=\"%{INT:fwrule}\" initf=\"%{NOTSPACE:initf}\" srcmac=\"%{MAC:srcmac}\" dstmac=\"%{MAC:dstmac}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" proto=\"%{WORD:protocol}\" length=\"%{INT:length}\" tos=\"%{DATA:tos}\" prec=\"%{DATA:prec}\" ttl=\"%{INT:ttl}\" srcport=\"%{INT:srcport}\" dstport=\"%{INT:dstport}\" "}
 }
grok {
  match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" fwrule=\"%{INT:fwrule}\" outitf=\"%{NOTSPACE:outitf}\" srcmac=\"%{MAC:srcmac}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" proto=\"%{WORD:protocol}\" length=\"%{INT:length}\" tos=\"%{DATA:tos}\" prec=\"%{DATA:prec}\" ttl=\"%{INT:ttl}\" srcport=\"%{INT:srcport}\" dstport=\"%{INT:dstport}\" "}
 }
grok {
  match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" fwrule=\"%{INT:fwrule}\" initf=\"%{NOTSPACE:initf}\" outitf=\"%{NOTSPACE:outitf}\" srcmac=\"%{MAC:srcmac}\" dstmac=\"%{MAC:dstmac}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" proto=\"%{WORD:protocol}\" length=\"%{INT:length}\" tos=\"%{DATA:tos}\" prec=\"%{DATA:prec}\" ttl=\"%{INT:ttl}\" srcport=\"%{INT:srcport}\" dstport=\"%{INT:dstport}\" tcpflags=\"%{DATA:tcpflags}\" \"(,)\" "}
}
grok {
 match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" fwrule=\"%{INT:fwrule}\" initf=\"%{NOTSPACE:initf}\" srcmac=\"%{MAC:srcmac}\" dstmac=\"%{MAC:dstmac}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" proto=\"%{WORD:protocol}\" length=\"%{INT:length}\" tos=\"%{DATA:tos}\" prec=\"%{DATA:prec}\" ttl=\"%{INT:ttl}\" srcport=\"%{INT:srcport}\" dstport=\"%{INT:dstport}\" tcpflags=\"%{DATA:tcpflags}\" "}
}
grok {
 match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" fwrule=\"%{INT:fwrule}\" outitf=\"%{NOTSPACE:outitf}\" srcmac=\"%{MAC:srcmac}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" proto=\"%{WORD:protocol}\" length=\"%{INT:length}\" tos=\"%{DATA:tos}\" prec=\"%{DATA:prec}\" ttl=\"%{INT:ttl}\" srcport=\"%{INT:srcport}\" dstport=\"%{INT:dstport}\" tcpflags=\"%{DATA:tcpflags}\" "}
}
grok {
 match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" fwrule=\"%{INT:fwrule}\" initf=\"%{NOTSPACE:initf}\" outitf=\"%{NOTSPACE:outitf}\" mark=\"%{WORD:mark}\" app=\"%{WORD:app}\" srcmac=\"%{MAC:srcmac}\" dstmac=\"%{MAC:dstmac}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" proto=\"%{WORD:protocol}\" length=\"%{INT:length}\" tos=\"%{DATA:tos}\" prec=\"%{DATA:prec}\" ttl=\"%{INT:ttl}\" srcport=\"%{INT:srcport}\" dstport=\"%{INT:dstport}\" tcpflags=\"%{DATA:tcpflags}\" "}
}
grok {
 match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" fwrule=\"%{INT:fwrule}\" initf=\"%{NOTSPACE:initf}\" outitf=\"%{NOTSPACE:outitf}\" mark=\"%{WORD:mark}\" srcmac=\"%{MAC:srcmac}\" dstmac=\"%{MAC:dstmac}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" proto=\"%{WORD:protocol}\" length=\"%{INT:length}\" tos=\"%{DATA:tos}\" prec=\"%{DATA:prec}\" ttl=\"%{INT:ttl}\" srcport=\"%{INT:srcport}\" dstport=\"%{INT:dstport}\" tcpflags=\"%{DATA:tcpflags}\" "}
}
}
if "httpproxy" in [message]{
grok {
match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" method=\"%{WORD:method}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" user=\"%{DATA:user}\" group=\"%{DATA:group}\" ad_domain=\"%{DATA:ad_domain}\" statuscode=\"%{INT:statuscode}\" cached=\"%{INT:cached}\" profile=\"%{DATA:profile}\" filteraction=\"%{DATA:filteraction}\" size=\"%{INT:size}\" request=\"%{BASE16FLOAT:request}\" url=\"%{URI:url}\" referer=\"%{DATA:referer}\" error=\"%{DATA:error}\" authtime=\"%{INT:authtime}\" dnstime=\"%{INT:dnstime}\" cattime=\"%{INT:cattime}\" avscantime=\"%{INT:avscantime}\" fullreqtime=\"%{INT:fullreqtime}\" device=\"%{INT:device}\" auth=\"%{INT:auth}\" ua=\"%{DATA:ua}\" exceptions=\"%{DATA:exceptions}\" category=\"%{INT:category}\" reputation=\"%{WORD:reputation}\" categoryname=\"%{DATA:categoryname}\" country=\"%{DATA:country}\" application=\"%{WORD:application}\" app-id=\"%{INT:app-id}\" sandbox=\"%{DATA:sandbox}\" content-type=\"%{DATA:content-type}\" "}
}
grok {
match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" method=\"%{WORD:method}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" user=\"%{DATA:user}\" group=\"%{DATA:group}\" ad_domain=\"%{DATA:ad_domain}\" statuscode=\"%{INT:statuscode}\" cached=\"%{INT:cached}\" profile=\"%{DATA:profile}\" filteraction=\"%{DATA:filteraction}\" size=\"%{INT:size}\" request=\"%{BASE16FLOAT:request}\" url=\"%{URI:url}\" referer=\"%{DATA:referer}\" error=\"%{DATA:error}\" authtime=\"%{INT:authtime}\" dnstime=\"%{INT:dnstime}\" cattime=\"%{INT:cattime}\" avscantime=\"%{INT:avscantime}\" fullreqtime=\"%{INT:fullreqtime}\" device=\"%{INT:device}\" auth=\"%{INT:auth}\" ua=\"%{DATA:ua}\" exceptions=\"%{DATA:exceptions}\" overridecategory=\"%{INT:overridecategory}\" overridereputation=\"%{INT:overridereputation}\" category=\"%{INT:category}\" reputation=\"%{WORD:reputation}\" categoryname=\"%{DATA:categoryname}\" country=\"%{DATA:country}\" sandbox=\"%{DATA:sandbox}\" content-type=\"%{DATA:content-type}\" "}
}
grok {
 match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" method=\"%{WORD:method}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" user=\"%{DATA:user}\" group=\"%{DATA:group}\" ad_domain=\"%{DATA:ad_domain}\" statuscode=\"%{INT:statuscode}\" cached=\"%{INT:cached}\" profile=\"%{DATA:profile}\" filteraction=\"%{DATA:filteraction}\" size=\"%{INT:size}\" request=\"%{BASE16FLOAT:request}\" url=\"%{URI:url}\" referer=\"%{DATA:referer}\" error=\"%{DATA:error}\" authtime=\"%{INT:authtime}\" dnstime=\"%{INT:dnstime}\" cattime=\"%{INT:cattime}\" avscantime=\"%{INT:avscantime}\" fullreqtime=\"%{INT:fullreqtime}\" device=\"%{INT:device}\" auth=\"%{INT:auth}\" ua=\"%{DATA:ua}\" exceptions=\"%{DATA:exceptions}\" category=\"%{INT:category}\" reputation=\"%{WORD:reputation}\" categoryname=\"%{DATA:categoryname}\" sandbox=\"%{DATA:sandbox}\" content-type=\"%{DATA:content-type}\" "}
}
grok {
match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" method=\"%{WORD:method}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" user=\"%{DATA:user}\" group=\"%{DATA:group}\" ad_domain=\"%{DATA:ad_domain}\" statuscode=\"%{INT:statuscode}\" cached=\"%{INT:cached}\" profile=\"%{DATA:profile}\" filteraction=\"%{DATA:filteraction}\" size=\"%{INT:size}\" request=\"%{BASE16FLOAT:request}\" url=\"%{URI:url}\" referer=\"%{DATA:referer}\" error=\"%{DATA:error}\" authtime=\"%{INT:authtime}\" dnstime=\"%{INT:dnstime}\" cattime=\"%{INT:cattime}\" avscantime=\"%{INT:avscantime}\" fullreqtime=\"%{INT:fullreqtime}\" device=\"%{INT:device}\" auth=\"%{INT:auth}\" ua=\"%{DATA:ua}\" exceptions=\"%{DATA:exceptions}\" "}
}
grok {
match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" method=\"%{WORD:method}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" user=\"%{DATA:user}\" group=\"%{DATA:group}\" ad_domain=\"%{DATA:ad_domain}\" statuscode=\"%{INT:statuscode}\" cached=\"%{INT:cached}\" profile=\"%{DATA:profile}\" filteraction=\"%{DATA:filteraction}\" size=\"%{INT:size}\" request=\"%{BASE16FLOAT:request}\" url=\"%{URI:url}\" referer=\"%{DATA:referer}\" error=\"%{DATA:error}\" authtime=\"%{INT:authtime}\" dnstime=\"%{INT:dnstime}\" cattime=\"%{INT:cattime}\" avscantime=\"%{INT:avscantime}\" fullreqtime=\"%{INT:fullreqtime}\" device=\"%{INT:device}\" auth=\"%{INT:auth}\" ua=\"%{DATA:ua}\" exceptions=\"%{DATA:exceptions}\" overridecategory=\"%{INT:overridecategory}\" overridereputation=\"%{INT:overridereputation}\" country=\"%{DATA:country}\" "}
}
grok {
match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" method=\"%{WORD:method}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" user=\"%{DATA:user}\" group=\"%{DATA:group}\" ad_domain=\"%{DATA:ad_domain}\" statuscode=\"%{INT:statuscode}\" cached=\"%{INT:cached}\" profile=\"%{DATA:profile}\" filteraction=\"%{DATA:filteraction}\" size=\"%{INT:size}\" request=\"%{BASE16FLOAT:request}\" url=\"%{URI:url}\" referer=\"%{DATA:referer}\" error=\"%{DATA:error}\" authtime=\"%{INT:authtime}\" dnstime=\"%{INT:dnstime}\" cattime=\"%{INT:cattime}\" avscantime=\"%{INT:avscantime}\" fullreqtime=\"%{INT:fullreqtime}\" device=\"%{INT:device}\" auth=\"%{INT:auth}\" ua=\"%{DATA:ua}\" exceptions=\"%{DATA:exceptions}\" overridecategory=\"%{INT:overridecategory}\" overridereputation=\"%{INT:overridereputation}\" category=\"%{INT:category}\" reputation=\"%{WORD:reputation}\" categoryname=\"%{DATA:categoryname}\" country=\"%{DATA:country}\" "}
}
grok {
match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" method=\"%{WORD:method}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" user=\"%{DATA:user}\" group=\"%{DATA:group}\" ad_domain=\"%{DATA:ad_domain}\" statuscode=\"%{INT:statuscode}\" cached=\"%{INT:cached}\" profile=\"%{DATA:profile}\" filteraction=\"%{DATA:filteraction}\" size=\"%{INT:size}\" request=\"%{BASE16FLOAT:request}\" url=\"%{URI:url}\" referer=\"%{DATA:referer}\" error=\"%{DATA:error}\" authtime=\"%{INT:authtime}\" dnstime=\"%{INT:dnstime}\" cattime=\"%{INT:cattime}\" avscantime=\"%{INT:avscantime}\" fullreqtime=\"%{INT:fullreqtime}\" device=\"%{INT:device}\" auth=\"%{INT:auth}\" ua=\"%{DATA:ua}\" exceptions=\"%{DATA:exceptions}\" category=\"%{INT:category}\" reputation=\"%{WORD:reputation}\" categoryname=\"%{DATA:categoryname}\" country=\"%{DATA:country}\" "}
}
grok {
match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: id=\"%{INT:id}\" severity=\"%{WORD:severity}\" sys=\"%{WORD:sys}\" sub=\"%{WORD:sub}\" name=\"%{DATA:name}\" action=\"%{DATA:action}\" method=\"%{WORD:method}\" srcip=\"%{IPV4:source_ip}\" dstip=\"%{IPV4:destination_ip}\" user=\"%{DATA:user}\" group=\"%{DATA:group}\" ad_domain=\"%{DATA:ad_domain}\" statuscode=\"%{INT:statuscode}\" cached=\"%{INT:cached}\" profile=\"%{DATA:profile}\" filteraction=\"%{DATA:filteraction}\" size=\"%{INT:size}\" request=\"%{BASE16FLOAT:request}\" url=\"%{URI:url}\" referer=\"%{DATA:referer}\" error=\"%{DATA:error}\" authtime=\"%{INT:authtime}\" dnstime=\"%{INT:dnstime}\" cattime=\"%{INT:cattime}\" avscantime=\"%{INT:avscantime}\" fullreqtime=\"%{INT:fullreqtime}\" device=\"%{INT:device}\" auth=\"%{INT:auth}\" ua=\"%{DATA:ua}\" exceptions=\"%{DATA:exceptions}\" category=\"%{INT:category}\" reputation=\"%{WORD:reputation}\" categoryname=\"%{DATA:categoryname}\" country=\"%{DATA:country}\" application=\"%{WORD:application}\" app-id=\"%{INT:app-id}\" "}
}
}
}

As you can see I spent sometime trying to account for each variation of log entries the firewall will produce. For the most part the data is the same but after a certain point it changes. When logstash parse the data it produce multiple data for one field. Here is a pic of the duplicate fields screenshot from ELK

My assumption is that my grok patterns are too similar. I originally tried break_on_match => false but that produced the same duplicates.

Body:
Like @Phonolog said, since you have multiple grok filter, your message will be matched once with each filter. And if more than one filter successfully match its pattern against your message, you'll have duplicates of the field.

Instead of trying to match each possible combination of key-value pair at the end of your message, you can put the part with the key-value pairs in one field and then use the kv filter on it.

It would look like this:

grok {
  match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} (?<timestamp>%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}\[%{INT:pid}]\: %{GREEDYDATA:kvdata}"}
}

kv {
    source => "kvdata"
    trim_value => "\""
 }
}


来源: stackexchange

问:
How to change provisioning profile for existing app?
I have never developed apps for iPhone, but our client has an app, which was developed by his previous vendor, now that vendor has wrapped up his operation from our country, So now we have that source code for that app for further development. The immediate action we need to take is that the app is not running on iOS 11.0 and we need to provide build for 11.0.

So I opened the app in Xcode and set deployment target to iOS 11, the app build was successful and it is running on the simulator, So I thought lets check it on real device, but when I connected a device and asked Xcode to build and deploy on my device I received the error.

No profile for team '***********' matching 'PROTEST' found:  Xcode couldn't find any provisioning profiles matching '***********/PROTEST'. Install the profile (by dragging and dropping it onto Xcode's dock item) or select a different one in the General tab of the target editor.

Code signing is required for product type 'Application' in SDK 'iOS 11.1'

So as I have understood I need to create provisioning profile for app development for iOS.

Can someone please guide me, I need to know, how to create a provisioning profile and how should I use for existing project which was developed by someone else before.

NOTE: the *********** represent the key or provisioning profile ID, I am not really sure but I thought I should censor it.

答: If you have credential for apple developer account, and app is signed from that account's certificate and profile then, you can easily generate new certificate with private key from you mac and edit provisional profile. All you need to do is add client's account in xcode's accounts section. In-case, you don't have access to the developer account, which is used to make app live, you can't upload or update that app with same APP-ID using other apple developer account. You need to change APP-ID at least.</p>

ContentLicense: CC BY-SA 3.0


问:
jquery MonthYearDropdown
I want show month and year in one dropdown on jQuery datetimepicker above the month/year navbar. Is there any way to do this?</p>

答:
To show month and year you can use default options like this:

```<code>$( "#datepicker" ).datepicker({changeMonth: true,changeYear: true});
</code>```

but if you mean you want to show month and year in single dropdown (like jan-2011, feb-2011) then you will need to hack it and add a new dropdown and fill it will all allowed months-years. You will probably want to set minDate and maxDate values.

ContentLicense: CC BY-SA 2.5

Title: jquery MonthYearDropdown

Body:
To show month and year you can use default options like this:

$( "#datepicker" ).datepicker({changeMonth: true,changeYear: true});

but if you mean you want to show month and year in single dropdown (like jan-2011, feb-2011) then you will need to hack it and add a new dropdown and fill it will all allowed months-years. You will probably want to set minDate and maxDate values.


问:
Simulate click external domain
I try simulate click to share on G+ from my web site :

```<code><!doctype html>
<html>
    <head>
        <title>Teste share G+</title>
    </head>
    <body onload="alert('hiii');document.getElementById('share_g').click();">
        <a href="https://plus.google.com/share?url=http://www.example.com" id="share_g" onclick="javascript:window.open(this.href,
  '', 'menubar=no,toolbar=no,resizable=yes,scrollbars=yes,height=600,width=600');document.querySelector('span.RveJvd snByac').click(); return false;"><img
  src="https://www.gstatic.com/images/icons/gplus-64.png" alt="Share on Google+"/></a>

    </body>
</html>
</code>```

At this point work good, but i need click to POST, is possible simulate this click with javascript or pahntomjs ?

答: Have a look at the proper documentation Here, There is no need to post from javascript</p>

Title: Simulate click external domain

Body:
I try simulate click to share on G+ from my web site :

<!doctype html>

        <title>Teste share G+</title>

At this point work good, but i need click to POST, is possible simulate this click with javascript or pahntomjs ?


问:
How to implement on scrolldown pagination in Angular 4
I am new in angular 4. I want to implement scroll pagination in angular 4. Initially I want to show 20 records.
After scroll down I want to show next 20. I will do the same till end of list.

I tried to implement it using "angular2-infinite-scroll". but I am not able to show initially first 20 records as well scroll data.

component file :

```<code> import { Component, OnInit } from '@angular/core';
 import { InfiniteScrollModule } from 'angular2-infinite-scroll';

    @Component({
      selector: 'app-scroll',
      templateUrl: `./scroll.component.html`,
      styleUrls: ['./scroll.component.css']
    })
    export class ScrollComponent implements OnInit {

    let item = [{
        "Name": "XYz Compnay"
    },
    {
        "Name": "XYz Company1"
    }, {
        "Name": "XYz Company2"
    }, {
        "Name": "XYz Company3"
    }, {
        "Name": "XYz Company4"
    }, {
        "Name": "XYz Company5"
    }];
      constructor() {}
      ngOnInit() {}

      onScroll () {
        alert('scrolled!!');
      }

     }
</code>```

HTML file :

```<code><div
         infinite-scroll
         [infiniteScrollDistance]="2"
         [infiniteScrollThrottle]="10"
         (scrolled)="onScroll()"
         >
      <p *ngFor="let items of item">
        {{items.Name}}
      </p>
    </div>
</code>```

If anyone having about it please share same.

答:
I recommend using ngx-infinite-scroll against angular2-infinite-scroll due to new features and compatibility
with AOT.

First you need to specify the scrollWindow property if you are not using the whole window and using overflow: scroll
property to emulate a scrollable div. Also in your ScrollComponent class you need to have item as a property of class
and not a variable, so you should use public item instead of let item.

If the size of list is already known than you can leverage infiniteScrollDisabled to improve performance.

Also as it is grammatically incorrect to name multiple things as item and call single thing an item. I am going
to change item to items ( You can see that in the ngFor loop in template html )

```<code>@Component({
    selector: 'my-app',
    styles: [`
    .search-results {
        height: 100px;
        overflow: scroll;
    }
    `],
    template: `
    <div class="search-results"
    infiniteScroll
    [infiniteScrollDistance]="2"
    [infiniteScrollThrottle]="50"
    (scrolled)="onScroll()"
    [infiniteScrollDisabled]="isFullListDisplayed"
    [scrollWindow]="false">
        <p *ngFor="let item of itemsToShow; let i = index">
        {{i + ' ' + item.Name}}
        </p>
    </div>
    `
})
export class AppComponent {
    private noOfItemsToShowInitially: number = 5;
    // itemsToLoad - number of new items to be displayed
    private itemsToLoad: number = 5;
    // 18 items loaded for demo purposes
    private items = [
        {
            "Name": "XYz Company0"
        },
        {
            "Name": "XYz Company1"
        },
        {
            "Name": "XYz Company2"
        },
        {
            "Name": "XYz Company3"
        },
        {
            "Name": "XYz Company4"
        },
        {
            "Name": "XYz Company5"
        },
        {
            "Name": "XYz Company6"
        },
        {
            "Name": "XYz Company7"
        },
        {
            "Name": "XYz Company8"
        },
        {
            "Name": "XYz Company9"
        },
        {
            "Name": "XYz Company10"
        },
        {
            "Name": "XYz Company11"
        },
        {
            "Name": "XYz Company12"
        },
        {
            "Name": "XYz Company13"
        },
        {
            "Name": "XYz Company14"
        },
        {
            "Name": "XYz Company15"
        },
        {
            "Name": "XYz Company16"
        },
        {
            "Name": "XYz Company17"
        }
    ];
    // List that is going to be actually displayed to user
    public itemsToShow = this.items.slice(0, this.noOfItemsToShowInitially);
    // No need to call onScroll if full list has already been displayed
    public isFullListDisplayed: boolean = false;

    onScroll() {
        if (this.noOfItemsToShowInitially <= this.items.length) {
            // Update ending position to select more items from the array
            this.noOfItemsToShowInitially += this.itemsToLoad;
            this.itemsToShow = this.items.slice(0, this.noOfItemsToShowInitially);
            console.log("scrolled");
        } else {
            this.isFullListDisplayed = true;
        }
    }
}
</code>```

You will now successfully see an alert message when you scroll down the list.

Here is a working plunker of above code.

Body:
I am new in angular 4. I want to implement scroll pagination in angular 4. Initially I want to show 20 records.
After scroll down I want to show next 20. I will do the same till end of list.

I tried to implement it using "angular2-infinite-scroll". but I am not able to show initially first 20 records as well scroll data.

component file :

 import { Component, OnInit } from '@angular/core';
 import { InfiniteScrollModule } from 'angular2-infinite-scroll';

    @Component({
      selector: 'app-scroll',
      templateUrl: `./scroll.component.html`,
      styleUrls: ['./scroll.component.css']
    })
    export class ScrollComponent implements OnInit {

    let item = [{
        "Name": "XYz Compnay"
    },
    {
        "Name": "XYz Company1"
    }, {
        "Name": "XYz Company2"
    }, {
        "Name": "XYz Company3"
    }, {
        "Name": "XYz Company4"
    }, {
        "Name": "XYz Company5"
    }];
      constructor() {}
      ngOnInit() {}

      onScroll () {
        alert('scrolled!!');
      }

     }

HTML file :

        {{items.Name}}

If anyone having about it please share same.

Body:
I recommend using ngx-infinite-scroll against angular2-infinite-scroll due to new features and compatibility
with AOT.

First you need to specify the scrollWindow property if you are not using the whole window and using overflow: scroll
property to emulate a scrollable div. Also in your ScrollComponent class you need to have item as a property of class
and not a variable, so you should use public item instead of let item.

If the size of list is already known than you can leverage infiniteScrollDisabled to improve performance.

Also as it is grammatically incorrect to name multiple things as item and call single thing an item. I am going
to change item to items ( You can see that in the ngFor loop in template html )

@Component({
    selector: 'my-app',
    styles: [`
    .search-results {
        height: 100px;
        overflow: scroll;
    }
    `],
    template: `
    <div class="search-results"
    infiniteScroll
    [infiniteScrollDistance]="2"
    [infiniteScrollThrottle]="50"
    (scrolled)="onScroll()"
    [infiniteScrollDisabled]="isFullListDisplayed"
    [scrollWindow]="false">
        <p *ngFor="let item of itemsToShow; let i = index">
        {{i + ' ' + item.Name}}
        </p>
    </div>
    `
})
export class AppComponent {
    private noOfItemsToShowInitially: number = 5;
    // itemsToLoad - number of new items to be displayed
    private itemsToLoad: number = 5;
    // 18 items loaded for demo purposes
    private items = [
        {
            "Name": "XYz Company0"
        },
        {
            "Name": "XYz Company1"
        },
        {
            "Name": "XYz Company2"
        },
        {
            "Name": "XYz Company3"
        },
        {
            "Name": "XYz Company4"
        },
        {
            "Name": "XYz Company5"
        },
        {
            "Name": "XYz Company6"
        },
        {
            "Name": "XYz Company7"
        },
        {
            "Name": "XYz Company8"
        },
        {
            "Name": "XYz Company9"
        },
        {
            "Name": "XYz Company10"
        },
        {
            "Name": "XYz Company11"
        },
        {
            "Name": "XYz Company12"
        },
        {
            "Name": "XYz Company13"
        },
        {
            "Name": "XYz Company14"
        },
        {
            "Name": "XYz Company15"
        },
        {
            "Name": "XYz Company16"
        },
        {
            "Name": "XYz Company17"
        }
    ];
    // List that is going to be actually displayed to user
    public itemsToShow = this.items.slice(0, this.noOfItemsToShowInitially);
    // No need to call onScroll if full list has already been displayed
    public isFullListDisplayed: boolean = false;

    onScroll() {
        if (this.noOfItemsToShowInitially <= this.items.length) {
            // Update ending position to select more items from the array
            this.noOfItemsToShowInitially += this.itemsToLoad;
            this.itemsToShow = this.items.slice(0, this.noOfItemsToShowInitially);
            console.log("scrolled");
        } else {
            this.isFullListDisplayed = true;
        }
    }
}

You will now successfully see an alert message when you scroll down the list.

Here is a working plunker of above code.


问:
How to replace relective method access by Java 1.8 functions?
Problem: We need to get a (String) key for different classes of objects.
For expendability we want to configure the Method to use to get the key String – instead of implementing many if-else with intanceOf…

Naive solution (with example data) is:

```<code>public static String getKey(Object object, Map<Class<?>, Method> keySources) {
    Method source = keySources.get(object.getClass());

    if (source == null) {
        return null;
    }

    try {
        return (String) source.invoke(object);
    } catch (IllegalAccessException | IllegalArgumentException | InvocationTargetException e) {
        throw new RuntimeException("Error at 'invoke': " + e.getMessage(), e);
    }
}

public static void main(String[] args) {
    Map<Class<?>, Method> keySources = new HashMap<>();

    try {
        keySources.put(String.class, String.class.getMethod("toString"));
        keySources.put(Thread.class, Thread.class.getMethod("getName"));
    } catch (NoSuchMethodException | SecurityException e) {
        throw new RuntimeException("Error at 'getMethod': " + e.getMessage(), e);
    }

    System.out.println(getKey("test", keySources));
    System.out.println(getKey(new Thread("name"), keySources));
}
</code>```

Desired solution would be like:

```<code>public static String getKey(Object object, Map<Class<?>, Function<Object, String>> keySources) {
    Function<Object, String> source = keySources.get(object.getClass());

    if (source == null) {
        return null;
    }

    return source.apply(object);
}

public static void main(String[] args) {
    Map<Class<?>, Function<Object, String>> keySources = new HashMap<>();

    keySources.put(String.class, String::toString);
    keySources.put(Thread.class, Thread::getName);

    System.out.println(getKey("test", keySources));
    System.out.println(getKey(new Thread("name"), keySources));
}
</code>```

But String::toString is giving compilation error: The type String does not define toString(Object) that is applicable here

Constraints: We cannot modify the classes since they were generated.

答:
A Function<Object, String> is a function that accepts Object, in other words arbitrary objects as argument, so a function like String::toString, that requires its arguments to be String instances can’t fulfill the contract. That’s easy to fix, as you can use Object::toString instead, however, for Thread::getName, which requires the arguments to be Thread instances, there is no such replacement.

Since you are ensuring that the arguments are of the right type due to the map keys, you can solve this by converting each specific function to a Function<Object,String> that does a type cast:

```<code>public static <T,R> void put(Class<T> cl,
                             Function<T,R> f, Map<Class<?>,Function<Object,R>> map) {
    map.put(cl, obj -> f.apply(cl.cast(obj)));
}
public static String getKey(Object object,
                            Map<Class<?>, Function<Object, String>> keySources) {
    return keySources.getOrDefault(object.getClass(), x -> null).apply(object);
}
public static void main(String[] args) {
    Map<Class<?>, Function<Object, String>> keySources = new HashMap<>();

    put(String.class, String::toString, keySources);
    // or put(String.class, Function.identity(), keySources);
    put(Thread.class, Thread::getName, keySources);

    System.out.println(getKey("test", keySources));
    System.out.println(getKey(new Thread("name"), keySources));
}
</code>```

Body:
Problem: We need to get a (String) key for different classes of objects.
For expendability we want to configure the Method to use to get the key String – instead of implementing many if-else with intanceOf…

Naive solution (with example data) is:

public static String getKey(Object object, Map<Class<?>, Method> keySources) {
    Method source = keySources.get(object.getClass());

    if (source == null) {
        return null;
    }

    try {
        return (String) source.invoke(object);
    } catch (IllegalAccessException | IllegalArgumentException | InvocationTargetException e) {
        throw new RuntimeException("Error at 'invoke': " + e.getMessage(), e);
    }
}

public static void main(String[] args) {
    Map<Class<?>, Method> keySources = new HashMap<>();

    try {
        keySources.put(String.class, String.class.getMethod("toString"));
        keySources.put(Thread.class, Thread.class.getMethod("getName"));
    } catch (NoSuchMethodException | SecurityException e) {
        throw new RuntimeException("Error at 'getMethod': " + e.getMessage(), e);
    }

    System.out.println(getKey("test", keySources));
    System.out.println(getKey(new Thread("name"), keySources));
}

Desired solution would be like:

public static String getKey(Object object, Map<Class<?>, Function<Object, String>> keySources) {
    Function<Object, String> source = keySources.get(object.getClass());

    if (source == null) {
        return null;
    }

    return source.apply(object);
}

public static void main(String[] args) {
    Map<Class<?>, Function<Object, String>> keySources = new HashMap<>();

    keySources.put(String.class, String::toString);
    keySources.put(Thread.class, Thread::getName);

    System.out.println(getKey("test", keySources));
    System.out.println(getKey(new Thread("name"), keySources));
}

But String::toString is giving compilation error: The type String does not define toString(Object) that is applicable here

Constraints: We cannot modify the classes since they were generated.

Body:
A Function<Object, String> is a function that accepts Object, in other words arbitrary objects as argument, so a function like String::toString, that requires its arguments to be String instances can’t fulfill the contract. That’s easy to fix, as you can use Object::toString instead, however, for Thread::getName, which requires the arguments to be Thread instances, there is no such replacement.

Since you are ensuring that the arguments are of the right type due to the map keys, you can solve this by converting each specific function to a Function<Object,String> that does a type cast:

public static <T,R> void put(Class<T> cl,
                             Function<T,R> f, Map<Class<?>,Function<Object,R>> map) {
    map.put(cl, obj -> f.apply(cl.cast(obj)));
}
public static String getKey(Object object,
                            Map<Class<?>, Function<Object, String>> keySources) {
    return keySources.getOrDefault(object.getClass(), x -> null).apply(object);
}
public static void main(String[] args) {
    Map<Class<?>, Function<Object, String>> keySources = new HashMap<>();

    put(String.class, String::toString, keySources);
    // or put(String.class, Function.identity(), keySources);
    put(Thread.class, Thread::getName, keySources);

    System.out.println(getKey("test", keySources));
    System.out.println(getKey(new Thread("name"), keySources));
}


问:
Assign jQuery datepicker to div element and get the value
I want to show a datepicker using jQuery when user click on some text, and after the user select the date on datepicker, I can get the value from the date picker as javascript variable. My code is like this:

```<code><div id="datepicker-container" style="display: none;">
    <div id="select-delivery-date-input"> </div>
</div>
<a id="show-datepicker">Select Delivery Date</a>
<script>
  $("#show-datepicker").click(function(){
      $("#datepicker-container").show();
  });
  $('#select-delivery-date-input').datepicker({
      dateFormat:'yy-m-d',
      minDate: new Date(),
  });
</script>
</code>```

The problem is when the datepicker popup, when I tried to select the date on datepicker popup it won't close the datepicker popup.

Fiddle example

答:
You can use datepicker's onSelect option:

Called when the datepicker is selected. The function receives the selected date as text and the datepicker instance as parameters.

Then you can use jQuery hide().

Here a live sample:

$(document).ready(function() {
  $("#show-datepicker").click(function(){
    $("#datepicker-container").show();
  });
  $('#select-delivery-date-input').datepicker({
    dateFormat:'yy-m-d',
    minDate: new Date(),
    onSelect: function(selectedDate){
      console.log(selectedDate);
      $("#datepicker-container").hide();
    }
  });
});```
<pre class="snippet-code-html lang-html prettyprint-override"><code><script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/jqueryui/1.11.4/jquery-ui.js"></script>
<link href="https://ajax.googleapis.com/ajax/libs/jqueryui/1.11.4/themes/smoothness/jquery-ui.css" rel="stylesheet"/>

<div id="datepicker-container" style="display: none;">
  <div id="select-delivery-date-input"> </div>
</div>
Select Delivery Date</code>```

Body:
I want to show a datepicker using jQuery when user click on some text, and after the user select the date on datepicker, I can get the value from the date picker as javascript variable. My code is like this:

Select Delivery Date

The problem is when the datepicker popup, when I tried to select the date on datepicker popup it won't close the datepicker popup.

Fiddle example

Body:
You can use datepicker's onSelect option:

Called when the datepicker is selected. The function receives the selected date as text and the datepicker instance as parameters.

Then you can use jQuery hide().

Here a live sample:

$(document).ready(function() {
  $("#show-datepicker").click(function(){
    $("#datepicker-container").show();
  });
  $('#select-delivery-date-input').datepicker({
    dateFormat:'yy-m-d',
    minDate: new Date(),
    onSelect: function(selectedDate){
      console.log(selectedDate);
      $("#datepicker-container").hide();
    }
  });
});

<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/jqueryui/1.11.4/jquery-ui.js"></script>
<link href="https://ajax.googleapis.com/ajax/libs/jqueryui/1.11.4/themes/smoothness/jquery-ui.css" rel="stylesheet"/>

<div id="datepicker-container" style="display: none;">
  <div id="select-delivery-date-input"> </div>
</div>
<a id="show-datepicker">Select Delivery Date</a>


问:
Need to show/hide a dropdown after checking/unchecking a checkbox in angular 4
I have a specific requirement in Angular4.
I need to display a dropdown after selecting/checking a checkbox and if I uncheck the checkbox then dropdown will hide.
Below is the code of the checkbox.

```<code>    <div *ngIf="item.showOperationField">
        <p-checkbox
            value ="inflationaryImpactCheck"
            label="Save for Inflationary Impact"
            name = "calculationSaveInflation"
            [(ngModel)]="item.inflationaryImpact"
            pTooltip="Check the Inflationary Impact box to map this calculation to an index for Inflationary Pressure calculation.">
        </p-checkbox>
    </div>
</code>```

After clicking/unclicking the checkbox

```<code>    <div *ngIf="item.showOperationField">
        <p-dropdown
            [style]="{'width':'200px'}"
            [options]="inflation"
            name = "calculationInflation"
            [(ngModel)]="item.selectedInflation">
        </p-dropdown>
    </div>
</code>```

Please help me and let me know how to do this

答:
you can add click event on checkbox element, you check in your component if checked you set "show dropdown" to true:

```<code>    <p-checkbox
        value ="inflationaryImpactCheck"
        label="Save for Inflationary Impact"
        name = "calculationSaveInflation"
        [(ngModel)]="item.inflationaryImpact"
        (change)="showHideDropDown($event)" /* add this */
        pTooltip="Check the Inflationary Impact box to map this calculation to an index for Inflationary Pressure calculation.">
    </p-checkbox>
</code>```

and in your component.ts

```<code>showHideDropDown(event){
 if(event.target.checked){
  this.showDropDown = true;
} else {
   this.showDropDown = false;
}
</code>```

in your viw you show of hide based on this variable

```<code><div *ngIf="item.showOperationField && showDropDown ">
    <p-dropdown
        [style]="{'width':'200px'}"
        [options]="inflation"
        name = "calculationInflation"
        [(ngModel)]="item.selectedInflation">
    </p-dropdown>
</div>
</code>```

Body:
I have a specific requirement in Angular4.
I need to display a dropdown after selecting/checking a checkbox and if I uncheck the checkbox then dropdown will hide.
Below is the code of the checkbox.

After clicking/unclicking the checkbox

Please help me and let me know how to do this

Body:
you can add click event on checkbox element, you check in your component if checked you set "show dropdown" to true:

    <p-checkbox
        value ="inflationaryImpactCheck"
        label="Save for Inflationary Impact"
        name = "calculationSaveInflation"
        [(ngModel)]="item.inflationaryImpact"
        (change)="showHideDropDown($event)" /* add this */
        pTooltip="Check the Inflationary Impact box to map this calculation to an index for Inflationary Pressure calculation.">
    </p-checkbox>

and in your component.ts

showHideDropDown(event){
 if(event.target.checked){
  this.showDropDown = true;
} else {
   this.showDropDown = false;
}

in your viw you show of hide based on this variable

<div *ngIf="item.showOperationField && showDropDown ">
    <p-dropdown
        [style]="{'width':'200px'}"
        [options]="inflation"
        name = "calculationInflation"
        [(ngModel)]="item.selectedInflation">
    </p-dropdown>
</div>


问:
Android API Response handling
I want to handle the json response from my API in my Android app, but i can't find a lot of tutorials on how to do this, does anyone have some tips?

Android code:

```<code>                    URL url = new URL("http://thelinktomyapi.com");
                    HttpURLConnection conn = (HttpURLConnection) url.openConnection();
                    conn.setRequestMethod("POST");
                    conn.setRequestProperty("Content-Type", "application/json;charset=UTF-8");
                    conn.setRequestProperty("Accept", "application/json");
                    conn.setDoOutput(true);
                    conn.setDoInput(true);

                    Log.i("JSON", jsonObj.toString());
                    DataOutputStream os = new DataOutputStream(conn.getOutputStream());
                    //os.writeBytes(URLEncoder.encode(jsonParam.toString(), "UTF-8"));
                    os.writeBytes(jsonObj.toString());

                    os.flush();
                    os.close();

                    Log.i("STATUS", String.valueOf(conn.getResponseCode()));
                    Log.i("MSG", conn.getResponseMessage());

                    conn.disconnect();
</code>```

Reponse part of my flask code(very simple)

```<code>respdata = {
        'imgurl2': 'google.nl'
    }
    resp = jsonify(respdata)
    resp.status_code = 200
    return resp
</code>```

答:
Try,

```<code>String response = "";
        try {
            URL url = new URL(reqUrl);
            HttpURLConnection conn = (HttpURLConnection) url.openConnection();

            BufferedReader br = new BufferedReader(new InputStreamReader((conn.getInputStream())));
            String output;
            while ((output = br.readLine()) != null) {
                response = response + output;
            }
            conn.disconnect();
        } catch (MalformedURLException e) {
            Log.e(TAG, "MalformedURLException: " + e.getMessage());
        } catch (ProtocolException e) {
            Log.e(TAG, "ProtocolException: " + e.getMessage());
        } catch (IOException e) {
            Log.e(TAG, "IOException: " + e.getMessage());
        } catch (Exception e) {
            Log.e(TAG, "Exception: " + e.getMessage());
        }
</code>```

You will get your response in response variable.

Title: Android API Response handling

Body:
I want to handle the json response from my API in my Android app, but i can't find a lot of tutorials on how to do this, does anyone have some tips?

Android code:

                    URL url = new URL("http://thelinktomyapi.com");
                    HttpURLConnection conn = (HttpURLConnection) url.openConnection();
                    conn.setRequestMethod("POST");
                    conn.setRequestProperty("Content-Type", "application/json;charset=UTF-8");
                    conn.setRequestProperty("Accept", "application/json");
                    conn.setDoOutput(true);
                    conn.setDoInput(true);

                    Log.i("JSON", jsonObj.toString());
                    DataOutputStream os = new DataOutputStream(conn.getOutputStream());
                    //os.writeBytes(URLEncoder.encode(jsonParam.toString(), "UTF-8"));
                    os.writeBytes(jsonObj.toString());

                    os.flush();
                    os.close();

                    Log.i("STATUS", String.valueOf(conn.getResponseCode()));
                    Log.i("MSG", conn.getResponseMessage());

                    conn.disconnect();

Reponse part of my flask code(very simple)

respdata = {
        'imgurl2': 'google.nl'
    }
    resp = jsonify(respdata)
    resp.status_code = 200
    return resp

Body:
Try,

String response = "";
        try {
            URL url = new URL(reqUrl);
            HttpURLConnection conn = (HttpURLConnection) url.openConnection();

            BufferedReader br = new BufferedReader(new InputStreamReader((conn.getInputStream())));
            String output;
            while ((output = br.readLine()) != null) {
                response = response + output;
            }
            conn.disconnect();
        } catch (MalformedURLException e) {
            Log.e(TAG, "MalformedURLException: " + e.getMessage());
        } catch (ProtocolException e) {
            Log.e(TAG, "ProtocolException: " + e.getMessage());
        } catch (IOException e) {
            Log.e(TAG, "IOException: " + e.getMessage());
        } catch (Exception e) {
            Log.e(TAG, "Exception: " + e.getMessage());
        }

You will get your response in response variable.


问:
Oracle SQL Developer- How to force 00:00:00 hour when inserting a new DATE value
In my Oracle SQL Developer, i have a table with a column with DATE format. When i insert a new row into this table, and insert a new value in this column, it automatically suggestes me the current date with the current hour.
I would like that it automatically suggestes me current date, but with 00:00:00 hour . Is there some setting or parameter that i can set in my SQL Developer to have this result?

答:
We can't able to insert 00:00:00 hours ... the hour value should be between 1 to 12...

we can use below query to insert 00:00:00 hours but the value will be changed to 12:00:00

INSERT INTO TABLE (DATE_COL) VALUES
( TO_DATE ('11/16/2017 00:00:00 ', 'MM/DD/YYYY HH24:MI:SS '));


问:
Error using Google Stackdriver Logging in App Engine Standard python
My Stack:

    Google App Engine Standard

    Python (2.7)

Goal:

To create named logs in Google Stackdriver Logging, https://console.cloud.google.com/logs/viewer

Docs - Stackdriver Logging:
https://google-cloud-python.readthedocs.io/en/latest/logging/usage.html

Code:

```<code>from google.cloud import logging as stack_logging
from google.cloud.logging.resource import Resource
import threading

class StackdriverLogging:
    def __init__(self, resource=Resource(type='project', labels={'project_id': 'project_id'}), project_id='project_id'):

    self.resource = resource
    self.client = stack_logging.Client(project=project_id)

    def delete_logger(self, logger_name):
        logger = self.client.logger(logger_name)
        logger.delete()

    def async_log(self, logger_name, sev, msg):
        t = threading.Thread(target=self.log, args=(logger_name, sev, msg,))
        t.start()

    def log(self, logger_name, sev, msg):
        logger = self.client.logger(logger_name)

    if isinstance(msg, str):
        logger.log_text(msg, severity=sev, resource=self.resource)
    elif isinstance(msg, dict):
        logger.log_struct(msg, severity=sev, resource=self.resource)

class hLog(webapp2.RequestHandler):
   def get(self):
      stackdriver_logger = StackdriverLogging()
      stackdriver_logger.async_log("my_new_log", "WARNING", msg="Hello")
      stackdriver_logger.async_log("my_new_log", "INFO", msg="world")
</code>```

ERROR:
Found 1 RPC request(s) without matching response

If this is not possible in Google App Engine Standard (Python) any way to get this code to work:

```<code>  from google.cloud import logging
  client = logging.Client()
  # client = logging.Client.from_service_account_json('credentials.json')
  logger = client.logger("my_new_log")
  logger.log_text("hello world")
</code>```

If credentials are required, I like to use the project service account.

Any help would be appreciated. Thank you.

答:
I usually tie the Python logging module directly into Google Stackdriver Logging.
To do this, I create a log_helper module:

```<code>from google.cloud import logging as gc_logging
import logging

logging_client = gc_logging.Client()
logging_client.setup_logging(logging.INFO)

from logging import *
</code>```

Which I then import into other files like this:

```<code>import log_helper as logging
</code>```

After which you can use the module like you would the default python logging module.

To create named logs with the default python logging module, use different loggers for different namespaces:

```<code>import log_helper as logging

test_logger = logging.getLogger('test')
test_logger.setLevel(logging.INFO)
test_logger.info('is the name of this logger')
</code>```

Output:

INFO:test:is the name of this logger

Body:
My Stack:

    Google App Engine Standard

    Python (2.7)

Goal:

To create named logs in Google Stackdriver Logging, https://console.cloud.google.com/logs/viewer

Docs - Stackdriver Logging:
https://google-cloud-python.readthedocs.io/en/latest/logging/usage.html

Code:

from google.cloud import logging as stack_logging
from google.cloud.logging.resource import Resource
import threading

class StackdriverLogging:
    def __init__(self, resource=Resource(type='project', labels={'project_id': 'project_id'}), project_id='project_id'):

    self.resource = resource
    self.client = stack_logging.Client(project=project_id)

    def delete_logger(self, logger_name):
        logger = self.client.logger(logger_name)
        logger.delete()

    def async_log(self, logger_name, sev, msg):
        t = threading.Thread(target=self.log, args=(logger_name, sev, msg,))
        t.start()

    def log(self, logger_name, sev, msg):
        logger = self.client.logger(logger_name)

    if isinstance(msg, str):
        logger.log_text(msg, severity=sev, resource=self.resource)
    elif isinstance(msg, dict):
        logger.log_struct(msg, severity=sev, resource=self.resource)

class hLog(webapp2.RequestHandler):
   def get(self):
      stackdriver_logger = StackdriverLogging()
      stackdriver_logger.async_log("my_new_log", "WARNING", msg="Hello")
      stackdriver_logger.async_log("my_new_log", "INFO", msg="world")

ERROR:
Found 1 RPC request(s) without matching response

If this is not possible in Google App Engine Standard (Python) any way to get this code to work:

  from google.cloud import logging
  client = logging.Client()
  # client = logging.Client.from_service_account_json('credentials.json')
  logger = client.logger("my_new_log")
  logger.log_text("hello world")

If credentials are required, I like to use the project service account.

Any help would be appreciated. Thank you.

Body:
I usually tie the Python logging module directly into Google Stackdriver Logging.
To do this, I create a log_helper module:

from google.cloud import logging as gc_logging
import logging

logging_client = gc_logging.Client()
logging_client.setup_logging(logging.INFO)

from logging import *

Which I then import into other files like this:

import log_helper as logging

After which you can use the module like you would the default python logging module.

To create named logs with the default python logging module, use different loggers for different namespaces:

import log_helper as logging

test_logger = logging.getLogger('test')
test_logger.setLevel(logging.INFO)
test_logger.info('is the name of this logger')

Output:

INFO:test:is the name of this logger


问:
Hadoop command not found
I have installed and configured hadoop in a linux machine .Now i am trying to run a sample MR job.I have started the hadoop via the command /usr/local/hadoop/bin/start-all.sh and the output is

```<code>namenode running as process 7876. Stop it first.
localhost: datanode running as process 8083. Stop it first.
localhost: secondarynamenode running as process 8304. Stop it first.
jobtracker running as process 8398. Stop it first.
localhost: tasktracker running as process 8612. Stop it first.
</code>```

so i think that my hadoop is configured successfully.But when i am tryinh to run below command it is giving

```<code>jeet@jeet-Vostro-2520:~$ hadoop fs -put gettysburg.txt /user/jeet/getty/gettysburg.txt
hadoop: command not found
</code>```

i am new in hadoop.somebody please help .I am also posting the screenshot of what i am trying

答:
once echo your path ,if your path hasn't set ,then go to your .bashrc file

```<code>vi ~/.bashrc
</code>```

and add folliwing in that.

```<code>export PATH=$PATH:/usr/local/hadoop/bin/
</code>```

Title: Hadoop command not found

Body:
I have installed and configured hadoop in a linux machine .Now i am trying to run a sample MR job.I have started the hadoop via the command /usr/local/hadoop/bin/start-all.sh and the output is

namenode running as process 7876. Stop it first.
localhost: datanode running as process 8083. Stop it first.
localhost: secondarynamenode running as process 8304. Stop it first.
jobtracker running as process 8398. Stop it first.
localhost: tasktracker running as process 8612. Stop it first.

so i think that my hadoop is configured successfully.But when i am tryinh to run below command it is giving

jeet@jeet-Vostro-2520:~$ hadoop fs -put gettysburg.txt /user/jeet/getty/gettysburg.txt
hadoop: command not found

i am new in hadoop.somebody please help .I am also posting the screenshot of what i am trying

Body:
once echo your path ,if your path hasn't set ,then go to your .bashrc file

vi ~/.bashrc

and add folliwing in that.

export PATH=$PATH:/usr/local/hadoop/bin/


问:
Using Windows Forms Locks up PowerShell ISE minutes after script has terminated
I have an interesting issue here.  I'm creating a calendar picker for use when we create accounts.  It works fine and is still in progress but I have noticed that when I run the script in powershell ISE, after a few minutes it locks up (I am able to edit and save the code for a few minutes prior to that). There is nothing in the event log. I get a dialog box saying that powershell is non responsive.  Memory usage seems normal as well.  I do not know what is happening.

This occurs no matter how I run Powershell ISE (Run as Administrator, Run as another account, and normal ISE) I am running windows 8.1.

A coworker suggested it may be the apartment model, so I've tried STA and MTA, but the problem occurs either way. It does not happen when the same code is run from the console host.

[void] [System.Reflection.Assembly]::LoadWithPartialName("System.Windows.Forms")
[void] [System.Reflection.Assembly]::LoadWithPartialName("System.Drawing")

$objForm = New-Object Windows.Forms.Form

$objForm.Text = "Select a Date"
$objForm.Size = New-Object Drawing.Size @(490,250)
$objForm.StartPosition = "CenterScreen"

$objForm.KeyPreview = $True

$objForm.Add_KeyDown({
    if ($_.KeyCode -eq "Enter")
        {
            $script:dtmDate=$objCalendar.SelectionStart
            $objForm.Close()
        }
    })

$objForm.Add_KeyDown({
    if ($_.KeyCode -eq "Escape")
        {
            $objForm.Close()
        }
    })

$objCalendar = New-Object System.Windows.Forms.MonthCalendar
$objCalendar.Text = "Start"
$objCalendar.ShowTodayCircle = $False
$objCalendar.MaxSelectionCount = 1
$objForm.Controls.Add($objCalendar)

$objForm.Topmost = $True

$objForm.Add_Shown({$objForm.Activate()})
[void] $objForm.ShowDialog()

if ($dtmDate)
    {
        Write-Host "Date selected: $dtmDate"
    }

$objForm.Dispose()
```

<hr>

<p>In Response to @The Unique Paul Smith</p>

<pre class="lang-psh prettyprint-override"><code>function Find-CalenderDateTest {
[CmdletBinding()]
param(
    [Parameter(
        Mandatory=$false
    )]
    [ValidateSet('long','short','powerpoint')]
    [ValidateNotNullOrEmpty()]
    [string]
    $DateFormat

)

Begin{
[void] [System.Reflection.Assembly]::LoadWithPartialName("System.Windows.Forms")
[void] [System.Reflection.Assembly]::LoadWithPartialName("System.Drawing")

$objForm = New-Object Windows.Forms.Form

$objForm.Text = "Select a Date"
$objForm.Size = New-Object Drawing.Size @(243,250)
$objForm.StartPosition = "CenterScreen"

$objForm.KeyPreview = $True

$dtmDate = $null

$objForm.Add_KeyDown( {
    if ($_.KeyCode -eq "Enter")
        {
            $dtmDate=$objCalendar.SelectionStart
            $objForm.Close()
        }
    })

$objForm.Add_KeyDown({
    if ($_.KeyCode -eq "Escape")
        {
            $objForm.Close()
        }
    })

#region   OK Button
$OKButton = New-Object System.Windows.Forms.Button
$OKButton.Location = New-Object System.Drawing.Size(20,175)
$OKButton.Size = New-Object System.Drawing.Size(75,23)
$OKButton.Text = "OK"

# Got rid of the Click event for OK Button, and instead just assigned its DialogResult property to OK.
$OKButton.DialogResult = [System.Windows.Forms.DialogResult]::OK

$objForm.Controls.Add($OKButton)

# Setting the form's AcceptButton property causes it to automatically intercept the Enter keystroke and
# treat it as clicking the OK button (without having to write your own KeyDown events).
$objForm.AcceptButton = $OKButton
#endregion

#region Cancel Button
$CancelButton = New-Object System.Windows.Forms.Button
$CancelButton.Location = New-Object System.Drawing.Size(80,175)
$CancelButton.Size = New-Object System.Drawing.Size(75,23)
$CancelButton.Text = "Cancel"

# Got rid of the Click event for Cancel Button, and instead just assigned its DialogResult property to Cancel.
$CancelButton.DialogResult = [System.Windows.Forms.DialogResult]::Cancel

$objForm.Controls.Add($CancelButton)

# Setting the form's CancelButton property causes it to automatically intercept the Escape keystroke and
# treat it as clicking the OK button (without having to write your own KeyDown events).
$objForm.CancelButton = $CancelButton
#endregion

$objCalendar = New-Object System.Windows.Forms.MonthCalendar
$objCalendar.ShowTodayCircle = $False
$objCalendar.MaxSelectionCount = 1
$objForm.Controls.Add($objCalendar)

$objForm.Topmost = $True

$objForm.Add_Shown({$objForm.Activate()})

$Results = $objForm.ShowDialog()
}

Process{}

End{
if ($Results -eq "OK")
    {
    $objCalendar.SelectionStart
    }

$objForm.Dispose()

}
}
</code>```

答:
The error is MTA/STA

Don't use

```<code>$form.showDialog()
</code>```

Use

```<code>[system.windows.forms.application]::run($form)
</code>```

instead

and it works fine every time

Another way is to put it in another thread:

```<code>$code
{
  //form code here
  $form.showDialog()
}
$newThread = [Powershell]::Create()
$newThread.AddScript($code)
$handle = $newThread.BeginInvoke()
</code>```

Provide variables from the calling script:

$newThread.Runspace.SessionStateProxy.SetVariable("variablenname",value)

before the BeginInvoke use variablenname without $...

Body:
I have an interesting issue here.  I'm creating a calendar picker for use when we create accounts.  It works fine and is still in progress but I have noticed that when I run the script in powershell ISE, after a few minutes it locks up (I am able to edit and save the code for a few minutes prior to that). There is nothing in the event log. I get a dialog box saying that powershell is non responsive.  Memory usage seems normal as well.  I do not know what is happening.

This occurs no matter how I run Powershell ISE (Run as Administrator, Run as another account, and normal ISE) I am running windows 8.1.

A coworker suggested it may be the apartment model, so I've tried STA and MTA, but the problem occurs either way. It does not happen when the same code is run from the console host.

[void] [System.Reflection.Assembly]::LoadWithPartialName("System.Windows.Forms")
[void] [System.Reflection.Assembly]::LoadWithPartialName("System.Drawing")

$objForm = New-Object Windows.Forms.Form

$objForm.Text = "Select a Date"
$objForm.Size = New-Object Drawing.Size @(490,250)
$objForm.StartPosition = "CenterScreen"

$objForm.KeyPreview = $True

$objForm.Add_KeyDown({
    if ($_.KeyCode -eq "Enter")
        {
            $script:dtmDate=$objCalendar.SelectionStart
            $objForm.Close()
        }
    })

$objForm.Add_KeyDown({
    if ($_.KeyCode -eq "Escape")
        {
            $objForm.Close()
        }
    })

$objCalendar = New-Object System.Windows.Forms.MonthCalendar
$objCalendar.Text = "Start"
$objCalendar.ShowTodayCircle = $False
$objCalendar.MaxSelectionCount = 1
$objForm.Controls.Add($objCalendar)

$objForm.Topmost = $True

$objForm.Add_Shown({$objForm.Activate()})
[void] $objForm.ShowDialog()

if ($dtmDate)
    {
        Write-Host "Date selected: $dtmDate"
    }

$objForm.Dispose()

In Response to @The Unique Paul Smith

function Find-CalenderDateTest {
[CmdletBinding()]
param(
    [Parameter(
        Mandatory=$false
    )]
    [ValidateSet('long','short','powerpoint')]
    [ValidateNotNullOrEmpty()]
    [string]
    $DateFormat

)

Begin{
[void] [System.Reflection.Assembly]::LoadWithPartialName("System.Windows.Forms")
[void] [System.Reflection.Assembly]::LoadWithPartialName("System.Drawing")

$objForm = New-Object Windows.Forms.Form

$objForm.Text = "Select a Date"
$objForm.Size = New-Object Drawing.Size @(243,250)
$objForm.StartPosition = "CenterScreen"

$objForm.KeyPreview = $True

$dtmDate = $null

$objForm.Add_KeyDown( {
    if ($_.KeyCode -eq "Enter")
        {
            $dtmDate=$objCalendar.SelectionStart
            $objForm.Close()
        }
    })

$objForm.Add_KeyDown({
    if ($_.KeyCode -eq "Escape")
        {
            $objForm.Close()
        }
    })

#region   OK Button
$OKButton = New-Object System.Windows.Forms.Button
$OKButton.Location = New-Object System.Drawing.Size(20,175)
$OKButton.Size = New-Object System.Drawing.Size(75,23)
$OKButton.Text = "OK"

# Got rid of the Click event for OK Button, and instead just assigned its DialogResult property to OK.
$OKButton.DialogResult = [System.Windows.Forms.DialogResult]::OK

$objForm.Controls.Add($OKButton)

# Setting the form's AcceptButton property causes it to automatically intercept the Enter keystroke and
# treat it as clicking the OK button (without having to write your own KeyDown events).
$objForm.AcceptButton = $OKButton
#endregion

#region Cancel Button
$CancelButton = New-Object System.Windows.Forms.Button
$CancelButton.Location = New-Object System.Drawing.Size(80,175)
$CancelButton.Size = New-Object System.Drawing.Size(75,23)
$CancelButton.Text = "Cancel"

# Got rid of the Click event for Cancel Button, and instead just assigned its DialogResult property to Cancel.
$CancelButton.DialogResult = [System.Windows.Forms.DialogResult]::Cancel

$objForm.Controls.Add($CancelButton)

# Setting the form's CancelButton property causes it to automatically intercept the Escape keystroke and
# treat it as clicking the OK button (without having to write your own KeyDown events).
$objForm.CancelButton = $CancelButton
#endregion

$objCalendar = New-Object System.Windows.Forms.MonthCalendar
$objCalendar.ShowTodayCircle = $False
$objCalendar.MaxSelectionCount = 1
$objForm.Controls.Add($objCalendar)

$objForm.Topmost = $True

$objForm.Add_Shown({$objForm.Activate()})

$Results = $objForm.ShowDialog()
}

Process{}

End{
if ($Results -eq "OK")
    {
    $objCalendar.SelectionStart
    }

$objForm.Dispose()

}
}

Body:
The error is MTA/STA

Don't use

$form.showDialog()

Use

[system.windows.forms.application]::run($form)

instead

and it works fine every time

Another way is to put it in another thread:

$code
{
  //form code here
  $form.showDialog()
}
$newThread = [Powershell]::Create()
$newThread.AddScript($code)
$handle = $newThread.BeginInvoke()

Provide variables from the calling script:

$newThread.Runspace.SessionStateProxy.SetVariable("variablenname",value)

before the BeginInvoke use variablenname without $...


问:
java swing jtable add two table from database in single table
I have table named  tbl_income and tbl_expenses...
tbl_income contain four column and four row and tbl_expenses contain four column and three row link in picture below.

I want to mank single table in java swing JTable link picture in below.

how can i make like this.... sorry for bad english..

This is what i've tried:

```<code> DefaultTableModel model = (DefaultTableModel) tbldailybook.getModel();
        try {
            Statement stmt = db.cn.createStatement();
            ResultSet rs = stmt.executeQuery("select * from tblincome,tblexpenses");
            while (rs.next()) {
                model.addRow(new Object[]{rs.getString(2), rs.getString(3), rs.getString(4), rs.getString(6), rs.getString(7), rs.getString(8)});
            }
        } catch (Exception e) {
        }
</code>```

tblincome have single row but row reply

Body:
I have table named  tbl_income and tbl_expenses...
tbl_income contain four column and four row and tbl_expenses contain four column and three row link in picture below.

I want to mank single table in java swing JTable link picture in below.

how can i make like this.... sorry for bad english..

This is what i've tried:

 DefaultTableModel model = (DefaultTableModel) tbldailybook.getModel();
        try {
            Statement stmt = db.cn.createStatement();
            ResultSet rs = stmt.executeQuery("select * from tblincome,tblexpenses");
            while (rs.next()) {
                model.addRow(new Object[]{rs.getString(2), rs.getString(3), rs.getString(4), rs.getString(6), rs.getString(7), rs.getString(8)});
            }
        } catch (Exception e) {
        }

tblincome have single row but row reply

Body:
+----+------------+----------------+--------+
| id | dateValue  |   particular   | amount |
+----+------------+----------------+--------+
|  1 | 2017-02-02 | Cash Deposit   |      1 |
|  2 | 2017-02-03 | Cheque Deposit |      2 |
|  3 | 2017-02-08 | Cash Deposit   |      3 |
|  4 | 2017-02-07 | Product Sales  |      4 |
+----+------------+----------------+--------+

+----+------------+------------+--------+
| id | dateValue  | particular | amount |
+----+------------+------------+--------+
|  1 | 2017-05-07 | Factory    |      6 |
|  2 | 2017-06-02 | Staff      |      7 |
|  3 | 2017-06-03 | Travel     |      8 |
+----+------------+------------+--------+

I then used select * from income left join expenses on income.id = expenses.id to query the database and got:

+----+------------+----------------+--------+----+------------+------------+--------+--+
| id | dateValue  |   particular   | amount | id | dateValue  | particular | amount |  |
+----+------------+----------------+--------+----+------------+------------+--------+--+
|  1 | 2017-02-02 | Cash Deposit   |      1 |  1 | 2017-05-07 | Factory    |      6 |  |
|  2 | 2017-02-03 | Cheque Deposit |      2 |  2 | 2017-06-02 | Staff      |      7 |  |
|  3 | 2017-02-08 | Cash Deposit   |      3 |  3 | 2017-06-03 | Travel     |      8 |  |
|  4 | 2017-02-07 | Product Sales  |      4 |    |            |            |        |  |
+----+------------+----------------+--------+----+------------+------------+--------+--+

Which based on the currently available information is probably the best I can do

You could the use something like Most simple code to populate JTable from ResultSet or Retrieving Data from JDBC Database into Jtable to build the JTable ... as available examples

if second table have more row than table one .. record will not display which are more id in second table – raman dangol 8 hours ago

Also, if there are inconsistent IDs, the records will be lost. That's why I said this was not a trivial question

Then in those cases, something like FULL OUTER JOIN would be more useful. However, based on the information I have, full outer join is not supported in MySQL, because that would be useful.  However, based on How to do a FULL OUTER JOIN in MySQL? we could do something like...

select * from income left join expenses on income.id = expenses.id union select * from income right join expenses on income.id = expenses.id where income.id is null

which can generator something like

+----+------------+----------------+--------+----+------------+------------+--------+
| id | datevalue  |   particular   | amount | id | datevalue  | particular | amount |
+----+------------+----------------+--------+----+------------+------------+--------+
|  1 | 2017-02-02 | Cash Deposit   |    1.0 |  1 | 2017-05-07 | Factory    |    6.0 |
|  2 | 2017-02-03 | Cheque Deposit |    2.0 |  2 | 2017-06-02 | Staff      |    7.0 |
|  3 | 2017-02-03 | Cash Deposit   |    3.0 |  3 | 2017-06-03 | Travel     |    8.0 |
|  4 | 2017-02-03 | Product Sales  |    4.0 |  4 | 2017-10-01 | Test 1     |   10.0 |
|  5 | 2017-10-02 | Test 2         |   20.0 |    |            |            |        |
+----+------------+----------------+--------+----+------------+------------+--------+

Or, if you prefer to keep things aligned to their "selected sides", something like

select income.id, income.datevalue, income.PARTICULAR, income.AMOUNT,
       expenses.id, expenses.datevalue, expenses.PARTICULAR, expenses.AMOUNT
       from income join expenses on income.id = expenses.id
union all
select income.id, income.datevalue, income.PARTICULAR, income.AMOUNT,
       null, null, null, null
       from INCOME where not exists (select expenses.id from expenses where expenses.id = income.id)
union all
select null, null, null, null,
       expenses.id, expenses.datevalue, expenses.PARTICULAR, expenses.AMOUNT
       from expenses where not exists (select income.id from income where income.id = expenses.id)

Which can generate something like...

+----+------------+----------------+--------+----+------------+------------+--------+
| id | datevalue  |   particular   | amount | id | datevalue  | particular | amount |
+----+------------+----------------+--------+----+------------+------------+--------+
|  1 | 2017-02-02 | Cash Deposit   |    1.0 |  1 | 2017-05-07 | Factory    |    6.0 |
|  2 | 2017-02-03 | Cheque Deposit |    2.0 |  2 | 2017-06-02 | Staff      |    7.0 |
|  3 | 2017-02-03 | Cash Deposit   |    3.0 |  3 | 2017-06-03 | Travel     |    8.0 |
|  4 | 2017-02-03 | Product Sales  |    4.0 |  4 | 2017-10-01 | Test 1     |   10.0 |
|    |            |                |        |  5 | 2017-10-02 | Test 2     |   20.0 |
+----+------------+----------------+--------+----+------------+------------+--------+

At the end of the day, it's still a database issue.

For simplicity, you could create one or more database views in order to simply the query