Count unique IP address in a date range from Apache log I use the following command to get specific entries between certain dates from Apache log files. Then I filter out my own address and some bots, then I print out the IP, sort it, get the uniques, output the ip with count, and for summary I count lines to get a total.

Is there a way to improve this, reducing the amount of awk commands? 
Or to get this process down more efficient? I'm not that proficient with awk, but I have the feeling this should be accomplish able in maybe one awk instruction.

zgrep '"GET /my/path/to/page.html' other_vhosts_access* 
| awk -F'[][]' 
    -v dstart=`date -d"2019-12-09" +%Y%m%dT%0H:%0M:%0S` 
    -v dend=`date -d"2020-01-09" +%Y%m%dT%0H:%0M:%0S` 
    '{ $2 = substr($2,8,4)sprintf("%02d",(match("JanFebMarAprMayJunJulAugSepOctNovDec",substr($2,4,3))+2)/3)substr($2,1,2)"T"substr($2,13,8); 
    if ($2 &gt;= dstart &amp;&amp; $2 &lt; dend) print }' 
| awk '$0 !~ /127\.0\.0\.1|bot\.|bot\/|dotbot|crawler/' 
| awk '{print $2}' 
| sort 
| uniq -c 
| sort -n 
| wc -l


the one liner: 

zgrep '"GET /my/path/to/page.html' other_vhosts_access* | awk -F'[][]' -v dstart=`date -d"2019-12-09" +%Y%m%dT%0H:%0M:%0S` -v dend=`date -d"2020-01-09" +%Y%m%dT%0H:%0M:%0S` '{ $2 = substr($2,8,4)sprintf("%02d",(match("JanFebMarAprMayJunJulAugSepOctNovDec",substr($2,4,3))+2)/3)substr($2,1,2)"T"substr($2,13,8); if ($2 &gt;= dstart &amp;&amp; $2 &lt; dend) print }' | awk '$0 !~ /127\.0\.0\.1|bot\.|bot\/|dotbot|crawler/' | awk '{print $2}' | sort | uniq -c | sort -n | wc -l
I assume there's a simply copy-paste error where you have | at the start of a line, instead of at the end of the preceding line?

If we're just going to count lines, there's no need for sort | uniq -c | sort -n - we can replace all that with sort -u.

These two Awk programs can be trivially combined:


awk '$0 !~ /127\.0\.0\.1|bot\.|bot\/|dotbot|crawler/' | awk '{print $2}'



That would become

awk '$0 !~ /127\.0\.0\.1|bot\.|bot\/|dotbot|crawler/ {print $2}'