I need additional help, based on similar question I have located here: Convert multilines to single lineCurrent SED example(sed  -e'y/)},={/(((((/' \  -e's/-\([^(I]*\)[^0-9]*\([0-9]*\)[( ]*/- \2 -\1/;=' |paste -d- - - |sort  -t- -nk3,3 -nk1,1 |sed   -e's/^[^-]*-//;:n' -e'h;$!N' \  -e's/\(-\([^-]*-\).*[^ ]\) *\n\([^-]*-\)\{2\}\2/\1 - \3/;tn' \  -ex\;:t -e's/\(\([^-]*-\)[^/]*\) - *\2/\1,/;tt'   -e'p;g;D' ) \< in.txt > out.txtFollowing multiline log6/13/2015 12:00:47 AM - {   562} START Web 6/13/2015 12:00:47 AM - Requested Web connection from 123.125.71.103[123.125.71.103], ID=562 6/13/2015 12:01:24 AM - {   563} START POP3 6/13/2015 12:01:24 AM - Requested POP3 connection from 10.127.251.37 [10.127.251.37], ID=563 6/13/2015 12:01:24 AM - (   563) USER test.mail@test.me 6/13/2015 12:01:24 AM - POP3 connection with 10.127.251.37 [10.127.251.37] ended. ID=563 6/13/2015 12:01:24 AM - {   563} END POP36/13/2015 12:01:24 AM - {   564} START POP3 6/13/2015 12:01:24 AM - Requested POP3 connection from 10.127.251.37 [10.127.251.37], ID=564 6/13/2015 12:01:24 AM - (   564) USER test.mail@test.me 6/13/2015 12:01:24 AM - POP3 connection with 10.127.251.37 [10.127.251.37] ended. ID=564 6/13/2015 12:01:24 AM - {   564} END POP36/13/2015 12:01:40 AM - Web connection with 123.125.71.103 [123.125.71.103] ended. ID=562 6/13/2015 12:01:40 AM - {   562} END WebIs converted to6/13/2015 12:00:47 AM - 562 - START Web, Requested Web connection from 123.125.71.103 [123.125.71.103] - 6/13/2015 12:01:40 AM - Web connection with 123.125.71.103 [123.125.71.103] ended., END Web6/13/2015 12:01:24 AM - 563 - START POP3, Requested POP3 connection from 10.127.251.37 [10.127.251.37], USER test.mail@test.me, POP3 connection with 10.127.251.37 [10.127.251.37] ended., END POP36/13/2015 12:01:24 AM - 564 - START POP3, Requested POP3 connection from 10.127.251.37 [10.127.251.37], USER test.mail@test.me, POP3 connection with 10.127.251.37 [10.127.251.37] ended., END POP3I would like to modify SED to include also follwoing logs where IDs are in square brackets (example: [ 667]).Log example6/13/2015 2:15:09 AM - Starting delivery thread, ID=6676/13/2015 2:15:09 AM - {   667} START Delivery6/13/2015 2:15:09 AM - Active delivery threads (ID=667 start): 06/13/2015 2:15:09 AM - [   667] Delivering to test.mail@test.com6/13/2015 2:15:09 AM - [   667] Forwarding mail...6/13/2015 2:15:09 AM - [   667] Forwarding message to 1 address(es)6/13/2015 2:15:09 AM - Delivery thread, ID=667, terminated after running for 0.031 seconds.6/13/2015 2:15:09 AM - {   667} END Delivery6/13/2015 2:15:09 AM - Active delivery threads (ID=667 end): 0Second question I have is how should I ignore logs where there is no ID.This is not so critical, but would be nice to filter out.For example:6/13/2015 2:43:24 AM - Exended logging6/13/2015 2:43:24 AM - Setting msgSizeLimit 15Thanks in advance.
Convert multilines to single line with SED
shell script;sed
It's only the first sed which needs modifying to handle all of the possibilities you list here. Specifically, rather than handling all possibilities in a single regexp, instead you'll need to address two different kinds of lines separately - the kind on which the ID immediately follows the first - dash on a line, and the other kind on which it is found labeled as ID= somewhere else.(   sed -e= -e's/- [({[] *\([0-9]*\) *[])}]/- \1 -/;t' \            -e's/-\(.*[^, ]\)[, ]*ID=\([0-9]*\)[, ]\{0,1\}/- \2 -\1/' |    paste   -d- - - |    sort    -t- -nk3,3 -nk1,1 |    sed     -e's/^[^-]*-//;:n' -e'h;$!N' \            -e's/\(-\([^-]*-\).*[^ ]\) *\n\([^-]*-\)\{2\}\2/\1 - \3/;tn' \            -ex\;:t -e's/\(\([^-]*-\)[^/]*\)- *\2/\1:::/;tt'   -e'p;g;D') <in >outHere it is a second time, but written with -Extended regular expression syntax. This will work at least with BSD/GNU/AST seds:(   sed -Ee= -e's/- [({[] *([0-9]+) *[])}]/- \1 -/;t' \             -e's/-(.*[^, ])[, ]*ID=([0-9]+)[, ]?/- \2 -\1/' |    paste    -d- - - |    sort     -t- -nk3,3 -nk1,1 |    sed      -Ee's/^[^-]*-//;:n' -e'h;$!N' \             -e's/(-([^-]+-).*[^ ]) *\n([^-]+-){2}\2/\1 - \3/;tn' \             -ex\;:t -e's/(([^-]+-)[^/]*)- *\2/\1:::/;tt' -e'p;g;D') <in >outNow I tested that on a concatenation of all of your log data:6/13/2015 12:00:47 AM - {   562} START Web6/13/2015 12:00:47 AM - Requested Web connection from 123.125.71.103[123.125.71.103], ID=5626/13/2015 12:01:24 AM - {   563} START POP36/13/2015 12:01:24 AM - Requested POP3 connection from 10.127.251.37 [10.127.251.37], ID=5636/13/2015 12:01:24 AM - (   563) USER test.mail@test.me6/13/2015 12:01:24 AM - POP3 connection with 10.127.251.37 [10.127.251.37] ended. ID=5636/13/2015 12:01:24 AM - {   563} END POP36/13/2015 12:01:24 AM - {   564} START POP36/13/2015 12:01:24 AM - Requested POP3 connection from 10.127.251.37 [10.127.251.37], ID=5646/13/2015 12:01:24 AM - (   564) USER test.mail@test.me6/13/2015 12:01:24 AM - POP3 connection with 10.127.251.37 [10.127.251.37] ended. ID=5646/13/2015 12:01:24 AM - {   564} END POP36/13/2015 12:01:40 AM - Web connection with 123.125.71.103 [123.125.71.103] ended. ID=5626/13/2015 12:01:40 AM - {   562} END Web6/13/2015 2:15:09 AM - Starting delivery thread, ID=6676/13/2015 2:15:09 AM - {   667} START Delivery6/13/2015 2:15:09 AM - Active delivery threads (ID=667 start): 06/13/2015 2:15:09 AM - [   667] Delivering to test.mail@test.com6/13/2015 2:15:09 AM - [   667] Forwarding mail...6/13/2015 2:15:09 AM - [   667] Forwarding message to 1 address(es)6/13/2015 2:15:09 AM - Delivery thread, ID=667, terminated after running for 0.031 seconds.6/13/2015 2:15:09 AM - {   667} END Delivery6/13/2015 2:15:09 AM - Active delivery threads (ID=667 end): 0And my results were:6/13/2015 12:00:47 AM - 562 - START Web ::: Requested Web connection from 123.125.71.103[123.125.71.103] - 6/13/2015 12:01:40 AM - Web connection with 123.125.71.103 [123.125.71.103] ended. ::: END Web6/13/2015 12:01:24 AM - 563 - START POP3 ::: Requested POP3 connection from 10.127.251.37 [10.127.251.37] ::: USER test.mail@test.me ::: POP3 connection with 10.127.251.37 [10.127.251.37] ended. ::: END POP36/13/2015 12:01:24 AM - 564 - START POP3 ::: Requested POP3 connection from 10.127.251.37 [10.127.251.37] ::: USER test.mail@test.me ::: POP3 connection with 10.127.251.37 [10.127.251.37] ended. ::: END POP36/13/2015 2:15:09 AM - 667 - Starting delivery thread ::: START Delivery ::: Active delivery threads (start): 0 ::: Delivering to test.mail@test.com ::: Forwarding mail... ::: Forwarding message to 1 address(es) ::: Delivery thread terminated after running for 0.031 seconds. ::: END Delivery ::: Active delivery threads (end): 0