Python - Reading a File in Single/Multi-line Pieces
<p>I'm pretty new to Python, I have found answers to most of my issues here but this one has me stumped.</p>

<p>I am processing log files using Python, generally every line starts with a date/time stamp such as:</p>

<pre><code>[1/4/13 18:37:37:848 PST]
</code></pre>

<p>In 99% of cases I can read line by line, look for items of interest and process them accordingly however occasionally an entry in the log file will include a message that has carriage return / new line characters in it, as such it will span multiple lines.</p>

<p>Is there a way I can easily read the file "between timestamps" such that when this situation occurs the multiple lines would be consolidated in to one read instead? For example:</p>

<pre><code>[1/4/13 18:37:37:848 PST] A log entry
[1/4/13 18:37:37:848 PST] Another log entry
[1/4/13 18:37:37:848 PST] A log entry that somehow
got some new line
characters mixed in
[1/4/13 18:37:37:848 PST] The last log entry
</code></pre>

<p>Would be read as four lines rather than six as it is now.</p>

<p>Thanks in advance for any help.</p>

<p>Chris,</p>

<p>Update....</p>

<p>myTestFile.log contains the exact text above, here is my script:</p>

<pre><code>import sys, getopt, os, re
sourceFolder = 'C:/MaxLogs'
logFileName = sourceFolder + "/myTestFile.log"
lines = []

def timestamp_split(file):
    pattern = re.compile("\[(0?[1-9]|[12][0-9]|3[01])(\/)(0?[1-9]|[12][0-9]|3[01])(\/)([0-9]{2})(\ )")
    current = []
    for line in file:
        if not re.match(pattern,line):
            if current:
                yield "".join(current)
            current == [line]
        else:
            current.append(line)
    yield "".join(current)

print "--- START ----"
with open(logFileName) as file:
    for entry in timestamp_split(file):
        print entry
        print "- Record Separator -"
print "--- DONE ----"
</code></pre>

<p>When I run it I get this:</p>

<pre><code>--- START ----
[1/4/13 18:37:37:848 PST] A log entry
[1/4/13 18:37:37:848 PST] Another log entry
[1/4/13 18:37:37:848 PST] A log entry that somehow

- Record Separator -
[1/4/13 18:37:37:848 PST] A log entry
[1/4/13 18:37:37:848 PST] Another log entry
[1/4/13 18:37:37:848 PST] A log entry that somehow

- Record Separator -
[1/4/13 18:37:37:848 PST] A log entry
[1/4/13 18:37:37:848 PST] Another log entry
[1/4/13 18:37:37:848 PST] A log entry that somehow
[1/4/13 18:37:37:848 PST] The last log entry
- Record Separator -
--- DONE ----
</code></pre>

<p>I seem to be iterating through the lines too many times, what I was expecting (hoping) for was this:</p>

<pre><code>--- START ----
[1/4/13 18:37:37:848 PST] A log entry
- Record Separator -
[1/4/13 18:37:37:848 PST] Another log entry
- Record Separator -
[1/4/13 18:37:37:848 PST] A log entry that somehow got some new line characters mixed in
- Record Separator -
[1/4/13 18:37:37:848 PST] The last log entry
- Record Separator -
--- DONE ----
</code></pre>

<p>As discussed in the comments I accidentally left the <em>not</em> in the comparison with the regex pattern from when I was testing, if I remove it then I get all of the partial lines which confuses me even more!</p>

<pre><code>--- START ----
got some new line
characters mixed in

- Record Separator -
got some new line
characters mixed in

- Record Separator -
--- DONE ----
</code></pre>
