Extract Text from a Binary File (using Python 2.7 on Windows 7)
Question: I have a binary file of size about 5MB.. which has lots of interspersed text..
and control characters..

This is actually an equivalent of an outlook .pst file for SITATEX Application
(from SITA).

The file contains all the TEXT MESSAGES sent and received to and from outside
world...(but the text has to be extracted through the binary control
characters).. all the text messages are clearly available... with line ending
^M characters... etc.

for example: assume ^@ ^X are control characters... \xaa with HEX aa, etc.
loads of them around my required text extraction.

    
    
    ^@^@^@^@^@^@^@^@^@^@^@BLLBBCC^X^X^X^X^X^X^X^X^X
    ^X^X^X
    MVT^M
    EA1123 TEXT TEXT TEXT^M
    END^M
    \xaa^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
     ^@^@^@^@^@^@^@^@^@^@^@TTBBTT^X^X^X^X^X^X^X^X^X
       ^X^X^X blah blah blah... of control characters.. and then the message comes..
       MVT MESSAGE 2
       ED1123
       etc.
    

and so on.. for several messages.

Using Perl.. it is easy to do:

    
    
    while (<>) {
      use regular expression to split messages
      m/   /
    
    
    }
    

How would one do this in python easily..

  1. How to read the file? binary and text interspersed
  2. Eliminate unnecessary control characters
  3. parse the messages in between two \xaa USEFUL TEXT INFORMATION \xaa (HEX 'aa')
  4. print out the required stuff
  5. Loop through all the lines.. and more files.

In the text file sample... I am interested in seeing.. BLLBBCC... and MVT and
EA1123 and so on.

Please assist... If it is going to be very difficult in python.. I will have
to think through the logic in perl itself.. as it (perl) doesn't throw lots of
errors at me at least for the looping part of binary and text stuff.. and the
regex.

Thanks.

**Update 02Jan after reading your answers/comments**

After going through S.Lott's comments and others... This is where I am at..
and it is working 80% ok.

    
    
    import fileinput
    import sys
    import re
    
    strfile = r'C:\Users\' \
              r'\Learn\python\mvt\sitatex_test.msgs'
    
    f = open(strfile, 'rb')
    
    contents = f.read() # read whole file in contents
    
    #extract the string between two \xaaU.. multiline pattern match
    #with look ahead assertion
    #and this is stored in a list with all msgs
    msgs = re.findall(r'\xaaU.*?(?=\xaaU)', contents, re.I|re.DOTALL|re.M)
    
    for msg in msgs:
        #loop through msgs.. to find the first msg then next and so on.
        print "## NEW MESSAGE STARTS HERE ##"
    
        #for each msg split the lines.. to read line by line
        # stored as list in msglines
        msglines = msg.splitlines()
        line = 0
    #then process each msgline with a message
    for msgline in msglines:
        line += 1
        #msgline = re.sub(r'[\x00]+', r' ', msgline)
        mystr = msgline
        print mystr
        textstrings = re.findall(r'[\x00\x20-\x7E]+', msgline)
    

So far so good.. still I am not completely done.. because I need to parse the
text line by line and word by word.. to pickup (as an example) the origin
address and headers, subject line, message body... by parsing the message
through the control characters.

Now I am stuck with... how to print line by line with the control characters
converted to `\x00\x02..` etc (using the `\xHH` format).. but leave the normal
readable text alone.

For example.. say I have this: assume `^@` and `^X` are some control
characters `line1 = '^@UG^@^@^@^@^@^@^@^@^@^@BLLBBCC^X^X^X^X^X^X^X^X^X'` (on
the first line).

When I print the line as it is on IDLE.. `print line1`.. it prints only say
the first 2 or 3 characters.. and ignores the rest due to the control
characters get choked.

However, when I print with this: `print re.findall(r'.*', line1)`

    
    
    ['\xaaUG\x02\x05\x00\x04\x00\x00\x00\x05\x00\x00\x00....
    x00\x00\x00..BLLBBCC\x00\x00N\x00N\\x00
     002 010 180000 DEC 11', '']
    

It prints nicely with all the control characters converted to \xHH format..
and ascii text intact.. (just as I want it)..with one catch.. the list has two
items.. with '' in the end.

  1. What is the explanation for the empty string in the end?
  2. How to avoid it... I just want the line converted nicely to a string (not a list). i.e. one line of binary/text to be converted to a string with \xHH codes.. leave the ASCII TEXT alone.

Is using `re.findall(r'.*', line1)` is the only easy solution.. to do this
conversion.. or are there any other straightforward method.. to convert a
`'\x00string'` to \xHH and TEXT (where it is a printable character or
whitespace).

Also.. any other useful comments to get the lines out nicely.

Thanks.

**Update 2Jan2011 - Part 2**

I have found out that `re.findall(r'.+', line1)` strips to

    
    
    ['\xaaUG\x02\x05\x00\x04\x00\x00\x00\x05\x00\x00\x00....
        x00\x00\x00..BLLBBCC\x00\x00N\x00N\\x00
         002 010 180000 DEC 11']
    

without the extra blank '' item in the list. This finding after numerous trial
and errors.

Still I will need assistance to eliminate the list altogether but return just
a string. like this:

    
    
    '\xaaUG\x02\x05\x00\x04..BLLBBCC..002 010 180000 DEC 11'
    

**Added Info on 05Jan:**

@John Machin

1) \xaaU is the delimiter between messages.. In the example.. I may have just
left out in the samples. Please see below for one actual message that ends
with \xaaU (but left out). Following text is obtained from repr(msg between
`r'\xaaU.*?(?=\xaaU)'`)

I am trying to understand the binary format.. this is a typical message which
is sent out the first 'JJJOWXH' is the sender address.. anything that follows
that has 7 alphanumeric is the receiver addresses.. Based on the sender
address.. I can know whether this is a 'SND' or 'RCV'.. as the source is
'JJJOWXH'... This msg is a 'SND' as we are 'JJJOWXH'.

The message is addressed to: JJJKLXH.... JJJKRXH.... and so on.

As soon as all the.. \x00000000 finishes.. the sita header and subject starts
In this particular case... `"\x00QN\x00HX\x00180001 \x00"` this is the
header.. and I am only interested all the stuff between \x00.

and the body comes next.. after the final \x00 or any other control
character... In this case... it is:

> COR\r\nMVT \r\nHX9136/17.BLNZ.JJJ\r\nAD2309/2314 EA0128
> BBB\r\nDLRA/CI/0032/0022\r\nSI EET 02:14 HRS\r\n RA / 0032 DUE TO LATE ARVL
> ACFT\r\n CI / 0022 OFFLOAD OVERHANG PALLET DUE INADEQUATE PACKING LEADING TO
> \r\n SPACE PROBLEM

once the readable text ends... the first control character that appears until
the end \xaaU is to be ignored... In above cases.. "SPACE PROBLEM".. is the
last one.. then control characters starts... so to be ignored... sometimes the
control characters are not there till the next \xaaU.

This is one complete message.

>
> "\xaaU\x1c\x04\x02\x00\x05\x06\x1f\x00\x19\x00\x00\x00\xc4\x9d\xedN\x1a\x00?\x02\x02\x00B\x02\x02\x00E\x02\x07\x00\xff\xff\x00\x00\xff\xff\x00\x00\xff\xff\x00\x00M\x02\xec\x00\xff\xff\x00\x00\x00\x00?\x02M\x02\xec\x00\xff\xff\x00\x00\xff\xff\x00\x00\xff\xff\x00\x00\xff\xff\x00\x00\xff\xff\x00\x00:\x03\x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x7f\x00JJJOWXH\x00\x05w\x01x\x01\x00\x01JJJKLXH\x00\x00\x7f\x01\x80\x01\x00\x01JJJKRXH\x00F\x87\x01\x88\x01\x00\x01JJJFFXH\x00\xff\x8f\x01\x90\x01\x00\x01JJJFCXH\x00\xff\x97\x01\x98\x01\x00\x01JJJFAXH\x00\x00\x9f\x01\xa0\x01\x00\x01JJJKPXH\x00\x00\xa7\x01\xa8\x01\x00\x01HAKUOHU\x00\x00\xaf\x01\xb0\x01\x00\x01BBBHRXH\x00\x00\xb7\x01\xb8\x01\x00\x01BBBFFHX\x00\x00\xbf\x01\xc0\x01\x00\x01BBBOMHX\x00\x00\xc7\x01\xc8\x01\x00\x01BBBFMXH\x00\x00\xcf\x01\xd0\x01\x00\x01JJJHBER\x00\x00\xd7\x01\xd8\x01\x00\x01BBBFRUO\x00\x00\xdf\x01\xe0\x01\x00\x01BBBKKHX\x00\x00\xe7\x01\xe8\x01\x00\x01JJJLOTG\x00\x01\xef\x01\xf0\x01\x00\x01JJJLCTG\x00\x00\xf7\x01\xf8\x01\x00\x01HDQOMTG\x005\xff\x01\x00\x02\x00\x01CHACSHX\x00K\x07\x02\x08\x02\x00\x01JJJKZXH\x00F\x0f\x02\x10\x02\x00\x01BBBOMUO\x00
> \x17\x02\x18\x02\x00\x01BBBORXH\x00 \x1f\x02
> \x02\x00\x01BBBOPXH\x00W'\x02(\x02\x00\x01CHACSHX\x00
> /\x020\x02\x00\x01JJJDBXH\x0007\x028\x02\x00010000\x00\x00000000\x00\x00000000\x00\x00000000\x00\x00000000\x00\x00000000\x00\x00000000\x00\x00000000\x00\x00000000\x00\x00000000\x00\x00000000\x00\x00000000\x00\x00000000\x00\x00000000\x00\x00000000\x00\x00000000\x00\x00000000\x00\x00000000\x00\x00000000\x00\x00000000\x00\x00000000\x00\x00000000\x00\x00000000\x00\x00000000\x00\x00000000\x00QN\x00HX\x00180001
> \x00COR\r\nMVT \r\nHX9136/17.BLNZ.JJJ\r\nAD2309/2314 EA0128
> BBB\r\nDLRA/CI/0032/0022\r\nSI EET 02:14 HRS\r\n RA / 0032 DUE TO LATE ARVL
> ACFT\r\n CI / 0022 OFFLOAD OVERHANG PALLET DUE INADEQUATE PACKING LEADING TO
> \r\n SPACE PROBLEM\x00D-\xedN\x00\x04\x1a\x00t<\x93\x01x\x00M_\x00"

2) I am not using .+ anymore after the 'repr' is known.

3) each Message is multiline.. and i need to preserve all the control
characters to make some sense of this proprietary format.. that is why i
needed repr to see it up close.

Hope this explains... This is just 1 message out of 1000s with in the file...
and some are 'SND' and some are 'RCV'... and for 'RCV' there will not be
'000000'.. and occasionally there are minor exceptions to the rule... but
usually that is okay.

Any further suggestions anyone.. I am still working with the file.. to
retrieve the text out intact... with sender and receiver addresses.

Thank you.


Answer: **Q: How to read the file? binary and text interspersed**

_A: Don't bother, just read it as normal text and you'll be able to keep your
binary/text dichotomy (otherwise you won't be able to regex it as easily)_

    
    
    fh = open('/path/to/my/file.ext', 'r')
    fh.read()
    

Just in case you want to read binary later for some reason, you just add a b
to the second input of the open:

    
    
    fh = open('/path/to/my/file.ext', 'rb')
    

**Q: Eliminate unnecessary control characters**

_A: Use the python[re](http://docs.python.org/library/re.html) module. Your
next question sorta ask how_

**Q: parse the messages in between two \xaa USEFUL TEXT INFORMATION \xaa (HEX
'aa')**

_A: re module has a findall function that works as you (mostly) expect._

    
    
    import re
    
    mytext = '\xaaUseful text that I want to keep\xaa^X^X^X\xaaOther text i like\xaa'
    usefultext = re.findall('\xaa([a-zA-Z^!-~0-9 ]+)\xaa', mytext)
    

**Q: print out the required stuff**

*A: There's a print function...
    
    
    print usefultext
    

**Q: Loop through all the lines.. and more files.**

    
    
    fh = open('/some/file.ext','r')
    
    for lines in fh.readlines():
        #do stuff
    

I'll let you figure out the [os](http://docs.python.org/library/os.html)
module to figure out what files exist/how to iterate through them.

