Python - filter char based on type into a structure
<p>I've been banging my head against this for a couple of days, tried a variety of approaches, none of them seem to work in a way that I can use... </p>

<p>Problem. </p>

<p>I am given an arbitrary byte stream. There is some semantic elements buried in the bytes. There are curly brackets, square brackets and brackets. These indicated three different things - {} is a no. of bytes range, e.g. {17} is 17 bytes. [] is a byte value, e.g. [90:95] is bytes x90, x91, x92, x93, x94, x95. () is byte value 'OR' option, e.g. (46|47) means  either x46 or x47. </p>

<p>There are other grammatical constructs that I have to detect, "!", "*", "?" and ":". </p>

<p>An example bytestream: 524946(46|58){4}434452367672736E</p>

<p>I'm trying to filter it, so I get something like: </p>

<pre><code>1 string 524946
2 token (46|58)
3 token {4}
4 string 434452367672736E
</code></pre>

<p>Once I have it split up, I can then process it further.</p>

<p>The closest I have come to getting it work (its ugly ugly ugly code...): <a href="http://pastebin.com/XLg2H0PW" rel="nofollow">http://pastebin.com/XLg2H0PW</a> </p>

<p>I tried with some regex, but I could get it to not count the string bytes inside the grammatical units as normal string elements: </p>

<pre><code>range_masks_list =  [(m_mask1.span()) for m_mask1 in re.finditer("\{([0-9]+|[0-9]+-[0-9]+|[0-9]+-\*)\}",sequence)] ## looks for {int}, {int-int} and {int-*}
byte_masks_list =  [(m_mask2.span()) for m_mask2 in re.finditer("\[[a-fA-F0-9]{2}:[a-fA-F0-9]{2}]",sequence)] ## looks for [a:b] where a and b are byte ranges
options_sets_list = [(m_mask3.span()) for m_mask3 in re.finditer("\(([a-fA-F0-9]{2})+\|([a-fA-F0-9]{2})+(\|([a-fA-F0-9]{2})+)*\)",sequence)] ## looks for regex or clauses e.g. (a|b)
string_chunk_list = [(m_mask4.span()) for m_mask4 in re.finditer("([a-fA-F0-9]{2})+",sequence)] ## looks for uninterrupted hex byte spans 
</code></pre>

<p>which would look like: </p>

<pre><code>def do_fragmenter(self,sequence):
    """ converts the grep grammer normalised string into a set of fragments and offsets for sig population"""
    sequence = sequence.replace(" ","")
    range_masks_list =  [(m_mask1.span()) for m_mask1 in re.finditer("\{([0-9]+|[0-9]+-[0-9]+|[0-9]+-\*)\}",sequence)] ## looks for {int}, {int-int} and {int-*}
    byte_masks_list =  [(m_mask2.span()) for m_mask2 in re.finditer("\[[a-fA-F0-9]{2}:[a-fA-F0-9]{2}]",sequence)] ## looks for [a:b] where a and b are byte ranges
    options_sets_list = [(m_mask3.span()) for m_mask3 in re.finditer("\(([a-fA-F0-9]{2})+\|([a-fA-F0-9]{2})+(\|([a-fA-F0-9]{2})+)*\)",sequence)] ## looks for regex or clauses e.g. (a|b)
    string_chunk_list = [(m_mask4.span()) for m_mask4 in re.finditer("([a-fA-F0-9]{2})+",sequence)] ## looks for uninterupted hex byte spans 
    string_chunks = []
    string_chunks_len = []
    for pair in string_chunk_list:
        string_chunks.append(sequence[pair[0]:pair[1]])
        string_chunks_len.append(len(sequence[pair[0]:pair[1]]))
    print zip(string_chunks,string_chunks_len) 
</code></pre>
