text extraction line splitting across multiple lines with python
Question: I have the following code:

    
    
    f = open('./dat.txt', 'r')
    array = []
    for line in f:
        # if "1\t\"Overall evaluation" in line:
        #   words = line.split("1\t\"Overall evaluation")
        #   print words[0]
        number = int(line.split(':')[1].strip('"\n'))
        print number
    

This is capable of grabbing the last int from my data, which looks like this:

    
    
    299 1   "Overall evaluation: 3
    Invite to interview: 3
    Strength or novelty of the idea (1): 4
    Strength or novelty of the idea (2): 3
    Strength or novelty of the idea (3): 3
    Use or provision of open data (1): 4
    Use or provision of open data (2): 3
    ""Open by default"" (1): 2
    ""Open by default"" (2): 3
    Value proposition and potential scale (1): 4
    Value proposition and potential scale (2): 2
    Market opportunity and timing (1): 4
    Market opportunity and timing (2): 4
    Triple bottom line impact (1): 4
    Triple bottom line impact (2): 2
    Triple bottom line impact (3): 2
    Knowledge and skills of the team (1): 3
    Knowledge and skills of the team (2): 4
    Capacity to realise the idea (1): 4
    Capacity to realise the idea (2): 3
    Capacity to realise the idea (3): 4
    Appropriateness of the budget to realise the idea: 3"
    299 2   "Overall evaluation: 3
    Invite to interview: 3
    Strength or novelty of the idea (1): 3
    Strength or novelty of the idea (2): 2
    Strength or novelty of the idea (3): 4
    Use or provision of open data (1): 4
    Use or provision of open data (2): 3
    ""Open by default"" (1): 3
    ""Open by default"" (2): 2
    Value proposition and potential scale (1): 4
    Value proposition and potential scale (2): 3
    Market opportunity and timing (1): 4
    Market opportunity and timing (2): 3
    Triple bottom line impact (1): 3
    Triple bottom line impact (2): 2
    Triple bottom line impact (3): 1
    Knowledge and skills of the team (1): 4
    Knowledge and skills of the team (2): 4
    Capacity to realise the idea (1): 4
    Capacity to realise the idea (2): 4
    Capacity to realise the idea (3): 4
    Appropriateness of the budget to realise the idea: 2"
    
    364 1   "Overall evaluation: 3
    Invite to interview: 3
    ...
    

I also need to grab the "record identifier" which in the above example would
be `299` for the first two instances and then `364` for the next one.

The commented out code above, if I delete the last lines and just use it, as
so:

    
    
    f = open('./dat.txt', 'r')
    array = []
    for line in f:
        if "1\t\"Overall evaluation" in line:
            words = line.split("1\t\"Overall evaluation")
            print words[0]
        # number = int(line.split(':')[1].strip('"\n'))
        # print number
    

can grab the record identifiers.

but I'm having trouble putting the two together.

Ideally what I want is something like the following:

    
    
    368
    
    =2+3+3+3+4+3+2+3+2+3+2+3+2+3+2+3+2+4+3+2+3+2
    
    =2+3+3+3+4+3+2+3+2+3+2+3+2+3+2+3+2+4+3+2+3+2
    

and so on for all records.

How can I combine the above two script components to achieve that?


Answer: Regex is the ticket. You can do it with two patterns. Something like this:

    
    
    import re
    
    with open('./dat.txt') as fin:
        for line in fin:
            ma = re.match(r'^(\d+) \d.+Overall evaluation', line)
            if ma:
                print("record identifier %r" % ma.group(1))
                continue
            ma = re.search(r': (\d+)$', line)
            if ma:
                print(ma.group(1))
                continue
            print("unrecognized line: %s" % line)
    

Note: The last print statement is not part of your requirements, but whenever
I debug regex, I always add some sort of catchall to assist with debugging bad
regex statements. Once I get my patterns straight, I remove the catchall.

