Processing lexical analyses of sentences using the Perl split function I have two kinds of lexical analyses of sentences that I need to process.  One type of data comes in a "tagged" format, and the other comes in a "parsed" format.



Tagged

The input (@subsentences) looks like:

5.4_CD Passive_NNP Processes_NNP of_IN Membrane_NNP Transport_NNP 85_CD We_PRP have_VBP examined_VBN membrane_NN structure_NN and_CC how_WRB it_PRP is_VBZ used_VBN to_TO perform_VB one_CD membrane_NN function_NN :_: the_DT binding_JJ of_IN one_CD cell_NN to_TO another_DT ._.


Desired output

5.4 Passive Processes of Membrane Transport 85 We have examined membrane stru....

My code

@finalsentence = split(/_\S+/,$subsentences[$j]);




Parsed

   Parsing [sent. 1 len. 31]:
        nsubj(85-7, Processes-3)
        nn(Transport-6, Membrane-5)
        prep_of(Processes-3, Transport-6)
        nsubj(examined-10, We-8)
        nsubjpass(used-17, it-15)
        xsubj(perform-19, it-15)
        conj_and(examined-10, used-17)
        xcomp(used-17, perform-19)
        dobj(perform-19, function-22)
        prep_of(binding-25, cell-28) &lt;- refer to this for examples below


Desired output (for the last line)


the sent. number (ie. sent. 1 )
the grammar function (ie. prep_of )
the first dependency word (ie. binding ) 
the second dependency word (ie. cell )


My code

Here is how I do it, but when I check for word boundaries (\b), sometimes they're not defined and on top of that, it's pretty crude:

For the sent. number:

@parsesentcounter = split (/.*sent\.\s/, $typeddependencies[$i]);
@parsesentcounter = split (/\s/, $typeddependencies[$i]); 


This (crude method) leaves the sent. number (sent. 1) at $parsesentcounter[2]

For the grammar function:

@grammarfunction = split(/\(\S+\s\S+\s/,$typeddependencies[$i]);


This leaves the grammar function(prep_of) at $grammarfunction[0]

For the dependency words, I do it in a few steps (I think I get lost a bit here):

@dependencywords = split (/,\s+/,$typeddependencies[$i]); ## Take out all commas, there was also a space associated
@dependencywords = split (/-\S+\s+/,$typeddependencies[$i]); ## Take out all -digits and space


This leaves the second dependency word(cell) at $dependencywords[1].

Then for first dependency word:

@firstdependencyword = split(/.*subj\w*.|.*obj\w*.|.*prep\w*\(|.*xcomp\w*\(|.*agent\(|purpcl\(|.*conj_and\(/,$dependencywords[0]);


This leaves the first dependency word (binding) at $firstdependencyword[1]
Match and substitute are far better suited for this problem than split().
To delete all of the _XX things, use a substitute global s/(_\S+)//g;.
So generating your desired output is actually easy.

To see demo of this and how to parse your report, see the code below. Note that if the left hand side contains variables in a list context like:($gramFunc, $dep1, $dep2) then they will be assigned $1,$2,$3 from the match respectively. I check if $dep2 is defined to mean that the match succeeded because if that variable is defined, then the variables $gramFunc and $dep1 are too!

This line $sentNum = $1 if /\[sent\.\s*(\d+)/; has the same meaning as:

if (/\[sent\.\s*(\d+)/)
{
   $sentNum = $1;
}


I cannot assign to $sentNum directly in this case because I want to know if the match worked or not. $1 is undefined if the match fails and the scalar value of /\[sent\.\s*(\d+)/ will be zero. I presume that more such "[sent" lines will appear in the report that you are trying to parse. This statement just updates as necessary the current sent # for the report.

Code:

#!/usr/bin/perl -w
use strict;

my $str = '5.4_CD Passive_NNP Processes_NNP of_IN Membrane_NNP Transport_NNP 85_CD We_PRP have_VBP examined_VBN membrane_NN structure_NN and_CC how_WRB it_PRP is_VBZ used_VBN to_TO perform_VB one_CD membrane_NN function_NN :_: the_DT binding_JJ of_IN one_CD cell_NN to_TO another_DT ._.';

$str =~ s/(_\S+)//g;  # deletes all of the "_XX" tokens
print $str, "\n";

my $sentNum = 0;
while (&lt;DATA&gt;)
{
   $sentNum = $1 if /\[sent\.\s*(\d+)/;

   my ($gramFunc, $dep1, $dep2) = $_ =~ /\s+(\w+)\((\w+).*?(\w+)-/;
   if (defined $dep2)
   {
      printf "sent num=$sentNum %-10s %-15s %-15s\n", $gramFunc, $dep1, $dep2;
   }
}

=program output
5.4 Passive Processes of Membrane Transport 85 We have examined membrane structure and how it is used to perform one membrane function : the binding of one cell to another .
sent num=1 nsubj      85              Processes      
sent num=1 nn         Transport       Membrane       
sent num=1 prep_of    Processes       Transport      
sent num=1 nsubj      examined        We             
sent num=1 nsubjpass  used            it             
sent num=1 xsubj      perform         it             
sent num=1 conj_and   examined        used           
sent num=1 xcomp      used            perform        
sent num=1 dobj       perform         function       
sent num=1 prep_of    binding         cell       
=cut


__DATA__
Parsing [sent. 1 len. 31]:
        nsubj(85-7, Processes-3)
        nn(Transport-6, Membrane-5)
        prep_of(Processes-3, Transport-6)
        nsubj(examined-10, We-8)
        nsubjpass(used-17, it-15)
        xsubj(perform-19, it-15)
        conj_and(examined-10, used-17)
        xcomp(used-17, perform-19)
        dobj(perform-19, function-22)
        prep_of(binding-25, cell-28) &lt;- refer to this for examples below