How to output a file where named entities are replaced by tags using Stanford CoreNLP in Python?
Question: I'm working with Stanford NLP using Python. So, I have a function that inputs
some text files and converts them to xml files(generated by Stanford CoreNLP).
Now, I want to write another function that inputs these xml files and outputs
the corresponding file that contains the same text but with named entities
replaced by their tag and marking the ends of sentences with the word "STOP",
removing the punctuations. The beginning of the file also has the word "STOP"
. The function giving the xml files is:

    
    
    import subprocess
    def generate_xml(input,output):
        p = subprocess.Popen('java -cp stanford-corenlp-2012-07-09.jar:stanford-corenlp-2012-07-06-models.jar:xom.jar:joda-time.jar -Xmx3g edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators tokenize,ssplit,pos,lemma,ner -filelist /Users/akritibahal/Downloads/stanford-corenlp-2012-07-09/myfile_list.txt -outputDirectory /Users/akritibahal/Downloads/stanford-corenlp-2012-07-09', shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
        for line in p.stdout.readlines():
            print line
    
        retval = p.wait()
    

The function is that will give out an output file with named entity tags:

    
    
    def process_file(input_xml,output_file):
    

Could anybody help me how to get such output file with named entity tags?


Answer: I've been parsing the output of the CoreNLP using minidom. Here is some
starter code you may want to use but you may want to check
<https://github.com/dasmith/stanford-corenlp-python>

Note that you need to get the tokenization used by the Stanford CoreNLP since
the data returned is based on offsets of sentences and tokens.

    
    
    from xml.dom import minidom    
    xmldoc = minidom.parseString(raw_xml_data)
    for sentence_xml in xmldoc.getElementsByTagName('sentences')[0].getElementsByTagName('sentence'):
        parse = parser.parse(sentence_xml.getElementsByTagName('parse')[0].firstChild.nodeValue)
        tokens = [(i,j) for i,j in zip(sentence_xml.getElementsByTagName('tokens')[0].getElementsByTagName('token'),parse.get_leaves())]
        # example for processing dependencies
        elements = sentence_xml.getElementsByTagName('dependencies')
        for element in elements:
            if element.getAttribute('type')=="collapsed-ccprocessed-dependencies":
                dependencies += [i for i in element.getElementsByTagName('dep')]
    

