Paragraph Matching Python
Question: **Background information**

I have a Python script which generates word documents with the `docx` module.
These documents are generated based on a log and then printed and stored as
records. However, the log can be edited retroactively, so the document records
need to be revised, and these revisions must be tracked. I'm not actually
revising the documents, but generating a new one which shows the difference
between what is currently in the log, and what will soon be in the log (the
log is updated after the revised file is printed). When a revision occurs, my
script uses `diff_match_patch` to generate a mark-up of what's changed with
the following function:

    
    
    def revFinder(str1,str2):
        dmp = dmp_module.diff_match_patch()
    
        diffs = dmp.diff_main(str1,str2)
        paratext = []
    
        for diff in diffs:
            paratext.append((diff[1], '' if diff[0] == 0 else ('s' if diff[0] == -1 else 'b')))
    
        return paratext
    

`docx` can take text either as strings, or by tuple if word-by-word formatting
is required, so [see second bullet in "Some Things to Note"]

    
    
    [("Hello, ", ''), ("my name ", 'b'), ("is Brad", 's')]
    

produces

> Hello **my name** ~~is Brad~~

* * *

**The Problem**

`diff_match_patch` is a very efficient code which finds the difference between
two texts. Unfortuanly, its a little too efficient, so replacing `redundant`
with `dune` results in

> ~~re~~ dun~~ant~~**e**

This is ugly, but its fine for single words. However, if an entire paragraph
gets replaced, the results will be entirely unreadable. That is not ok.

Previously I addressed this by collapsing all the text into a single
paragraph, but this was less than ideal because it became very cluttered and
was still pretty ugly.

* * *

**The Solution So Far**

I have a function which creates the revision document. This function gets
passed a list of tuples set up like this:

    
    
    [(fieldName, original, revised)]
    

So the document is set up as

    
    
    Orignial fieldName (With Markup)
      result of revFinder diffing orignal and revised
    
    Revised fieldName    
      revised
    

I assume in order to resolve the problem, I'll need to do some sort of
matching between paragraphs to make sure I don't diff two completely separate
paragraphs. I'm also assuming this matching will depend if paragraphs are
added or removed. Here's the code I have so far:

    
    
    if len(item[1].split('\n')) + len(item[1].split('\n'))) == 2:
    
        body.append(heading("Original {} (With Markup)".format(item[0]),2))
        body.append(paragraph(revFinder(item[1],item[2])))
        body.append(paragraph("",style="BodyTextKeep"))
        body.append(heading("Revised {}".format(item[0]),2))
        body.append(paragraph(item[2]))
        body.append(paragraph(""))
    
    else:
        diff = len(item[1].split('\n')) - len(item[1].split('\n'))
        if diff == 0:       
    
            body.append(heading("Original {} (With Markup)".format(item[0]),2))
            for orPara, revPara in zip(item[1].split('\n'),item[2].split('\n')):
                body.append(paragraph(revFinder(orPara,revPara)))
            body.append(paragraph("",style="BodyTextKeep"))
            body.append(heading("Revised {}".format(item[0]),2))
            for para in item[2].split('\n'):
                body.append(paragraph("{}".format(para)))     
            body.append(paragraph(""))
    
        elif diff > 0:
        #Removed paragraphs
    
    
    
        elif diff < 0: 
        #Added paragraphs  
    

So far I've planned on using something like `difflib` to do paragraph
matching. But if there's a better way to avoid this problem that is a
completely different approach, that's great too.

* * *

**Some Things to Note:**

  * I'm running Python 2.7.6 32-bit on Windows 7 64-bit
  * I've made some changes to my local copy of `docx` (namely adding the strike through formatting) so if you test this code you will not be able to replicate what I'm doing in that regard

* * *

**Description of the Entire Process (with the revision steps in bold):**

1) User opens Python script and uses GUI to add information to a thing called
a "Condition Report" (CR)

> **NOTE:** A full CR contains 4 parts, all completed by different people. But
> each part gets individually printed. All 4 parts are stored together in the
> log

2) When the user is finished, the information is saved to a log (described
below), and then printed as a `.docx` file

3) The printed document is signed and stored

**4)** When the user wants to revise a part of the CR, the open the GUI, and
edit the information in each of the fields. I am only concerned about a few of
the fields in this question, and those are the multiline text controls (which
can result in multiple paragraphs)

**5)** Once the user is done with the revision, the code generates the tuple
list I described in the "Solution So Far" section, and sends this to the
function which generates the revision document

**6)** The revision document is created, printed, signed, and stored with the
original document for that part of that CR

**7)** The log is completely rewritten to include the revised information

* * *

**The Log:**

The log is simply a giant `dict` which stores all the information on all of
the CRs. The general format is

    
    
    {"Unique ID Number": [list of CR info]}
    

The log doesn't store past versions of a CR, so when a CR is revised the old
information is overwritten (which is what we want for the system). As I
mentioned earlier, every time the log is edited, the whole thing is rewritten.
To get at the information in the log, I `import` it (since it always lives in
the same directory as the script)


Answer: Try using the post-diff cleanup options that diff_match_patch that @tzaman
mentioned above, in particular, check out the `diff_cleanupSemantic` function
which is intended for use when the diff output is intended to be human-
readable.

Cleanup options are NOT run automatically, since diff_match_patch provides
several cleanup options from which you may choose (depending on your needs).

Here is an example:

    
    
    import diff_match_patch
    
    dmp = diff_match_patch.diff_match_patch()
    diffs = dmp.diff_main('This is my original paragraph.', 'My paragraph is much better now.')
    print diffs  # pre-cleanup
    
    dmp.diff_cleanupSemantic(diffs)
    print diffs  # post cleanup
    

Output:

    
    
    [(-1, 'This is m'), (1, 'M'), (0, 'y'), (-1, ' original'), (0, ' paragraph'), (1, ' is much better now'), (0, '.')]
    [(-1, 'This is my original paragraph'), (1, 'My paragraph is much better now'), (0, '.')]
    

As you can see, the first diff is optimal but unreadable, while the second dif
(after cleanup) is exactly what you are looking for.

