Issues converting larger HTML files to docx in pypandoc (pandoc)
Question: My question is related to [How to increase heap memory in pandoc
execution?](http://stackoverflow.com/questions/30704736/how-to-increase-heap-
memory-in-pandoc-execution), but adds a Python-specific component.

Background: I'm trying to generate scientific reports automatically. I've
written data to an HTML file, and I'd like to use Pandoc.exe (a file
conversion program) to convert to a .docx Word Document. I've got the process
to work for a smaller HTML file with an image, table, ect. That file was
307KB.

The problem starts when I try to convert a larger file (~4.5MB) with multiple
graphs embedded. I've been using `pypandoc` to convert, like this:

    
    
    import pypandoc
    PANDOC_PATH = r"C:\Program Files\RStudio\bin\pandoc"
    
    infile = savepath + os.sep + 'Results ' + name + '.html'
    outfile = savepath + os.sep + 'Results ' + name + '.docx'
    
    output = pypandoc.convert(source=infile, format='html', to='docx', \
    outputfile=outfile, extra_args=["+RTS", "-K64m", "-RTS"])
    

But I get a variety of errors. Usually:

    
    
    RuntimeError: Pandoc died with exitcode "2" during conversion: 
    b"Stack space overflow: current size 33692 bytes.\nUse `+RTS -Ksize -RTS' to increase it.\n"
    

or if I crank the value of -Ksize up to 256m, something like this:

    
    
    RuntimeError: Pandoc died with exitcode "1" during conversion: b'pandoc: out of memory\r\n'
    

* * *

**Can someone explain what is happening, here, and some way I can get around
this difficulty?** One solution I've thought about is making my images a lot
smaller. I've just been scaling down the (80 - 500KB) originals like this,
where the width and the height of each image is dependent on it's original
dimensions:

    
    
    data_uri = base64.b64encode(open(formats[graph][0], 'rb').read()).decode('utf-8')
    
    img_tag = ('<img src="data:image/jpg;base64,{0}" height='+formats[graph][2][0]+'
                 width='+formats[graph][2][1]+'>').format(data_uri) 
    

Thanks for your help


Answer: Thanks very much to help from
[user2407038](http://stackoverflow.com/users/2407038/user2407038) on this one!

Two fixes finally allowed me to convert my larger HTML file to a docx file
with `pypandoc`:

The first, as suggested, was

> increasing the maximum size of the heap, e.g. add -M2GB to extra_args

That is:

`output = pypandoc.convert(source=infile, format='html', to='docx',
outputfile=outfile, extra_args=["-M2GB", "+RTS", "-K64m", "-RTS"])`

* * *

After increasing the heap size, I still had a second problem, so I wasn't sure
if the solution worked. Python returned an error message like this:

> RuntimeError: Pandoc died with exitcode "1" during conversion: b"pandoc:
> Cannot decode byte '\x91': Data.Text.Internal.Encoding.Fusion.streamUtf8:
> Invalid UTF-8 stream\n"

Which was fixed by changing how the html file was opened in the first place.
Setting the `encoding` keyword argument to `'utf8'` allows for the conversion
to work:

    
    
    report = open(savepath + os.sep + 'Results ' + name + '.html', 'w', encoding='utf8')
    

