Python Parsing XML with a complex hierarchy - Nuke9.0v8
Question: I am working with NukeX9.0v8, Adobe Premiere Pro CC 2015 and nukes internal
python interrupter.

    
    
    # Result: 2.7.3 (default, Jul 24 2013, 15:50:23) 
    [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
    

I am a vfx artist and I'm trying to wrap my brain around the best method to
parse xml files in order to: create a folder structure, batch create .nk comp
files and plug in the data within specific parts as I make my .nk comps. I
have a bit of a grasp of how to do each of these things in isolation, but
plugging it all together, and trying to find tutorials on such complex parse
as ground me to a halt.

I know the scope of this is big but any small pieces of advice are
appreciated.

Right now I have a nuke comp that has a node tree that takes in camera inputs
and stitches them into a latlong image for 360 video, I am going to wrap that
up into a gizmo for each different kind of rig configuration. This just
simplifies the .nk files that are created and I can expose the parts of that
gizmo I can feed data into.

Every day we receive a ton of footage from a shoot and we have to make a new
.nk comp for each shot and set it to render right away. What I want to do is
have the guys on set create a premiere project and organize the files based on
this folder structure. That premiere project will be exported as an .xml file.

The design of the structure in premiere.

  * Day_01 (the day of the shoot)
  * -^-R001 (Roll number for the shots. R referring to camera type)
  * \--^-R001_C001 (The name of the shot)
  * \---^-Acamera clip (path to file name, video in point as frame#)
  * \---^-Bcamera clip (path to file name, video in point as frame#)
  * \---^-Ccamera clip (path to file name, video in point as frame#)

Right now in my script panel inside Nuke I can enter the information of where
is the xml for the day what day to look for. Then it is suppose to look into
each folder name for the roll, and using the first letter (R for RED camera)
and looks inside for the clip folder. It then uses the pathurl directory for
the camera files on the drive and also can take it data like the in and out
points if present in the xml. I also have points to enter for the template
version if I update a stitch process. That will tell the nuke comp which gizmo
to use.

Here is my panel in Nuke.

    
    
    def sesquixmlparse():
    '''
    This imports the xml file from premiere. It looks for the bin that it is working for today and starts looking in what is inside the bins
    It then sees the bins inside and uses them to create nuke scripts with these as inputs
    It asks what template version to use for the rig. things change or maybe even get better
    '''
    
    # Lets build the Nuke Panel that tells us our inputs
    p = nuke.Panel("Sesqui XML Parse for Dailies")
    xml_file = 'Daily XML'
    daynumber = 'Day_##'
    nk_output_dir = 'Directory to build VFX folder structure'
    dnx_render_dir = 'Directory for write nodes'
    r_template_vr = 'VER1'
    g_template_vr = 'VER1'
    c_template_vr = 'VER1'
    
    p.addFilenameSearch("Daily XML", xml_file)
    p.addSingleLineInput("Bin to process", daynumber)
    p.addFilenameSearch("Directory to build VFX folder structure", nk_output_dir)
    p.addFilenameSearch("Directory to render from write nodes", dnx_render_dir)
    p.addSingleLineInput("3 Red stmap version", r_template_vr)
    p.addSingleLineInput("6 Gopro stmap verison", g_template_vr)
    p.addSingleLineInput("5 Canon stmap verison", c_template_vr)
    p.setWidth(600)
    print "Panel created"
    if not p.show():
        return
    
    # Assign var from nuke panel user-entered data
    xml_file = p.value("Daily XML")
    daynumber = p.value("Bin to process")
    nk_output_dir = p.value("Directory to build VFX folder structure")
    dnx_render_dir = p.value("Directory to render from write nodes")
    r_template_vr = p.value("3 Red stmap version")
    g_template_vr = p.value("6 Gopro stmap verison")
    c_template_vr = p.value("5 Canon stmap verison")
    print "var's assigned from panel"
    
    # Create paths for render directory if it does not exist
    if not os.path.isdir(dnx_render_dir):
        os.mkdir(dnx_render_dir)
        print  dnx_render_dir + " directory created"
    if not os.path.isdir(nk_output_dir):
        os.mkdir(nk_output_dir)
        print nk_output_dir + " directory created"
    

I am at a loss on how to best read the xml file. All the tutorials I have seen
on both `DOM` and `elementtree` are very basic and deal with direct code to
read known XML tags and break data down to a simple `str` output.

I need to enter variables, which then constrain the parsing to a specific part
of the tree, and go into an unknown hierarchy setup and seeing what is inside,
and then make decisions on what to do with what it finds.

Here is a sample of my test XML file. The eventual plan is to have other
different roll types that reference different camera types but for now I'm
just working with 3 camera red rigs.

# It's a very big file so here is a pastebin: <http://pastebin.com/vLaRA0X8>

Basically I am wanting to constrain the script to looking within my variable
`<bin><name>'daynumber'</name>~~~~</bin>.` In this case looking in the
`Day_00` bin. If there is anything else in the root hierarchy I want to ignore
it as sequences, unused clips and other data can get very very huge. I then
want to create the directory of `daynumber` in the `nk_output_dir` &
`dnx_render_dir` so that everything for this shoot day is contained in that
folder.

A annoying part of the XML file is the name of a bin is a child to the `<bin>`
itself, so once a bin name is found, any `<children>` of that bin would be the
same level of the tree as the `<name>`. I can't find sample code of locating a
tag and then looking working with the tags that are in the same branch instead
of it's children.

Now that it has found the bin for the day I want it to start to look for all
the bins in `<children></children>`. Example being
`<bin><name>R001</name>~~~</bin>` and create directories inside the Day_00
folder I made in `nk_output_dir` & `dnx_render_dir` for each bin it finds in
this part of the structure. Every time the camera reloads that will roll up to
R002, R003, etc etc. Also different camera types like Gopros will create G001,
G002, G003.

Then I want to look for in the `<children>` of the above bins and find all the
bins inside like `<bin><name>R001_C001</name>~~~</bin>` and create folders in
the `nk_output_dir\`daynumber`\~whatever bin this is contained~\~name of this
bin~\`. Which is user created of the roll number and clip number. (R001_C001,
R001_C002, etc etc) This will be the new clip name, the name of the .nk comp
that will be generated and the file name of the render on the write node.

The goal here is to recreate the bin folder structure in the directory I've
choosen for `nk_output_dir`.

The `dnx_render_dir` that is for being plugged into the write nodes of my nuke
scripts later to where the files should be rendered to. It's separate because
I'd have a different RAID drive that it will go to that will change as they
fill up. The renders just need to be put in a directory for the
`daynumber\~rollnumber~` but doesn't need to be constrained into a folder for
the clipname.

Here is where I am really lost. Now, because I have to account for user error,
I can't be entirely sure how deep in the tree I need to be going. I know I
want the `<pathurl>~</pathurl>` which I can plug into the .nk (nuke) scripts I
make. With red camera files they can either be the directly here .R3D or the
folder structure which can been 2-3 bins deep. I know that I can't 100% rely
on the guys on set to be consistent on how they make this bin.

All I can trust them to do is make sure they are in correct alphabetic order.
If you look at the xml so the order of them is important. I also know if I am
looking at a R### roll bin that I need 3 `<pathurl></pathurl>` and if im
looking inside G### I need 6 and for C### only 5.

The order of them is important as they can rename the name tag inside `~~~~ to
rename cameras that were the wrong setting without renaming source files.
(which breaks important metadata that is needed in other programs)

While in this part of the tree I'd also like to grab the `<clip
id=~><in>###</in>` to grab the in marker frame offset. If the cameras have
gone out of sync and their start points can be set. But of course this tag is
not child to the `<pathurl></pathurl>` and is actually 3 parents up! Also this
tag won't be on every clip so I can't look for it first!

    
    
       <clip id="masterclip-40" explodedTracks="true" frameBlend="FALSE">
        <uuid>85f87acc-308f-401e-bf82-55e8ea41e55a</uuid>
        <masterclipid>masterclip-40</masterclipid>
        <ismasterclip>TRUE</ismasterclip>
        <duration>5355</duration>
        <rate>
            <timebase>30</timebase>
            <ntsc>TRUE</ntsc>
        </rate>
        <in>876</in>
        <name>B002_C002_0216AM_002.R3D</name>
        <media>
        <video>
            <track>
                <clipitem id="clipitem-118" frameBlend="FALSE">
                    <masterclipid>masterclip-40</masterclipid>
                    <name>B002_C002_0216AM_002.R3D</name>
                    <rate>
                        <timebase>30</timebase>
                        <ntsc>TRUE</ntsc>
                    </rate>
                    <alphatype>none</alphatype>
                    <pixelaspectratio>square</pixelaspectratio>
                    <anamorphic>FALSE</anamorphic>
                    <file id="file-40">
                        <name>B002_C002_0216AM_002.R3D</name>
                        <pathurl>file://localhost/Volumes/REDLAB_3A/SESQUI/MASTER_FILES/DAY_00/RED/R002/B002/B002_0216G4.RDM/B002_C002_0216AM.RDC/B002_C002_0216AM_002.R3D</pathurl>
    

So once I've parsed all this the information I'd like to have is.

  * The original bin folder structure of the XML contained in the `daynumber`. Take the names of the bins and construct the same folder structure in the `nk_output_dir` (Day_00/R001/R001_C001 etc etc) 
  * I also want to make a `daynumber` directory in the `dnx_render_dir` folder and a directory for each bin referencing a camera roll.
  * Based on if the clipname is starts with a R, G or C I want to be able to access that for selecting what kind of .nk to make.
  * I want the pathurl information for each bin that is referring to a clip and plug. I also want any `<in>` information if there is any for that clip. That way I can plug it into the read node information for my nuke gizmo. 

I think once I figure out how to parse such a complicated xml tree I'll able
to fuss and fumble the rest of the process.

I am just really struggling with finding examples of parsing an complicated
XML file like this.


Answer: Whenever faced with a complex XML, consider an XSLT script to transform your
XML into a simpler structure. As information,
[XSLT](https://www.w3.org/TR/xslt) is a special-purpose, declarative language
(same type as SQL) designed to transform XML into various structures for end
use needs. Python like other general purpose languages maintains an XSLT
processor, specifically in its [lxml](http://lxml.de/) module.

While this transformation does not address your entire needs, you can parse
the simpler structure for your Nuke application needs. Directories and names
are simplified and labeled for daynumber, rollnumber, shotnames, and clip with
pathurls.

**XSLT** script _(save as .xsl or .xslt to be referenced in .py script below)_

    
    
    <xsl:transform xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
    <xsl:output version="1.0" encoding="UTF-8" indent="yes" />
    <xsl:strip-space elements="*"/>
    
    <xsl:key name="idkey" match="ctype" use="@id" />
    
      <xsl:template match="/">
        <root>
          <xsl:apply-templates select="*"/>
        </root>
      </xsl:template>
    
      <xsl:template match="xmeml/bin">
        <daynumber><xsl:value-of select="name"/></daynumber>
        <xsl:apply-templates select="children/bin"/>
      </xsl:template>
    
      <xsl:template match="xmeml/bin/children/bin">
        <roll>
          <rollnumber><xsl:value-of select="name"/></rollnumber>
          <rollnumberdir><xsl:value-of select="concat(ancestor::bin/name, 
                                                      '/', name)"/></rollnumberdir>
          <xsl:apply-templates select="children/bin"/>
        </roll>
      </xsl:template>
    
      <xsl:template match="xmeml/bin/children/bin/children/bin">
        <shot>
          <shotname><xsl:value-of select="name"/></shotname>
          <shotnamedir><xsl:value-of select="concat(/xmeml/bin/name, '/', 
                                   /xmeml/bin/children/bin/name, '/', name)"/></shotnamedir>
          <xsl:apply-templates select="descendant::clip[position() &lt; 4]"/>
        </shot>
      </xsl:template>
    
      <xsl:template match="clip">
        <clip>
          <clipname><xsl:value-of select="descendant::name"/></clipname>
          <xsl:copy-of select="in"/>
          <pathurl><xsl:value-of select="descendant::pathurl"/></pathurl>
        </clip>
      </xsl:template>
    
    </xsl:transform>
    

**Python** script _(transform, parse, and export simpler structure)_

    
    
    #!/usr/bin/python
    import lxml.etree as ET
    
    # LOAD INPUT XML AND XSLT
    dom = ET.parse('Input.xml'))
    xslt = ET.parse('XSLTScript.xsl')
    
    # TRANSFORM XML (SIMPLER NEWDOM CAN BE FURTHER PARSED: ITER(), FINDALL(), XPATH())
    transform = ET.XSLT(xslt)
    newdom = transform(dom)
    
    # XPATH EXPRESSIONS (LIST OUTPUTS)
    daynumber = newdom.xpath('//daynumber/text()')
    # ['Day_00']
    rolls = newdom.xpath('//rollnumber/text()')
    # ['R001', 'R002']
    shots = newdom.xpath('//shotname/text()')
    # ['R001_C001', 'R002_C001', 'R002_C002']
    
    # CONVERT TO STRING (IF NEEDED)
    tree_out = ET.tostring(newdom, encoding='UTF-8', pretty_print=True,  xml_declaration=True)
    print(tree_out.decode("utf-8"))
    
    # OUTPUT TO FILE (IF NEEDED)
    xmlfile = open('Output.xml'),'wb')
    xmlfile.write(tree_out)
    xmlfile.close()
    

**TRANSFORMED XML** _(contained in newdom object in .py script)_

    
    
    <?xml version='1.0' encoding='UTF-8'?>
    <root>
      <daynumber>Day_00</daynumber>
      <roll>
        <rollnumber>R001</rollnumber>
        <rollnumberdir>Day_00/R001</rollnumberdir>
        <shot>
          <shotname>R001_C001</shotname>
          <shotnamedir>Day_00/R001/R001_C001</shotnamedir>
          <clip>
            <clipname>A002_C001_0216MW_001.R3D</clipname>
            <pathurl>file://localhost/Volumes/REDLAB_3A/SESQUI/MASTER_FILES/DAY_00/RED/R001/A002/A002_0216FE.RDM/A002_C001_0216MW.RDC/A002_C001_0216MW_001.R3D</pathurl>
          </clip>
          <clip>
            <clipname>A002_C001_0216MW_002.R3D</clipname>
            <pathurl>file://localhost/Volumes/REDLAB_3A/SESQUI/MASTER_FILES/DAY_00/RED/R001/A002/A002_0216FE.RDM/A002_C001_0216MW.RDC/A002_C001_0216MW_002.R3D</pathurl>
          </clip>
          <clip>
            <clipname>A002_C001_0216MW_003.R3D</clipname>
            <pathurl>file://localhost/Volumes/REDLAB_3A/SESQUI/MASTER_FILES/DAY_00/RED/R001/A002/A002_0216FE.RDM/A002_C001_0216MW.RDC/A002_C001_0216MW_003.R3D</pathurl>
          </clip>
        </shot>
      </roll>
      <roll>
        <rollnumber>R002</rollnumber>
        <rollnumberdir>Day_00/R002</rollnumberdir>
        <shot>
          <shotname>R002_C001</shotname>
          <shotnamedir>Day_00/R001/R002_C001</shotnamedir>
          <clip>
            <clipname>A003_C001_0216XI_001.R3D</clipname>
            <pathurl>file://localhost/Volumes/REDLAB_3A/SESQUI/MASTER_FILES/DAY_00/RED/R002/A003/A003_0216XO.RDM/A003_C001_0216XI.RDC/A003_C001_0216XI_001.R3D</pathurl>
          </clip>
          <clip>
            <clipname>B002_C001_02169H_002.R3D</clipname>
            <pathurl>file://localhost/Volumes/REDLAB_3A/SESQUI/MASTER_FILES/DAY_00/RED/R002/B002/B002_0216G4.RDM/B002_C001_02169H.RDC/B002_C001_02169H_002.R3D</pathurl>
          </clip>
          <clip>
            <clipname>C002_C001_02168R_001.R3D</clipname>
            <pathurl>file://localhost/Volumes/REDLAB_3A/SESQUI/MASTER_FILES/DAY_00/RED/R002/C002/C002_0216RL.RDM/C002_C001_02168R.RDC/C002_C001_02168R_001.R3D</pathurl>
          </clip>
        </shot>
        <shot>
          <shotname>R002_C002</shotname>
          <shotnamedir>Day_00/R001/R002_C002</shotnamedir>
          <clip>
            <clipname>C002_C002_0216M9_001.R3D</clipname>
            <pathurl>file://localhost/Volumes/REDLAB_3A/SESQUI/MASTER_FILES/DAY_00/RED/R002/C002/C002_0216RL.RDM/C002_C002_0216M9.RDC/C002_C002_0216M9_001.R3D</pathurl>
          </clip>
          <clip>
            <clipname>C002_C002_0216M9_002.R3D</clipname>
            <pathurl>file://localhost/Volumes/REDLAB_3A/SESQUI/MASTER_FILES/DAY_00/RED/R002/C002/C002_0216RL.RDM/C002_C002_0216M9.RDC/C002_C002_0216M9_002.R3D</pathurl>
          </clip>
          <clip>
            <clipname>C002_C002_0216M9_003.R3D</clipname>
            <pathurl>file://localhost/Volumes/REDLAB_3A/SESQUI/MASTER_FILES/DAY_00/RED/R002/C002/C002_0216RL.RDM/C002_C002_0216M9.RDC/C002_C002_0216M9_003.R3D</pathurl>
          </clip>
        </shot>
      </roll>
    </root>
    

