Flattening XML and selecting nodes based on input to convert to CSV I have some XML that I want to flatten based on its XPath. The task is to acquire certain nodes that are passed as input. That said, I have to get to parent nodes and look for nodes that might be part of the configuration. I am avoiding lists in a given iteration.

This is my code:

from lxml import etree

tree = etree.parse("sample.xml")
path = "/DATA_DS/G_1"


def getNodesDict(node_element, nodes_to_pick):
    some_dict = {}
    for element in node_element:
        if not isinstance(element, list):
            if element.tag in nodes_to_pick:
                some_dict[element.tag] = element.text

    return some_dict


def fetchNodesListBasedOnXpath(Xpath, tree, selected_nodes):
    """

    :param Xpath:
    :param tree:
    :param selected_nodes:
    :return:
    """
    temp_list = []
    temp_list1 = []
    element_split = Xpath.split('/')
    for i in range(len(element_split), 1, -1):
        path = "/".join(element_split[0:i])
        try:
            element = tree.xpath(path)
            for j, node_element in enumerate(element):
                value = getNodesDict(node_element, selected_nodes)
                if len(temp_list) &gt; 0 and len(temp_list) - 1 &gt; j:
                    for k, item in enumerate(temp_list):
                        dall = {}
                        dall.update(value)
                        dall.update(item)

                        if item:
                            temp_list1.append(dall)
                    temp_list = []
                else:
                    temp_list.append(value)
        except:
            pass


nodes_to_pick = ['SUBSCRIPTION_ID', "PRODUCT_NUMBER", "SUBSCRIPTION_ID1", "SALES_PRODUCT_TYPE1"]

fetchNodesListBasedOnXpath(path, tree, nodes_to_pick)


This is my sample XML:

&lt;DATA_DS&gt;
    &lt;SUBSCRIPTION_ID1&gt;300000031070490&lt;/SUBSCRIPTION_ID1&gt;
    &lt;SALES_PRODUCT_TYPE1&gt;SUBSCRIPTION&lt;/SALES_PRODUCT_TYPE1&gt;
    &lt;G_1&gt;
        &lt;SUBSCRIPTION_ID&gt;300000031070490&lt;/SUBSCRIPTION_ID&gt;
        &lt;PRODUCT_NUMBER&gt;1&lt;/PRODUCT_NUMBER&gt;
        &lt;SALES_PRODUCT_TYPE&gt;SUBSCRIPTION&lt;/SALES_PRODUCT_TYPE&gt;
        &lt;PRODUCT_NAME&gt;Demo4&lt;/PRODUCT_NAME&gt;
        &lt;STATUS&gt;ORA_EXPIRED&lt;/STATUS&gt;
        &lt;START_DATE&gt;2019-06-01T00:00:00.000+00:00&lt;/START_DATE&gt;
        &lt;END_DATE&gt;2019-06-01T00:00:00.000+00:00&lt;/END_DATE&gt;
    &lt;/G_1&gt;

    &lt;G_1&gt;
        &lt;SUBSCRIPTION_ID&gt;300000031070491&lt;/SUBSCRIPTION_ID&gt;
        &lt;PRODUCT_NUMBER&gt;4&lt;/PRODUCT_NUMBER&gt;
        &lt;SALES_PRODUCT_TYPE&gt;SUBSCRIPTION&lt;/SALES_PRODUCT_TYPE&gt;
        &lt;PRODUCT_NAME&gt;Demo4&lt;/PRODUCT_NAME&gt;
        &lt;STATUS&gt;ORA_ACTIVE&lt;/STATUS&gt;
        &lt;START_DATE&gt;2019-06-01T00:00:00.000+00:00&lt;/START_DATE&gt;
        &lt;END_DATE&gt;2020-05-31T00:00:00.000+00:00&lt;/END_DATE&gt;
    &lt;/G_1&gt;
    &lt;G_1&gt;
        &lt;SUBSCRIPTION_ID&gt;300000031070492&lt;/SUBSCRIPTION_ID&gt;
        &lt;PRODUCT_NUMBER&gt;2&lt;/PRODUCT_NUMBER&gt;
        &lt;SALES_PRODUCT_TYPE&gt;SUBSCRIPTION&lt;/SALES_PRODUCT_TYPE&gt;
        &lt;PRODUCT_NAME&gt;Demo4&lt;/PRODUCT_NAME&gt;
        &lt;STATUS&gt;ORA_CLOSED&lt;/STATUS&gt;
        &lt;START_DATE&gt;2019-06-01T00:00:00.000+00:00&lt;/START_DATE&gt;
        &lt;END_DATE&gt;2019-06-30T00:00:00.000+00:00&lt;/END_DATE&gt;
    &lt;/G_1&gt;
    &lt;G_1&gt;
        &lt;SUBSCRIPTION_ID&gt;300000031070493&lt;/SUBSCRIPTION_ID&gt;
        &lt;PRODUCT_NUMBER&gt;3&lt;/PRODUCT_NUMBER&gt;
        &lt;SALES_PRODUCT_TYPE&gt;SUBSCRIPTION&lt;/SALES_PRODUCT_TYPE&gt;
        &lt;PRODUCT_NAME&gt;Demo4&lt;/PRODUCT_NAME&gt;
        &lt;STATUS&gt;ORA_ACTIVE&lt;/STATUS&gt;
        &lt;START_DATE&gt;2019-06-01T00:00:00.000+00:00&lt;/START_DATE&gt;
        &lt;END_DATE&gt;2019-08-31T00:00:00.000+00:00&lt;/END_DATE&gt;
    &lt;/G_1&gt;
&lt;/DATA_DS&gt;
Docstrings

You should include a docstring at the beginning of every module, class, and function you write. This will allow documentation to identify what your code is supposed to do. Docstrings are great for the understanding the functionality of the larger part of the code, i.e., the general purpose of any class, module or function. Comments are used for code, statement, and expressions which tend to be small.

Function Naming

You have these two function names:

fetchNodesListBasedOnXpath
getNodesDict


According to PEP-8's Guidelines for function names, these function names  should be snake_case. 

_ for unused loop variables

When you don't use a variable in a loop, like:

for i in range(5):
    print("Hi!")


You should use a _. This makes it clear that the variable used for the loop is not needed, and should be ignored.

Use of if len(...)


  For sequences, (strings, lists, tuples), use the fact that empty sequences are false.

Yes: if not seq:
     if seq:

No:  if len(seq):
     if not len(seq):



Essentially, if seq will return True if it's not empty, and if not seq will return True if it is empty. Since you are checking if len(temp_list) &gt; 0 ..., this should be changed to if temp_list ....

Constant Variable Naming

According to PEP8s rule on constants:


  Constants are usually defined on a module level and written in all capital letters with underscores separating words. Examples include MAX_OVERFLOW and TOTAL.


Long Version: 

In the Python community (as in many other communities) exist conventions about how to write code. This is different from working code: even if you write your constants all lowercase, your code still works. 

But there is community consensus (as documented in PEP8) which is "enforced" with tools like pylint. If you program for your own happiness, you may neglect the hints pylint gives you. If you want open exchange with the community, aka »someone besides myself should use my code«, you should prepare your code according to PEP8. [source

Main Guard

Having a main guard clause in a module allows you to both run code in the module directly and also use procedures and classes in the module from other modules. Without the main guard clause, the code to start your script would get run when the module is imported. This can cause a plethora of issues if you want to reuse this code. Wrapping it in a main guard (check updated code) protects against this.

Parameter Naming

Parameters, variables, and default parameters all follow the same PEP-8 Guidelines for naming. They should all be snake_case. You currently have a parameter named Xpath. A simple change to xpath will do the trick.

Catching Errors/Exceptions

Right now, your main function can catch an exception, but doesn't handle it. If your program suddenly fails, a detailed error message would do the trick. You should catch specific errors that you think your code will generate. Since you're dealing with files and paths, catching FileNotFoundError is a good idea. Then, after dealing with all the specific errors, you can use a finally. Since this will always be run, you can put your pass here.

Updated Code

"""
Module Docstring
A description of your program goes here
"""

from lxml import etree

def get_nodes_dict(node_element, nodes_to_pick):
    """
    A description of this method should go here
    """
    some_dict = {}
    for element in node_element:
        if not isinstance(element, list):
            if element.tag in nodes_to_pick:
                some_dict[element.tag] = element.text

    return some_dict


def fetch_nodes_list_based_on_xpath(xpath, tree, selected_nodes):
    """

    A description of this method should go here

    :param xpath:
    :param tree:
    :param selected_nodes:
    :return:
    """
    temp_list = []
    temp_list1 = []
    element_split = xpath.split('/')
    for i in range(len(element_split), 1, -1):
        path = "/".join(element_split[0:i])
        try:
            element = tree.xpath(path)
            for j, node_element in enumerate(element):
                value = get_nodes_dict(node_element, selected_nodes)
                if temp_list and len(temp_list) - 1 &gt; j:
                    for _, item in enumerate(temp_list):
                        dall = {}
                        dall.update(value)
                        dall.update(item)
                        if item:
                            temp_list1.append(dall)
                    temp_list = []
                else:
                    temp_list.append(value)
        except FileNotFoundError as _:
            print(f"The file at [{PATH}] doesn't exist!")
        finally:
            pass

if __name__ == '__main__':

    TREE = etree.parse("code/sample.xml")
    PATH = "/DATA_DS/G_1"

    NODES_TO_PICK = ['SUBSCRIPTION_ID', "PRODUCT_NUMBER", "SUBSCRIPTION_ID1", "SALES_PRODUCT_TYPE1"]

    fetch_nodes_list_based_on_xpath(PATH, TREE, NODES_TO_PICK)