How to remove duplicate nodes xml Python
Question: I have a special case xml file structure is something like :

    
    
    <Root>
        <parent1>
             <parent2>
                 <element id="Something" >
             </parent2>
         </parent1>
         <parent1>
             <element id="Something">
         </parent1>
    </Root>
    

My use case is to remove the duplicated element , I want to remove the
elements with same Id . I tried the following code with no positive outcome
(its not finding the duplicate node)

    
    
    import xml.etree.ElementTree as ET
    
    path = 'old.xml'
    
    tree = ET.parse(path)
    root = tree.getroot()
    prev = None
    
    def elements_equal(e1, e2):
        if type(e1) != type(e2):
            return False
        if e1.tag != e1.tag: return False
        if e1.text != e2.text: return False
        if e1.tail != e2.tail: return False
        if e1.attrib != e2.attrib: return False
        if len(e1) != len(e2): return False
        return all([elements_equal(c1, c2) for c1, c2 in zip(e1, e2)])
    
    for page in root:                     # iterate over pages
    elems_to_remove = []
    for elem in page:
       for insideelem in page:
           if elements_equal(elem, insideelem) and elem != insideelem:
               print("found duplicate: %s" % insideelem.text)   # equal function works well
               elems_to_remove.append(insideelem)
               continue
    
    for elem_to_remove in elems_to_remove:
        page.remove(elem_to_remove)
    # [...]
    tree.write("out.xml")
    

Can someone help me in letting me know how can i solve it. I am very new to
python with almost zero experience .


Answer: First of all what you're doing is a hard problem in the library you're using,
see this question: [How to remove a node inside an iterator in python
xml.etree.ElemenTree](https://stackoverflow.com/questions/19419754/how-to-
remove-a-node-inside-an-iterator-in-python-xml-etree-elementree)

The solution to this would be to use `lxml` which "implements the same API but
with additional enhancements". Then you can do the following fix.

You seem to be only traversing the second level of nodes in your XML tree.
You're getting `root`, then walking the children its children. This would get
you `parent2` from the first page and the `element` from your second page.
Furthermore you wouldn't be comparing across pages here:

**your comparison will only find second-level duplicates within the same
page.**

Select the right set of elements using a proper traversal function such as
`iter`:

    
    
    # Use a `set` to keep track of "visited" elements with good lookup time.
    visited = set()
    # The iter method does a recursive traversal
    for el in root.iter('element'):
        # Since the id is what defines a duplicate for you
        if 'id' in el.attr:
            current = el.get('id')
            # In visited already means it's a duplicate, remove it
            if current in visited:
                el.getparent().remove(el)
            # Otherwise mark this ID as "visited"
            else:
                visited.add(current)
    

