title: XML XSLT remove duplicate elements based on some, but not all, of sub-elements
body: <p>After backingup up my Windows Phone to a .msg archive (.xml in essence), I have come upon duplicate entries. I tidied up my XML to be better readable, and I will give you 6 entries (as 3 duplicate outgoing messages and 3 incoming duplicate messages) out of thousands of entries:</p>
  
  <pre><code>&lt;?xml version="1.0" encoding="utf-8"?&gt;
  &lt;ArrayOfMessage xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"&gt;
      &lt;Message&gt;
          &lt;Recepients&gt;
              &lt;string&gt;OutgoingNUMB&lt;/string&gt;
          &lt;/Recepients&gt;
          &lt;Body&gt;OutgoingMESG&lt;/Body&gt;
          &lt;IsIncoming&gt;false&lt;/IsIncoming&gt;
          &lt;IsRead&gt;true&lt;/IsRead&gt;
          &lt;Attachments /&gt;
          &lt;LocalTimestamp&gt;130000350501770000&lt;/LocalTimestamp&gt;
          &lt;Sender /&gt;
      &lt;/Message&gt;
      &lt;Message&gt;
          &lt;Recepients&gt;
              &lt;string&gt;OutgoingNUMB&lt;/string&gt;
          &lt;/Recepients&gt;
          &lt;Body&gt;OutgoingMESG&lt;/Body&gt;
          &lt;IsIncoming&gt;false&lt;/IsIncoming&gt;
          &lt;IsRead&gt;true&lt;/IsRead&gt;
          &lt;Attachments /&gt;
          &lt;LocalTimestamp&gt;130000350501770000&lt;/LocalTimestamp&gt;
          &lt;Sender /&gt;
      &lt;/Message&gt;
      &lt;Message&gt;
          &lt;Recepients&gt;
              &lt;string&gt;OutgoingNUMB&lt;/string&gt;
          &lt;/Recepients&gt;
          &lt;Body&gt;OutgoingMESG&lt;/Body&gt;
          &lt;IsIncoming&gt;false&lt;/IsIncoming&gt;
          &lt;IsRead&gt;true&lt;/IsRead&gt;
          &lt;Attachments /&gt;
          &lt;LocalTimestamp&gt;130000350501760000&lt;/LocalTimestamp&gt;
          &lt;Sender /&gt;
      &lt;/Message&gt;
      &lt;Message&gt;
          &lt;Recepients /&gt;
          &lt;Body&gt;IncomingMESG&lt;/Body&gt;
          &lt;IsIncoming&gt;true&lt;/IsIncoming&gt;
          &lt;IsRead&gt;true&lt;/IsRead&gt;
          &lt;Attachments /&gt;
          &lt;LocalTimestamp&gt;130000349290000000&lt;/LocalTimestamp&gt;
          &lt;Sender&gt;IncomingNUMB&lt;/Sender&gt;
      &lt;/Message&gt;
      &lt;Message&gt;
          &lt;Recepients /&gt;
          &lt;Body&gt;IncomingMESG&lt;/Body&gt;
          &lt;IsIncoming&gt;true&lt;/IsIncoming&gt;
          &lt;IsRead&gt;true&lt;/IsRead&gt;
          &lt;Attachments /&gt;
          &lt;LocalTimestamp&gt;130000349234630000&lt;/LocalTimestamp&gt;
          &lt;Sender&gt;IncomingNUMB&lt;/Sender&gt;
      &lt;/Message&gt;
      &lt;Message&gt;
          &lt;Recepients /&gt;
          &lt;Body&gt;IncomingMESG&lt;/Body&gt;
          &lt;IsIncoming&gt;true&lt;/IsIncoming&gt;
          &lt;IsRead&gt;true&lt;/IsRead&gt;
          &lt;Attachments /&gt;
          &lt;LocalTimestamp&gt;130000349234630000&lt;/LocalTimestamp&gt;
          &lt;Sender&gt;IncomingNUMB&lt;/Sender&gt;
      &lt;/Message&gt;
  &lt;/ArrayOfMessage&gt;
  </code></pre>
  
  <p>We have 3 outgoing messages that are essentially the same 1, as well as 3 incoming messages, which are essentially the same 1.
  The outgoing message format differs from the incoming format, but it actually doesn't matter. Notice that on the duplicates, I may have different "LocalTimestamp", as the phone may have messed it up. Therefore, LocalTimestamp should not be a filter for checking a message if it's a duplicate.</p>
  
  <p>What I want to do is the following:</p>
  
  <p>I want a XSLT to remove those duplicate entries, so that I only have the following:</p>
  
  <pre><code>&lt;?xml version="1.0" encoding="utf-8"?&gt;
  &lt;ArrayOfMessage xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"&gt;
      &lt;Message&gt;
          &lt;Recepients&gt;
              &lt;string&gt;OutgoingNUMB&lt;/string&gt;
          &lt;/Recepients&gt;
          &lt;Body&gt;OutgoingMESG&lt;/Body&gt;
          &lt;IsIncoming&gt;false&lt;/IsIncoming&gt;
          &lt;IsRead&gt;true&lt;/IsRead&gt;
          &lt;Attachments /&gt;
          &lt;LocalTimestamp&gt;130000350501770000&lt;/LocalTimestamp&gt;
          &lt;Sender /&gt;
      &lt;/Message&gt;
      &lt;Message&gt;
          &lt;Recepients /&gt;
          &lt;Body&gt;IncomingMESG&lt;/Body&gt;
          &lt;IsIncoming&gt;true&lt;/IsIncoming&gt;
          &lt;IsRead&gt;true&lt;/IsRead&gt;
          &lt;Attachments /&gt;
          &lt;LocalTimestamp&gt;130000349234630000&lt;/LocalTimestamp&gt;
          &lt;Sender&gt;IncomingNUMB&lt;/Sender&gt;
      &lt;/Message&gt;
  &lt;/ArrayOfMessage&gt;
  </code></pre>
  
  <p>The filter should check for duplicate "Message" tags, containing the same children ("Recepients" with or without "string" where applicable, "Body", "IsIncoming", "Attachments", "Sender"), ignoring the tags "IsRead" and "LocalTimestamp", but using the oldest (i.e. lowest number) "LocalTimestamp" as the one to keep, and tossing out the rest as duplicates.</p>
  
  <p>I have the feeling that it's possible to have only one filter (regardless of incoming or outgoing message structure), but it's up to you.</p>
  
  <p>Thanks for your help in advance.</p>
  
  <p>======UPDATE</p>
  
  <p>I studied and came up with this (Currently a large database is being checked for bugs)...</p>
  
  <pre><code>&lt;xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"&gt;
  &lt;xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes"/&gt;
  &lt;!--
  --&gt;
  &lt;xsl:strip-space elements="*"/&gt;
  &lt;xsl:template match="@*|node()"&gt;
  &lt;xsl:copy&gt;
       &lt;xsl:apply-templates select="@*|node()" /&gt;
  &lt;/xsl:copy&gt;
  &lt;/xsl:template&gt;
  &lt;xsl:template match="Message[
  (Recepients = following::Message/Recepients) 
  and 
  (Body = following::Message/Body) 
  and 
  (IsIncoming = following::Message/IsIncoming) 
  and 
  (Attachments = following::Message/Attachments) 
  and 
  (Sender = following::Message/Sender)
  ]"/&gt;  
  &lt;/xsl:stylesheet&gt;
  </code></pre>