title: batch script - to remove duplicate tokens in file
body: <p>I have duplicate tokens in text file I would like to create new text file without the duplicate tokens (keeping the delimiters)</p>
  
  <p>The delimiter is:~@^*^@~
  <BR>example file:</p>
  
  <pre><code>aaa~@^*^@~bbb~@^*^@~aaa~@^*^@~bbb~@^*^@~aaa~@^*^@~bbb~@^*^@~aaa~@^*^@~bbb~@^*^@~aaa~@^*^@~bbb~@^*^@~xxx~@^*^@~bbb~@^*^@~aaa~@^*^@~bbb~@^*^@~aaa~@^*^@~bbb~@^*^@~aaa~@^*^@~bbb~@^*^@~aaa~@^*^@~bbb~@^*^@~aaa~@^*^@~bbb~@^*^@~aaa~@^*^@~bbb~@^*^@~aaa~@^*^@~bbb~@^*^@~aaa~@^*^@~bbb~@^*^@~aaa~@^*^@~bbb
  </code></pre>
  
  <p>Result should be: </p>
  
  <pre><code>aaa~@^*^@~bbb~@^*^@~xxx
  </code></pre>
  
  <p>I <a href="http://www.computing.net/answers/programming/remove-duplicate-lines-in-pure-bat/15777.html" rel="nofollow">found script</a> that remove duplicate lines:</p>
  
  <pre><code>==================================
  @echo off &gt; outfile
  if %1'==' echo which file? &amp;&amp; goto :eof
  if not exist %1 echo %1 not found &amp;&amp; goto :eof
  
  for /f "tokens=* delims= " %%a in (%1) do (
  find "%%a" &lt; outfile &gt; nul
      if errorlevel 1 echo %%a &gt;&gt; outfile
  )
  </code></pre>
  
  <p>The script work nice for duplicate lines,
  <BR>So i modified the delims from:</p>
  
  <pre><code> "tokens=* delims="
  </code></pre>
  
  <p>to </p>
  
  <pre><code>"tokens=* delims=~@^*^@~"
  </code></pre>
  
  <p>But it wont work, What am i doing wrong? is one of the delimiter characters reserved word?
  <BR>Thank you for any suggestion.</p>