title: Clipboard content is messed up when copied from Firefox and read using Java in Ubuntu
body: <p><strong>Background</strong></p>
  
  <p>I am trying getting clipboard data in HTML data flavor using Java. Thus I copy them into the clipboard from browsers. Then I am using <a href="https://docs.oracle.com/javase/8/docs/api/java/awt/datatransfer/Clipboard.html" rel="noreferrer">java.awt.datatransfer.Clipboard</a> to get them.</p>
  
  <p>This works properly in Windows systems. But in Ubuntu there are some strange issues. The worst is when copied the data into clipboard from Firefox browser.</p>
  
  <p><strong>Example for reproducing the behavior</strong></p>
  
  <p>Java code:</p>
  
  <pre><code>import java.io.*;
  
  import java.awt.Toolkit;
  import java.awt.datatransfer.Clipboard;
  import java.awt.datatransfer.DataFlavor;
  
  public class WorkingWithClipboadData {
  
   static void doSomethingWithBytesFromClipboard(byte[] dataBytes, String paramCharset, int number) throws Exception {
  
    String fileName = "Result " + number + " " + paramCharset + ".txt";
  
    OutputStream fileOut = new FileOutputStream(fileName);
    fileOut.write(dataBytes, 0, dataBytes.length);
    fileOut.close();
  
   }
  
   public static void main(String[] args) throws Exception {
  
    Clipboard clipboard = Toolkit.getDefaultToolkit().getSystemClipboard();
  
    int count = 0;
  
    for (DataFlavor dataFlavor : clipboard.getAvailableDataFlavors()) {
  
  System.out.println(dataFlavor);
  
     String mimeType = dataFlavor.getHumanPresentableName();
     if ("text/html".equalsIgnoreCase(mimeType)) {
      String paramClass = dataFlavor.getParameter("class");
      if ("java.io.InputStream".equals(paramClass)) {
       String paramCharset = dataFlavor.getParameter("charset");
       if (paramCharset != null  &amp;&amp; paramCharset.startsWith("UTF")) {
  
  System.out.println("============================================");
  System.out.println(paramCharset);
  System.out.println("============================================");
  
        InputStream inputStream = (InputStream)clipboard.getData(dataFlavor);
  
        ByteArrayOutputStream data = new ByteArrayOutputStream();
  
        byte[] buffer = new byte[1024];
        int length = -1;
        while ((length = inputStream.read(buffer)) != -1) {
         data.write(buffer, 0, length);
        }
        data.flush();
        inputStream.close();
  
        byte[] dataBytes = data.toByteArray();
        data.close();
  
        doSomethingWithBytesFromClipboard(dataBytes, paramCharset, ++count);
  
       }
      }
     }
    }
   }
  
  }
  </code></pre>
  
  <p><strong>Problem description</strong></p>
  
  <p>What I am doing is, opening URL <a href="https://en.wikipedia.org/wiki/Germanic_umlaut" rel="noreferrer">https://en.wikipedia.org/wiki/Germanic_umlaut</a> in Firefox. Then I do select "letters: ä" there and copy this into clipboard. Then I run my Java program. After that the resulting files (only some of them as examples) looks like this:</p>
  
  <pre><code>axel@arichter:~/Dokumente/JAVA/poi/poi-3.17$ xxd "./Result 1 UTF-16.txt" 
  00000000: feff fffd fffd 006c 0000 0065 0000 0074  .......l...e...t
  00000010: 0000 0074 0000 0065 0000 0072 0000 0073  ...t...e...r...s
  00000020: 0000 003a 0000 0020 0000 003c 0000 0069  ...:... ...&lt;...i
  00000030: 0000 003e 0000 fffd 0000 003c 0000 002f  ...&gt;.......&lt;.../
  00000040: 0000 0069 0000 003e 0000                 ...i...&gt;..
  </code></pre>
  
  <p>OK the <code>FEFF</code> at the start looks like a <code>UTF-16BE</code> byte-order-mark. But what is the <code>FFFD</code>? And why are there those <code>0000</code> bytes between the single letters? <code>UTF-16</code> encoding of <code>l</code> is <code>006C</code> only. Seems as if all letters are encoded in 32 bit. But this is wrong for <code>UTF-16</code>. And all non ASCII charcters are encoded with <code>FFFD 0000</code> and so are lost.</p>
  
  <pre><code>axel@arichter:~/Dokumente/JAVA/poi/poi-3.17$ xxd "./Result 4 UTF-8.txt" 
  00000000: efbf bdef bfbd 6c00 6500 7400 7400 6500  ......l.e.t.t.e.
  00000010: 7200 7300 3a00 2000 3c00 6900 3e00 efbf  r.s.:. .&lt;.i.&gt;...
  00000020: bd00 3c00 2f00 6900 3e00                 ..&lt;./.i.&gt;.
  </code></pre>
  
  <p>Here the <code>EFBF BDEF BFBD</code> does not look like any known byte-order-mark. And all letters seems encoded in 16 bit, which is the double of the needed bits in <code>UTF-8</code>. So the bits used seems always be the double count as needed. See in <code>UTF-16</code> example above. And all not ASCII letters are encoded as <code>EFBFBD</code> and so also are lost.</p>
  
  <pre><code>axel@arichter:~/Dokumente/JAVA/poi/poi-3.17$ xxd "./Result 7 UTF-16BE.txt" 
  00000000: fffd fffd 006c 0000 0065 0000 0074 0000  .....l...e...t..
  00000010: 0074 0000 0065 0000 0072 0000 0073 0000  .t...e...r...s..
  00000020: 003a 0000 0020 0000 003c 0000 0069 0000  .:... ...&lt;...i..
  00000030: 003e 0000 fffd 0000 003c 0000 002f 0000  .&gt;.......&lt;.../..
  00000040: 0069 0000 003e 0000                      .i...&gt;..
  </code></pre>
  
  <p>Same picture as in the examples above. All letters are encoded using 32 bit. Only 16 bit shall be used in <code>UTF-16</code> except the supplementary characters which uses surrogate pairs. And all not ASCII letters are encoded with <code>FFFD 0000</code> and so are lost.</p>
  
  <pre><code>axel@arichter:~/Dokumente/JAVA/poi/poi-3.17$ xxd "./Result 10 UTF-16LE.txt" 
  00000000: fdff fdff 6c00 0000 6500 0000 7400 0000  ....l...e...t...
  00000010: 7400 0000 6500 0000 7200 0000 7300 0000  t...e...r...s...
  00000020: 3a00 0000 2000 0000 3c00 0000 6900 0000  :... ...&lt;...i...
  00000030: 3e00 0000 fdff 0000 3c00 0000 2f00 0000  &gt;.......&lt;.../...
  00000040: 6900 0000 3e00 0000                      i...&gt;...
  </code></pre>
  
  <p>Only for to be complete. Same picture as above.</p>
  
  <p>So the conclusion is that the Ubuntu clipboard is totally messed up after copying something into it from Firefox. At least for HTML data flavors and when reading the clipboard using Java.</p>
  
  <p><strong>Other browser used</strong></p>
  
  <p>When I do the same things using Chromium browser as the source of the data, then the problems becomes smaller.</p>
  
  <p>So I am opening URL <a href="https://en.wikipedia.org/wiki/Germanic_umlaut" rel="noreferrer">https://en.wikipedia.org/wiki/Germanic_umlaut</a> in Chromium. Then I do select "letters: ä" there and copy this into clipboard. Then I run my Java program. </p>
  
  <p>The result looks like:</p>
  
  <pre><code>axel@arichter:~/Dokumente/JAVA/poi/poi-3.17$ xxd "./Result 1 UTF-16.txt" 
  00000000: feff 003c 006d 0065 0074 0061 0020 0068  ...&lt;.m.e.t.a. .h
  ...
  00000800: 0061 006c 003b 0022 003e 00e4 003c 002f  .a.l.;.".&gt;...&lt;./
  00000810: 0069 003e 0000                           .i.&gt;..
  </code></pre>
  
  <p>Chromium has more HTML around the selected in the HTML data flavors in clipboard. But the encoding looks properly. Also for the not ASCII <code>ä</code> = <code>00E4</code>. But there also is a small problem, There are additional bytes <code>0000</code> at the end which should not be there. In <code>UTF-16</code> there are 2 additional <code>00</code> bytes at the end.</p>
  
  <pre><code>axel@arichter:~/Dokumente/JAVA/poi/poi-3.17$ xxd "./Result 4 UTF-8.txt" 
  00000000: 3c6d 6574 6120 6874 7470 2d65 7175 6976  &lt;meta http-equiv
  ...
  000003f0: 696f 6e2d 636f 6c6f 723a 2069 6e69 7469  ion-color: initi
  00000400: 616c 3b22 3ec3 a43c 2f69 3e00            al;"&gt;..&lt;/i&gt;.
  </code></pre>
  
  <p>Same as above. Encoding looks properly for <code>UTF-8</code>. But here also is one additional <code>00</code> byte at the end which not should be there.</p>
  
  <p><strong>Environment</strong></p>
  
  <pre><code>DISTRIB_ID=Ubuntu
  DISTRIB_RELEASE=16.04
  DISTRIB_CODENAME=xenial
  DISTRIB_DESCRIPTION="Ubuntu 16.04.4 LTS"
  
  
  Mozilla Firefox 61.0.1 (64-Bit)
  
  
  java version "1.8.0_101"
  Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
  Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
  </code></pre>
  
  <p><strong>Questions</strong></p>
  
  <p>Am I doing something wrong in my code?</p>
  
  <p>Can someone advise how to avoid that messed up content in clipboard? Since the not ASCII characters are lost, at least when copied from Firefox, I don't think that we can repair this content.</p>
  
  <p>Is this a known issue somehow? Can someone confirm the same behavior? If so, is  there already a bug report in Firefox about this?</p>
  
  <p>Or is this a problem which only occurs if Java code reads the clipboard content? Seems as if. Because if I copy content from Firefox and paste it in Libreoffice Writer then Unicode appears properly.  And if I then copy content from Writer to the clipboard and do reading it using my Java program, then <code>UTF</code> encodings are correct except the additional <code>00</code> bytes at the end. So clipboard content copied from Writer behaves like content copied from Chromium browser.</p>
  
  <hr>
  
  <p><strong>New insights</strong></p>
  
  <p>The bytes <code>0xFFFD</code> seems to be Unicode Character 'REPLACEMENT CHARACTER' (U+FFFD). So the <code>0xFDFF</code> is the little endian representation of this and the <code>0xEFBFBD</code> is the UTF-8 encoding of this. So all results seems to be results of wrong decoding and re encoding Unicode.</p>
  
  <p>Seems as it the clipboard content coming from Firefox is <code>UTF-16LE</code> with <code>BOM</code> always. But then <code>Java</code> gets this as <code>UTF-8</code>. So the 2 byte BOM becomes two messed up characters, which are replaced with 0xEFBFBD, each additional <code>0x00</code> sequence becomes their own <code>NUL</code> characters and all byte sequences which are not proper <code>UTF-8</code> byte sequences becomes messed up characters, which are replaced with 0xEFBFBD. Then this pseudo UTF-8 will be re encoded. Now the garbage is complete.</p>
  
  <p>Example:</p>
  
  <p>The sequence <code>aɛaüa</code> in UTF-16LE with BOM will be 
  <code>0xFFFE 6100 5B02 6100 FC00 6100</code>. </p>
  
  <p>This taken as UTF-8 (0xEFBFBD = not a proper UTF-8 byte sequence) = 
  0xEFBFBD 0xEFBFBD <code>a</code> <code>NUL</code> <code>[</code> <code>STX</code> <code>a</code> <code>NUL</code> 0xEFBFBD <code>NUL</code> <code>a</code> <code>NUL</code>. </p>
  
  <p>This pseudo ASCII re encoded to UTF-16LE will be:
  <code>0xFDFF FDFF 6100 0000 5B00 0200 6100 0000 FDFF 0000 6100 0000</code></p>
  
  <p>This pseudo ASCII re encoded to UTF-8 will be 
  <code>0xEFBF BDEF BFBD 6100 5B02 6100 EFBF BD00 6100</code></p>
  
  <p>And this is exactly what happens.</p>
  
  <p>Other examples: </p>
  
  <p><code>Â</code> = 0x00C2 = <code>C200</code> in UTF-16LE = 0xEFBFBD00 in pseudo UTF-8</p>
  
  <p><code>胂</code> = 0x80C2 = <code>C280</code> in UTF-16LE = 0xC280 in pseudo UTF-8</p>
  
  <p>So I think <code>Firefox</code> is not to blame for this but either <code>Ubuntu</code> or <code>Java</code>'s runtime environment. And because copy/paste from Firefox to Writer works in Ubuntu, I think <code>Java</code>'s runtime environment does not handle the Firefox data flavors in <code>Ubuntu</code> clipboard correctly.</p>
  
  <hr>
  
  <p><strong>New insights:</strong></p>
  
  <p>I have compared the <code>flavormap.properties</code> files of my <code>Windows 10</code> and my <code>Ubuntu</code> and there is a difference. In <code>Ubuntu</code> the native name of the <code>text/html</code> is <code>UTF8_STRING</code> while in <code>Windows</code> it is <code>HTML Format</code>. So I thought that this may be the problem. So I've added the line</p>
  
  <p><code>HTML\ Format=text/html;charset=utf-8;eoln="\n";terminators=0</code></p>
  
  <p>to my <code>flavormap.properties</code> file in <code>Ubuntu</code>.</p>
  
  <p>After that:</p>
  
  <pre><code>Map&lt;DataFlavor,String&gt; nativesForFlavors = SystemFlavorMap.getDefaultFlavorMap().getNativesForFlavors(
     new DataFlavor[]{
     new DataFlavor("text/html;charset=UTF-16LE")
     });
  
  System.out.println(nativesForFlavors);
  </code></pre>
  
  <p>prints</p>
  
  <pre><code>{java.awt.datatransfer.DataFlavor[mimetype=text/html;representationclass=java.io.InputStream;charset=UTF-16LE]=HTML Format}
  </code></pre>
  
  <p>But no changes in the results of the Ubuntu clipboard content when read by Java.</p>