<p>Is there an easy way to avoid dealing with text encoding problems?</p>

[{'answer_id': 62255, 'author': 'Tom Hawtin - tackline', 'author_id': 4725, 'author_profile': 'https://Stackoverflow.com/users/4725', 'pm_score': 3, 'selected': False, 'text': '<p>The obvious names for these classes are ReaderInputStream and WriterOutputStream. Unfortunately these are not included in the Java library. However, google is your friend.</p>\n\n<p>I\'m not sure that it is going to get around all text encoding problems, which are nightmarish.</p>\n\n<p><a href="http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4103785" rel="noreferrer">There is an RFE,</a> but it\'s Closed, will not fix.</p>\n'}, {'answer_id': 62257, 'author': 'Peter', 'author_id': 6094, 'author_profile': 'https://Stackoverflow.com/users/6094', 'pm_score': 7, 'selected': True, 'text': '<p>You can\'t really avoid dealing with the text encoding issues, but there are existing solutions in Apache Commons:</p>\n\n<ul>\n<li><code>Reader</code> to <code>InputStream</code>: <a href="https://commons.apache.org/proper/commons-io/javadocs/api-release/org/apache/commons/io/input/ReaderInputStream.html" rel="noreferrer"><code>ReaderInputStream</code></a></li>\n<li><code>Writer</code> to <code>OutputStream</code>: <a href="https://commons.apache.org/proper/commons-io/javadocs/api-release/org/apache/commons/io/output/WriterOutputStream.html" rel="noreferrer"><code>WriterOutputStream</code></a></li>\n</ul>\n\n<p>You just need to pick the encoding of your choice.</p>\n'}, {'answer_id': 1360466, 'author': 'Sam Barnum', 'author_id': 14467, 'author_profile': 'https://Stackoverflow.com/users/14467', 'pm_score': 2, 'selected': False, 'text': '<p>Are you trying to write the contents of a <code>Reader</code> to an <code>OutputStream</code>?  If so, you\'ll have an easier time wrapping the <code>OutputStream</code> in an <code>OutputStreamWriter</code> and write the <code>char</code>s from the <code>Reader</code> to the <code>Writer</code>, instead of trying to convert the reader to an <code>InputStream</code>:</p>\n\n<pre><code>final Writer writer = new BufferedWriter(new OutputStreamWriter( urlConnection.getOutputStream(), "UTF-8" ) );\nint charsRead;\nchar[] cbuf = new char[1024];\nwhile ((charsRead = data.read(cbuf)) != -1) {\n    writer.write(cbuf, 0, charsRead);\n}\nwriter.flush();\n// don\'t forget to close the writer in a finally {} block\n</code></pre>\n'}, {'answer_id': 2373846, 'author': 'Phil Harvey', 'author_id': 172157, 'author_profile': 'https://Stackoverflow.com/users/172157', 'pm_score': 4, 'selected': False, 'text': '<p>Also note that, if you\'re starting off with a String, you can skip creating a StringReader and create an InputStream in one step using org.apache.commons.io.IOUtils from <a href="http://commons.apache.org/io/" rel="noreferrer">Commons IO</a> like so:</p>\n\n<pre><code>InputStream myInputStream = IOUtils.toInputStream(reportContents, "UTF-8");\n</code></pre>\n\n<p>Of course you still need to think about the text encoding, but at least the conversion is happening in one step.</p>\n'}, {'answer_id': 3235846, 'author': 'Ritesh Tendulkar', 'author_id': 390304, 'author_profile': 'https://Stackoverflow.com/users/390304', 'pm_score': 7, 'selected': False, 'text': '<p>If you are starting off with a String you can also do the following:</p>\n\n<pre><code>new ByteArrayInputStream(inputString.getBytes("UTF-8"))\n</code></pre>\n'}, {'answer_id': 4268583, 'author': 'Bozho', 'author_id': 203907, 'author_profile': 'https://Stackoverflow.com/users/203907', 'pm_score': 3, 'selected': False, 'text': '<p><a href="http://commons.apache.org/io/" rel="nofollow noreferrer">commons-io 2.0</a> has <a href="https://commons.apache.org/proper/commons-io/javadocs/api-release/org/apache/commons/io/output/WriterOutputStream.html" rel="nofollow noreferrer"><code>WriterOutputStream</code></a></p>\n'}, {'answer_id': 10655241, 'author': 'dfrankow', 'author_id': 34935, 'author_profile': 'https://Stackoverflow.com/users/34935', 'pm_score': 3, 'selected': False, 'text': '<p>You can\'t avoid text encoding issues, but <a href="http://commons.apache.org/io/" rel="nofollow">Apache commons-io</a> has</p>\n\n<ul>\n<li><a href="https://commons.apache.org/proper/commons-io/javadocs/api-release/org/apache/commons/io/input/ReaderInputStream.html" rel="nofollow">ReaderInputStream</a></li>\n<li><a href="https://commons.apache.org/proper/commons-io/javadocs/api-release/org/apache/commons/io/output/WriterOutputStream.html" rel="nofollow">WriterOutputStream</a></li>\n</ul>\n\n<p>Note these are the libraries referred to in Peter\'s answer of koders.com, just links to the library instead of source code.</p>\n'}, {'answer_id': 12940771, 'author': 'Peter Ford', 'author_id': 745522, 'author_profile': 'https://Stackoverflow.com/users/745522', 'pm_score': 6, 'selected': False, 'text': '<p>Well, a Reader deals with characters and an InputStream deals with bytes.  The encoding specifies how you wish to represent your characters as bytes, so you can\'t really ignore the issue.  As for avoiding problems, my opinion is: pick one charset (e.g. "UTF-8") and stick with it.  </p>\n\n<p>Regarding how to actually do it, as has been pointed out, "<em>the obvious names for these classes are <strong>ReaderInputStream</strong> and <strong>WriterOutputStream</strong>.</em>"  Surprisingly, "<em>these are not included in the Java library</em>" even though the \'opposite\' classes, <a href="http://docs.oracle.com/javase/6/docs/api/java/io/InputStreamReader.html" rel="noreferrer">InputStreamReader</a> and <a href="http://docs.oracle.com/javase/6/docs/api/java/io/OutputStreamWriter.html" rel="noreferrer">OutputStreamWriter</a> <strong>are</strong> included.</p>\n\n<p>So, lots of people have come up with their own implementations, including <strong>Apache</strong> <a href="http://commons.apache.org/io/" rel="noreferrer">Commons IO</a>.  Depending on licensing issues, you will probably be able to include the commons-io library in your project, or even copy a portion of the source code (which is downloadable <a href="http://commons.apache.org/io/download_io.cgi" rel="noreferrer">here</a>).</p>\n\n<ul>\n<li>Apache ReaderInputStream: <a href="https://commons.apache.org/proper/commons-io/javadocs/api-release/org/apache/commons/io/input/ReaderInputStream.html" rel="noreferrer">API</a> / <a href="https://svn.apache.org/repos/asf/commons/_moved_to_git/io/tags/2.5/src/main/java/org/apache/commons/io/input/ReaderInputStream.java" rel="noreferrer">source code direct link</a></li>\n<li>Apache WriterOutputStream: <a href="https://commons.apache.org/proper/commons-io/javadocs/api-release/org/apache/commons/io/output/WriterOutputStream.html" rel="noreferrer">API</a> / <a href="https://svn.apache.org/repos/asf/commons/_moved_to_git/io/tags/2.5/src/main/java/org/apache/commons/io/output/WriterOutputStream.java" rel="noreferrer">source code direct link</a></li>\n</ul>\n\n<p>As you can see, both classes\' documentation states that "all charset encodings supported by the JRE are handled correctly".</p>\n\n<p>N.B.  A comment on one of the other answers here mentions <a href="https://issues.apache.org/bugzilla/show_bug.cgi?id=40455" rel="noreferrer">this bug</a>.  But that affects the Apache <strong>Ant</strong> ReaderInputStream class (<a href="https://git-wip-us.apache.org/repos/asf?p=ant.git;a=blob_plain;f=src/main/org/apache/tools/ant/util/ReaderInputStream.java;hb=c11dcf20fa01d13c96a4017a734488ba763cc075" rel="noreferrer">here</a>), <em>not</em> the Apache <strong>Commons IO</strong> ReaderInputStream class.</p>\n'}, {'answer_id': 17493111, 'author': 'romeara', 'author_id': 1128981, 'author_profile': 'https://Stackoverflow.com/users/1128981', 'pm_score': 1, 'selected': False, 'text': "<p>A warning when using WriterOutputStream - it doesn't always handle writing binary data to a file properly/the same as a regular output stream. I had an issue with this that took me awhile to track down. </p>\n\n<p>If you can, I'd recommend using an output stream as your base, and if you need to write strings, use an OUtputStreamWriter wrapper around the stream to do it. It is far more reliable to convert text to bytes than the other way around, which is likely why WriterOutputStream is not a part of the standard Java library</p>\n"}, {'answer_id': 27565648, 'author': 'Oliv', 'author_id': 952135, 'author_profile': 'https://Stackoverflow.com/users/952135', 'pm_score': 3, 'selected': False, 'text': '<p>Use:</p>\n\n<pre><code>new CharSequenceInputStream(html, StandardCharsets.UTF_8);\n</code></pre>\n\n<p>This way does not require an upfront conversion to <code>String</code> and then to <code>byte[]</code>, which allocates lot more heap memory, in case the report is large. It converts to bytes on the fly as the stream is read, right from the StringBuffer.</p>\n\n<p>It uses <a href="https://commons.apache.org/proper/commons-io/javadocs/api-2.4/org/apache/commons/io/input/CharSequenceInputStream.html" rel="noreferrer" title="CharSequenceInputStream">CharSequenceInputStream</a> from Apache Commons IO project.</p>\n'}, {'answer_id': 27821517, 'author': 'Aaron', 'author_id': 7659, 'author_profile': 'https://Stackoverflow.com/users/7659', 'pm_score': -1, 'selected': False, 'text': '<p>For Reading a string in a stream using just what java supplies.</p>\n\n<pre><code>InputStream s = new BufferedInputStream( new ReaderInputStream( new StringReader("a string")));\n</code></pre>\n'}, {'answer_id': 45535083, 'author': 'yegor256', 'author_id': 187141, 'author_profile': 'https://Stackoverflow.com/users/187141', 'pm_score': 2, 'selected': False, 'text': '<p>You can use <a href="http://www.cactoos.org" rel="nofollow noreferrer">Cactoos</a> (no static methods, only objects):</p>\n\n<ul>\n<li><a href="http://static.javadoc.io/org.cactoos/cactoos/0.13.1/org/cactoos/io/InputStreamOf.html" rel="nofollow noreferrer"><code>new InputStreamOf(reader)</code></a></li>\n<li><a href="http://static.javadoc.io/org.cactoos/cactoos/0.13.1/org/cactoos/io/OutputStreamTo.html" rel="nofollow noreferrer"><code>new OutputStreamTo(writer)</code></a></li>\n</ul>\n\n<p>You can convert the other way around too:</p>\n\n<ul>\n<li><a href="http://static.javadoc.io/org.cactoos/cactoos/0.13.1/org/cactoos/io/ReaderOf.html" rel="nofollow noreferrer"><code>new ReaderOf(inputStream)</code></a></li>\n<li><a href="http://static.javadoc.io/org.cactoos/cactoos/0.13.1/org/cactoos/io/WriterTo.html" rel="nofollow noreferrer"><code>new WriterTo(outputStream)</code></a></li>\n</ul>\n'}, {'answer_id': 66206820, 'author': 'Peter Kriens', 'author_id': 243991, 'author_profile': 'https://Stackoverflow.com/users/243991', 'pm_score': 0, 'selected': False, 'text': '<p>This is the source code for a simple UTF-8 based encoding WriterOutputStream and ReaderInputStream. Tested at the end.</p>\n<pre><code>    // https://www.woolha.com/tutorials/deno-utf-8-encoding-decoding-examples\n    public class WriterOutputStream extends OutputStream {\n        final Writer    writer;\n\n        int             count       = 0;\n        int             codepoint   = 0;\n\n        public WriterOutputStream(Writer writer) {\n            this.writer = writer;\n        }\n\n        @Override\n        public void write(int b) throws IOException {\n            b &amp;= 0xFF;\n            switch (b &gt;&gt; 4) {\n            case 0b0000:\n            case 0b0001:\n            case 0b0010:\n            case 0b0011:\n            case 0b0100:\n            case 0b0101:\n            case 0b0110:\n            case 0b0111:\n                count = 1;\n                codepoint = b;\n                break;\n\n            case 0b1000:\n            case 0b1001:\n            case 0b1010:\n            case 0b1011:\n                codepoint &lt;&lt;= 6;\n                codepoint |= b &amp; 0b0011_1111;\n                break;\n\n            case 0b1100:\n            case 0b1101:\n                count = 2;\n                codepoint = b &amp; 0b0001_1111;\n                break;\n\n            case 0b1110:\n                count = 3;\n                codepoint = b &amp; 0b0000_1111;\n                break;\n\n            case 0b1111:\n                count = 4;\n                codepoint = b &amp; 0b0000_0111;\n                break;\n            }\n            if (--count == 0) {\n                writer.write(codepoint);\n            }\n        }\n    }\n\n    public class ReaderInputStream extends InputStream {\n        final Reader    reader;\n        int             count   = 0;\n        int             codepoint;\n\n        public ReaderInputStream(Reader reader) {\n            this.reader = reader;\n        }\n\n        @Override\n        public int read() throws IOException {\n            if (count-- &gt; 0) {\n                int r = codepoint &gt;&gt; (count * 6);\n                r &amp;= 0b0011_1111;\n                r |= 0b1000_0000;\n                return r;\n            }\n\n            codepoint = reader.read();\n            if (codepoint &lt; 0)\n                return -1;\n            if (codepoint &gt; 0xFFFF)\n                return 0;\n\n            if (codepoint &lt; 0x80)\n                return codepoint;\n\n            if (codepoint &lt; 0x800) {\n                count = 1;\n                int v = (codepoint &gt;&gt; 6) | 0b1100_0000;\n                return v;\n            }\n            count = 2;\n            int v = (codepoint &gt;&gt; 12) | 0b1110_0000;\n            return v;\n        }\n    }\n</code></pre>\n<p>And the test case that verifies if each of the 65536 characters is properly encoded and decoded, as well as verifying it matches the Java encoding. The surrogates verification (2 character encoding) are ignored since this is handled in Java.</p>\n<pre><code>    @Test\n    public void testAll() throws IOException {\n        for (char i = 0; i &lt; 0xFFFF; i++) {\n            CharArrayReader car = new CharArrayReader(new char[] { i });\n            ReaderInputStream rtoi = new ReaderInputStream(car);\n            byte[] data = IO.read(rtoi);\n\n            CharArrayWriter caw = new CharArrayWriter();\n            try (WriterOutputStream wtoo = new WriterOutputStream(caw)) {\n                wtoo.write(data);\n                char[] translated = caw.toCharArray();\n                assertThat(translated.length).isEqualTo(1);\n                assertThat((int) translated[0]).isEqualTo(i);\n\n                if (!Character.isSurrogate((char) i)) {\n                    try (InputStream stream = new ByteArrayInputStream(data)) {\n                        caw = new CharArrayWriter();\n                        IO.copy(data, caw);\n                        translated = caw.toCharArray();\n                        assertThat(translated.length).isEqualTo(1);\n                        assertThat((int) translated[0]).isEqualTo(i);\n                    }\n                }\n            }\n        }\n    }\n\n</code></pre>\n'}]
['https://Stackoverflow.com/questions/62241', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/3885/']