<p>I'm using <code>ByteBuffers</code> and <code>FileChannels</code> to write binary data to a file. When doing that for big files or successively for multiple files, I get an <code>OutOfMemoryError</code> exception.
I've read elsewhere that using <code>Bytebuffers</code> with NIO is broken and should be avoided. Does any of you already faced this kind of problem and found a solution to efficiently save large amounts of binary data in a file in java?</p>

<p>Is the jvm option <code>-XX:MaxDirectMemorySize</code> the way to go?</p>

[{'answer_id': 28684, 'author': 'Tim Frey', 'author_id': 1471, 'author_profile': 'https://Stackoverflow.com/users/1471', 'pm_score': 4, 'selected': True, 'text': "<p>I would say don't create a huge ByteBuffer that contains ALL of the data at once.  Create a much smaller ByteBuffer, fill it with data, then write this data to the FileChannel.  Then reset the ByteBuffer and continue until all the data is written.</p>\n"}, {'answer_id': 28690, 'author': 'Marcin', 'author_id': 3105, 'author_profile': 'https://Stackoverflow.com/users/3105', 'pm_score': 1, 'selected': False, 'text': '<p>If you access files in a <em>random fashion</em> (read here, skip, write there, move back) then you have a problem ;-)</p>\n\n<p>But if you only write big files, you should <strong>seriously</strong> consider using streams. <code>java.io.FileOutputStream</code> can be used directly to write file byte after byte or wrapped in any other stream (i.e. <code>DataOutputStream</code>, <code>ObjectOutputStream</code>) for convenience of writing floats, ints, Strings or even serializeable objects. Similar classes exist for reading files.</p>\n\n<p>Streams offer you convenience of manipulating <strong>arbitrarily large files in (almost) arbitrarily small memory</strong>. They are preferred way of accessing file system in vast majority of cases.</p>\n'}, {'answer_id': 28743, 'author': 'Stu Thompson', 'author_id': 2961, 'author_profile': 'https://Stackoverflow.com/users/2961', 'pm_score': 3, 'selected': False, 'text': '<p>Check out Java\'s <strong><a href="http://java.sun.com/j2se/1.4.2/docs/api/java/nio/MappedByteBuffer.html" rel="nofollow noreferrer">Mapped Byte Buffers</a></strong>, also known as \'direct buffers\'.  Basically, this mechanism uses the OS\'s virtual memory paging system to \'map\' your buffer directly to disk.  The OS will manage moving the bytes to/from disk and memory auto-magically, very quickly, and you won\'t have to worry about changing your virtual machine options.  This will also allow you to take advantage of NIO\'s improved performance over traditional java stream-based i/o, without any weird hacks.  </p>\n\n<p>The only two catches that I can think of are: </p>\n\n<ol>\n<li>On 32-bit system, you are limited to  just under 4GB <em>total for all mapped byte buffers</em>.  (That is actually a limit for my application, and I now run on 64-bit architectures.)</li>\n<li>Implementation is JVM specific and not a requirement.  I use Sun\'s JVM and there are no problems, but YMMV.</li>\n</ol>\n\n<p>Kirk Pepperdine (a somewhat famous Java performance guru) is involved with a website, www.JavaPerformanceTuning.com, that has some more MBB details: <strong><a href="http://www.javaperformancetuning.com/tips/nio.shtml#REF1" rel="nofollow noreferrer">NIO Performance Tips</a></strong></p>\n'}, {'answer_id': 28747, 'author': 'Dana the Sane', 'author_id': 2567, 'author_profile': 'https://Stackoverflow.com/users/2567', 'pm_score': 0, 'selected': False, 'text': "<p>The previous two responses seem pretty reasonable. As for whether the command line switch will work, it depends how quickly your memory usage hits the limit. If you don't have enough ram and virtual memory available to at least triple the memory available, then you will need to use one of the alternate suggestions given.</p>\n"}, {'answer_id': 28829, 'author': 'Cagatay', 'author_id': 3071, 'author_profile': 'https://Stackoverflow.com/users/3071', 'pm_score': 0, 'selected': False, 'text': '<p>Using the <a href="http://java.sun.com/j2se/1.5.0/docs/api/java/nio/channels/FileChannel.html#transferFrom(java.nio.channels.ReadableByteChannel,%20long,%20long)" rel="nofollow noreferrer">transferFrom</a> method should help with this, assuming you write to the channel incrementally and not all at once as previous answers also point out.</p>\n'}, {'answer_id': 140145, 'author': 'Darron', 'author_id': 22704, 'author_profile': 'https://Stackoverflow.com/users/22704', 'pm_score': 0, 'selected': False, 'text': "<p>This can depend on the particular JDK vendor and version.</p>\n\n<p>There is a bug in GC in some Sun JVMs.  Shortages of direct memory will not trigger a GC in the main heap, but the direct memory is pinned down by garbage direct ByteBuffers in the main heap.  If the main heap is mostly empty they many not be collected for a long time.</p>\n\n<p>This can burn you even if you aren't using direct buffers on your own, because the JVM may be creating direct buffers on your behalf.  For instance, writing a non-direct ByteBuffer to a SocketChannel creates a direct buffer under the covers to use for the actual I/O operation.</p>\n\n<p>The workaround is to use a small number of direct buffers yourself, and keep them around for reuse.</p>\n"}]
['https://Stackoverflow.com/questions/28675', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/2612/']