Writing billions of lines to text file in java I'm trying to write billions of strings lines to a file, it works for up to 40 million lines, but it's throwing out "java.nio.BufferOverflowException" error for 400 million lines. I also think my solution is slow as it takes 45 seconds to write 40 million lines. Below is the code.

public static void main(String[] args) throws FileNotFoundException, IOException {

byte[] buffer = "Help I am trapped in a fortune cookie factory\n".getBytes();
int number_of_lines = 400000000;

FileChannel rwChannel = new RandomAccessFile("textfile.txt", "rw").getChannel();
ByteBuffer wrBuf = rwChannel.map(FileChannel.MapMode.READ_WRITE, 0, buffer.length * number_of_lines);
for (int i = 0; i &lt; number_of_lines; i++) {
    wrBuf.put(buffer);
}
rwChannel.close();
}


How can I make it be faster if possible and also write more than 2 billion lines of string?
It sure sounds like you're running out of memory as you try to build (or allocate) the buffer. 


You need to figure out how big a buffer you can (or want to) use. You'll need to ask the system how much RAM it has, and how much is available, and make your buffer that large.
You might get a small boost by opening the file as explicitly write-only.
You'll need to flush the buffer as you go. Something shaped like this psudo-code might work:


free = system.free_ram();
use = hungry * free;
buffer = new bytes_buffer_to_file("filename", size=use);
try{
  current_buffer = 0;
  while(line = get_bytes_needing_to_be_written()){
    if(current_buffer + line.length &gt; use){
      buffer.flush();
      current_buffer = 0;
    }
    buffer.append(line);
    current_buffer += line.length;
  }
}
finally{
  buffer.flush();
  buffer.destroy();
}