Creating multiple files from a list
The execution time will be dominated by I/O, since you are writing so many files.  The only suggestion I have that might change the I/O performance is to try the NIO API, namely Files.write().

There is another possible inefficiency if sampleSize is much smaller than the size of the collections.  In that case, shuffling the entire list would be overkill, and you might be better off writing your own function to perform sampling without replacement.  (That would basically be a Fisher-Yates Shuffle that terminates early.)



There are four similar copies of the code, which you might want to consolidate, especially if it is not important to preserve the behaviour of your System.out.println(i) progress indicator.

My first observation is that the pos and neg lists can be treated independently.  My second observation is that you could simplify the loop if you interleave the output of the test and train cases.

I think that this code would make it more obvious that the number of files created is sampleSize, for each of the lists neg and pos.

private static void writeSampleFiles(List&lt;String&gt; list, String name, int sampleSize) throws IOException {
    Collections.shuffle(list);
    for (int i = 0; i &lt; sampleSize; i++) {
        Path path = Paths.get(i &amp; 1 == 0 ? "train" : "test",
                              name,
                              (i &gt;&gt; 1) + ".txt");
        Files.write(path, list.get(i).getBytes());
    }
}

// elsewhere...
writeSampleFiles(neg, "neg", sampleSize);
writeSampleFiles(pos, "pos", sampleSize);


Note that this isn't exactly identical to your original code.  The shuffled entries are extracted in a different order, which could be an issue if you are concerned about "dealing from the top of the deck".  If the execution aborts prematurely due to an IOException, then the difference in the processing order would be visible to the user.