Matlab - Fastest way to parallel read many files line by line? I have a few hundred files in a directory that I need to read in parallel. Namely, the files that have the same name, but different file extensions (.a and .b), but they're pretty much plain text files. The corresponding .a and .b files have the same number of lines. I need to read each line into cell arrays x and y until all each pair of files in the directory have been read. Currently, I'm doing it with a nested loop which works fine, but just takes too long. Is there any way to do this faster?

x = {};
y = {};
dir1 = dir( [ directory, filesep, '*', 'a'] );
dir2 = dir( [ directory, filesep, '*', 'b'] );
i = 1;
for i=1:length(dir1)
    fid1 = fopen([directory, filesep, dir1(i).name]);
    fid2 = fopen([directory, filesep, dir2(i).name]);
    sentence1 = fgetl(fid1);
    sentence2 = fgetl(fid2);
    while ischar(sentence)          
        x{i} = sentence1;
        y{i} = sentence2;
        i = i+1;            
        sentence1 = fgetl(fid1);
        sentence2 = fgetl(fid2);  
    end
    fclose(fid1);
    fclose(fid2);
end
Performing this in parallel using PARFOR should be straightforward if you have the Parallel Computing Toolbox installed. Whether it goes any faster or not depends entirely on the size of your data, the speed of your disks etc. Here's how I would approach it:

function [x, y] = readOnePairOfFiles(name1, name2)
   x = {}; y = {};
   fid1 = fopen(name1); fid2 = fopen(name2);
   %# Read from fid1 and fid2, build up x and y
end

% Use PARFOR to invoke readOnePairOfFiles
x = {}; y = {};
parfor ii = 1:length(dir)
   name1 = fullfile(directory, dir1(ii).name);
   name2 = fullfile(directory, dir2(ii).name);
   [thisX, thisY] = readOnePairOfFiles(name1, name2);
   x = [x, thisX]; %# concatenation reduction to get x and y
   y = [y, thisY];
end