Processing files retrieved through FTP in parallel I am busy creating a file processor that needs to get some files from an Ftp client download the file and save the data into the database. I have a two part question.


How can i refactor the code to look a bit cleaner? Feels like the
    method ProcessFiles is doing a bit much. I can extract some of the
    methods but then i am just sitting with a lot of small methods and
    not really gaining anything.     
What will be the best way the handle exceptions? I need to be able
to capture the file that caused an error.


Code Sample

public void Run()
        {
            var ftpFileManager = new FtpFileManager("ftp.someurl.net", "username", "password");   
            var files = ftpFileManager.GetDirectory("somepath");

            try
            {
                ProcessFiles(files, ftpFileManager);    
            }
            catch (AggregateException ae)
            {
                foreach (var ex in ae.InnerExceptions)
                {
                    if (ex is ArgumentException)
                        Console.WriteLine(ex.Message);
                    else
                        throw ex;
                }
            }
        }



private void ProcessFiles(IEnumerable&lt;FtpListItem&gt; files, IFtpFileManager ftpFileManager)
        {
            var exceptions = new ConcurrentQueue&lt;Exception&gt;();
            files.AsParallel().ForAll(async _ =&gt;
            {
                try
                {
                    var dataFileService = _container.GetExportedValue&lt;IDataFileHandler&gt;();
                    var progress = new ProgressReporter(_container.GetExportedValue&lt;IDataFileHandler&gt;());

                var attr = FtpFileAttributes.Parse(_.Name);
                attr.DateOfFile = _.Modified;
                var dataFileId = await dataFileService.AddAsync(attr);

                await progress.Report(Status.Started, dataFileId);

                var file = ftpFileManager.DownloadFile(_.FullName);
                var fileHandler = FileHandlerFactory.Create(attr.FileType, attr);
                var list = fileHandler.ProcessFile(file);

                var handler = CreateHandler(attr.FileType);
                await handler.AddAsync(list, dataFileId);

                var dataFile = await dataFileService.GetAsync(dataFileId);
                dataFile.EndDate = DateTime.Now;
                dataFile.TotalRecords = list.Count();
                await dataFileService.UpdateAsync(dataFile);

                await progress.Report(Status.Completed, dataFileId);
            }
            catch (Exception e)
            {
                exceptions.Enqueue(e);
            }
        });

        if (exceptions.Count &gt; 0) 
            throw new AggregateException(exceptions);
    }
Exception order changes processing

This looks really odd to me:

catch (AggregateException ae)
{
    foreach (var ex in ae.InnerExceptions)
    {
        if (ex is ArgumentException)
            Console.WriteLine(ex.Message);
        else
            throw ex;
    }
}


It's basically saying write ArgumentExceptions to the Console, but only until you find an error that isn't an ArgumentException.  Then throw that exception.

I don't like it firstly because it could have different results based on order that the exceptions have been added to the queue and secondly because it discards any exceptions after the first one.

Exceptions for program flow

Having ProcessFiles which is a private method return void and throw an aggregate exception, which is immediately caught in the public caller, feels wrong to me.  You've already collected the exceptions into a queue, why not just have the method return the queue?  The caller can then decide what to do based on the collection being empty or not.