How to load textfiles into SQL Server Database?
Certainly you are person who knows a lot of technology tools etc. But I will just point here, that two things are probably worth of research:


Storage of the files on Linux filesystem. From my experience cames that even opening and listing directory with 100 000 or more files is a task which Microsoft cannot do in efficient way. Just using more efficient operating system here may give you at least some ergonomic gains.
Maybe you should research etl tools like Apatar. It works with many files by GUI, so in some cases you may have a visualization of your process, which probably makes it easier to automate or analyse.
It may be surprising in relation to this problem, but on the market there are very efficient data collection tools. Maybe thinking outside the box, and using clustered tools for system log acquisition, like Graylog log server, will be interesting choice. It has pipelining system, which takes data from network port, or through remote agents monitoring particular text files on remote system, and crunch it through various regex collectors, then has ability to divide that streams into output depending on given regex results. Output may be defined by you, and sent to another network device. Even on single machine acquisition of several hundreds of thousands records takes a seconds. It is very, very efficient. I believe at least part of your work may be automated here, for example it may test patterns and produce stream with wrong format entries ( or with good ones). And it gives you some basic statistical informations like dashboard with graphic representation of error types etc.