I'm looking for advice, as I haven't had to deal with large file parsing before and would like to avoid re-inventing the wheel if an open source solution already exists. Here's my situation:

I have roughly 200-300 large XML and text files being dropped in a directory by an automated machine process in the lab I work in. This happens on a regular basis. These files can range in size from a few hundred MB to multiple GBs in size. These files are regularly modified (couple times a week) at random intervals where the old file is simply overwritten with the modified one.

I need the ability to search these files and pull out records that match specific criteria. Of the roughly 20-30 million records in the files (combined), we might actually use < 100,000 of them, but we can't tell which ones until they are searched. 

My first thought it to setup a sort of regular file processing job that detects updates and processes the files into a database that can be searched. My only concern would be that it would probably get slower and slower to insert and update the records as it got larger and larger. 

Does anyone have any suggestions on a method that might be more suitable for my situation? Off the top of my head I was thinking some text searching system like Lucene, but having never used it I'm not positive if it would be anymore useful than a database...

Any help would be much appreciated.