Python (instantiate) locks to multiple outputs
It sounds like you will have potentially more than one thread writing to a single output file, so you want to make the writes thread-safe, while allowing another output file to be created and written to if a subject is added.
I would recommend having each thread simply lock the output file during the write. This would not prevent another output file from being created; you would simply ensure that that file gets locked during a write as well. Then, whether you have 16 or 32 threads or something in between, it doesn't matter how many output files you have -- simply ensure they are locked during the write, so any other thread that tries to write there will need to wait.
I am working on Python multi-threading application.
The scenario is:
The source data(thousands of small files per hour) contains data about many subjects(range 1-100). Each row starts with "subject1|col1|col2|...|coln|". 
Right now users are interested in only 10(example) subjects. But in future they can add(or remove) more subjects as they prefer.
For this I wrote a program, which created a Queue and adds all source files in the directory to it. Configurable # of threads start and get a file from the queue and start parsing the data. I want to generate one large file per subject. so since user currently wants 10 subject parsed, I will generate 10 output files.
Since there can be 16-32 threads parsing the source data and writing to 10 output files, I don't want to create 10 locks, one for each output files (hard code the subject_lock per output file) since I want to provide flexibility for the program to dynamically start parsing new subject.
I am not able to think of any way to dynamically control the thread locks to the target output files (other than old school way of creating a empty file touch file(with subject_timestamp for debugging) which is generated based on the subject list, which is read from configuration file every few seconds). 
Any recommendations or suggestions are appreciated.
Thanks,
Venkat