def sort(filename, key, outputFile, fields=None, watermark=<NUM_LIT> * <NUM_LIT> * <NUM_LIT:100>):
if fields is not None:<EOL><INDENT>assert set(key).issubset(set([f[<NUM_LIT:0>] for f in fields]))<EOL><DEDENT>with FileRecordStream(filename) as f:<EOL><INDENT>if fields:<EOL><INDENT>fieldNames = [ff[<NUM_LIT:0>] for ff in fields]<EOL>indices = [f.getFieldNames().index(name) for name in fieldNames]<EOL>assert len(indices) == len(fields)<EOL><DEDENT>else:<EOL><INDENT>fileds = f.getFields()<EOL>fieldNames = f.getFieldNames()<EOL>indices = None<EOL><DEDENT>key = [fieldNames.index(name) for name in key]<EOL>chunk = <NUM_LIT:0><EOL>records = []<EOL>for i, r in enumerate(f):<EOL><INDENT>if indices:<EOL><INDENT>temp = []<EOL>for i in indices:<EOL><INDENT>temp.append(r[i])<EOL><DEDENT>r = temp<EOL><DEDENT>records.append(r)<EOL>available_memory = psutil.avail_phymem()<EOL>if available_memory < watermark:<EOL><INDENT>_sortChunk(records, key, chunk, fields)<EOL>records = []<EOL>chunk += <NUM_LIT:1><EOL><DEDENT><DEDENT>if len(records) > <NUM_LIT:0>:<EOL><INDENT>_sortChunk(records, key, chunk, fields)<EOL>chunk += <NUM_LIT:1><EOL><DEDENT>_mergeFiles(key, chunk, outputFile, fields)<EOL><DEDENT>
Sort a potentially big file

    filename - the input file (standard File format)
    key - a list of field names to sort by
    outputFile - the name of the output file
    fields - a list of fields that should be included (all fields if None)
    watermark - when available memory goes bellow the watermark create a new chunk

    sort() works by reading as records from the file into memory
    and calling _sortChunk() on each chunk. In the process it gets
    rid of unneeded fields if any. Once all the chunks have been sorted and
    written to chunk files it calls _merge() to merge all the chunks into a
    single sorted file.

    Note, that sort() gets a key that contains field names, which it converts
    into field indices for _sortChunk() becuase _sortChunk() doesn't need to know
    the field name.

    sort() figures out by itself how many chunk files to use by reading records
    from the file until the low watermark value of availabel memory is hit and
    then it sorts the current records, generates a chunk file, clears the sorted
    records and starts on a new chunk.

    The key field names are turned into indices