How to get columns from unsorted rows in Pandas? (MALLET)
So one way to do it is to use python slice operators to grab every other value in the line and zip them (along with the filename) into 3-tuples, e.g.:

data = []
malletOutput = open('doc-topics','r').readlines()

for line in malletOutput:
    line = line.split('\t')[1:-1] # slicing out useless leading index and trailing \n
    _id = line[0]

    tIndicies = map(int,line[1::2])
    tVals = map(float, line[2::2])
    topics = sorted(zip(tIndicies, tVals))
    topics = [t + tuple([_id]) for t in topics] # add _id
    for t in topics:
        data.append(t)


Now you have a list of 3-tuples which you can add to a dataframe. A simple pivot from there results in the desired shape:

df = pd.DataFrame(data)
df = df.pivot(index=2,columns=0,values=1)