Transpose of a very large matrix with fewer than 1021 rows with Python
The golden rule of I/O performance is never read a block twice.  You're reading each block thousands of times.

I made a thousand-row test file with records of about 2MB each:

tr -dc '[:alnum:]' &lt; /dev/urandom  | fold -w2000000 | head -1000 &gt; test.txt


And modified the fps-building portion of your code thusly:

#!/usr/bin/env python
from prettyprinter import pprint
import sys
import os
import re

try:
  in_fn = sys.argv[1]
except ValueError as ve:
  sys.exit(-1)

sz = os.path.getsize(in_fn)
fps = [ open(in_fn, 'r') ]
bufsz = 100 * 2**20 # MB

with open(in_fn, 'r') as f:
  offset = 0
  buf = f.read(bufsz)
  while len(buf)&gt;0:
    for p in [m.start() for m in re.finditer(r'\n', buf)]:
      new_pos = offset+p+1
      if new_pos &lt; sz:
        new_fp = open(in_fn, 'r')
        new_fp.seek(new_pos, 0)
        fps.append(new_fp)

    offset = f.tell()
    buf = f.read(bufsz)

pprint( list( map( lambda f: f.tell(), fps ) ) )


This runs 65 times faster than the bytewise version on my machine.

The actual transposition will benefit similarly from this treatment.   Keep a buffer for each row and append to it with a read when it's empty.  

If you want to keep the logic simpler, you can compromise (with a performance penalty) by reading a few kilobytes at a time and rewind-seeking  the filehandle back to the first tab character.  This would remove the need to keep an array of buffers.