title: displaying a large amount of formatted text in Python
body: <p>I have two large identical-sized files.  One is ASCII plain text, and the other is a colour-coded overlay, one byte per text character in the corresponding file.</p>
  
  <p>These files can be large - upto 2.5 MB; possibly substantially more, perhaps over 100MB later.</p>
  
  <p>I want to display the text is a scrollable text viewer, using the second file as the overlay.  Looking something like this:</p>
  
  <p><a href="http://img713.imageshack.us/img713/2584/statsviewer01d.png">Tkinter Text window with tagged text for colours http://img713.imageshack.us/img713/2584/statsviewer01d.png</a></p>
  
  <p>So I made a simple tkinter Text widget window in Python with scrollbar and such.</p>
  
  <p>My code looks like this:</p>
  
  <pre><code>hottest = 0
  for heat in heatmap:
      hottest = max(hottest,ord(heat))
  hottest += 1
  for heat in xrange(0,hottest):
      factor = int((float(heat)/float(hottest))*100.0)
      # an observation; tkinter seems to normalise custom colours
      # to nearest in it's palette, which means I can't use custom
      # gradients of red; if anyone knows how to use subtle custom colours?
      bgcolour = "gray%d" % (100-factor)
      fgcolour = "gray%d" % factor
      text.tag_config("n%d"%heat,background=bgcolour,foreground=fgcolour)
  text.insert("1.0",f.read())
  ofs = 0
  for heat in heatmap:
      if 0 != ord(heat):
          coord_start = "1.0 + %d chars"%ofs
          coord_stop = "1.0 + %d chars"%(ofs+1)
          text.tag_add("n%d"%ord(heat),coord_start,coord_stop)
      ofs += 1
  text.config(state=DISABLED)
  text.focus()
  </code></pre>
  
  <p>However, I run into horrid performance problems:</p>
  
  <ol>
  <li><p>loading the text</p>
  
  <ul>
  <li>if I simply use textwidget.text = textfile.read() it loads instantly, even for the large file; but then the problem is colour-coding it.  <em>(see code above)</em>.   To add the tags, one character at a time, it seems I force it to take a <a href="http://www.joelonsoftware.com/articles/fog0000000319.html">painter algorithm</a>; by the time the file is about 300KB in size, I give up waiting for it to load</li>
  <li>if I read the next character from the textfile and the colourfile, and append a properly tagged character, it takes forever; a file bigger than about 200KB and I give up waiting for it to load</li>
  </ul></li>
  <li><p>Scrolling.  If this much formatting visible in the current window, it goes very slowly</p></li>
  </ol>
  
  <p>If I was approaching this problem in, say, Dephi or wxWidgets or whatever, I'd have a custom-drawn control.</p>
  
  <p>What's the most straightforward way in Python and Tkinter?</p>