How to import data from scanned text into Django models
Question: I have a hundreds of pages of "quiz" questions, multiple-choice options and
associated answer keys and explanations. I'm trying to create a simple Django
app to administer these questions. I have created a simple but effective
Python parser to parse the scanned, OCR'd pages into the proper objects.

I want to have a "utility" to enable the administrator of this Django app to
import quiz content from OCR'd paper into the relevant Django DB tables. This
will be a rare task, and something that is not necessarily appropriate for
inclusion in a web UI.

I've asked about using intermediate JSON/YAML fixtures and was told that a
more appropriate approach would be to directly create and save instances of my
models [1]. I then tried to create a standalone script along the lines
suggested by [2] and [3] but was unable to overcome the `kwargs =
{"app_label": model_module.__name__.split('.')[-2]} IndexError: list index out
of range` error.

I also came across [4] about creating custom django-admin.py/manage.py
commands. This seems like a logically appropriate way of dealing with the
task; but, I'd love to hear from those with more experience and brains (I've
eaten all mine :).

## References:

  1. [Importing data from scanned text into Django as YAML fixture or SQL](http://stackoverflow.com/questions/7669550/importing-data-from-scanned-text-into-django-as-yaml-fixture-or-sql)
  2. [what is the simplest way to create a table use django db api ,and base on 'Standalone Django scripts'](http://stackoverflow.com/questions/2211816/what-is-the-simplest-way-to-create-a-table-use-django-db-api-and-base-on-stand)
  3. [Standalone Scripts](http://www.b-list.org/weblog/2007/sep/22/standalone-django-scripts/)
  4. [Writing custom django-admin commands](https://docs.djangoproject.com/en/dev/howto/custom-management-commands/)

## Examples:

  * OCR'd Text

> Page 12 34\. Hiedegger is a _____ . (a) philosopher (b) boozy beggar (c)
> both a and b (d) none of these 35. ...

  * Django Models
    
        class Question(models.Model):
        text = models.TextField()
    class Choice(models.Model):
        question = models.ForeignKey(Question)
        order = models.IntegerField(default=1)
        text = models.TextField()
    

  * Goal, something like this...
    
        q = Question.objects.create(text="Hiedegger is a _____ .")
    q.save()
    c = Choice(text="philosopher", order=1, question=q.pk)
    c.save()
    


Answer: This is the working version I came up with. Dirty, but effective. Both @akonsu
and @Ivan Kharlamov were helpful. Thanks...

    
    
    import os, re, Levenshtein as lev, codecs
    from SimpleQuiz.quiz.models import Choice, Question
    from django.core.management.base import BaseCommand, CommandError
    import optparse
    
    class Command(BaseCommand):
        args = '--datapath=/path/to/text/data/'
        can_import_settings = True
        help = 'Imports scanned text into Questions and Choices'
        option_list = BaseCommand.option_list + (
            optparse.make_option('--datapath', action='store', type='string',
                                 dest='datapath',
                                 help='Path to OCRd text files to be parsed.'),
            )
        requires_model_validation = True
        # Parser REs
        BACKUP_RE = re.compile(r'\~$|bak$|back$|backup$')
        QUEST_RE = re.compile(r'^[0-9]{1,3}[.][ ]')
        CHOICE_RE = re.compile(r'^[a-e][.][ ]')
    
        def handle(self, *args, **options):
            # get the data path
            try:
                os.path.exists(options['datapath'])
            except Exception as e:
                raise CommandError("None or invalid path provided: %s" % e.message)
            self.datapath = os.path.expanduser(options['datapath'])
    
            # generate list of text strings from lines in target files
            self.data_lines = []
            for fn in os.listdir(os.path.join(self.datapath, 'questions/')):
                if self.BACKUP_RE.search(fn):
                    self.stderr.write("Skipping backup: %s\n" % (fn))
                else:
                    for line in codecs.open(os.path.join(self.datapath, 'questions/', fn), 'r', encoding='latin-1'):
                        if not self.is_boilerplate(line):
                            if not line.strip() == '':
                                self.data_lines.append(line)
    
        #----------------------------------------------------------------------- 
        #--------------------- Parse the text lines and create Questions/Choices
        #----------------------------------------------------------------------- 
            cur_quest = None
            cur_choice = None
            cur_is_quest = False
            questions = {}
            choices = {}
            for line in self.data_lines:
                if self.is_question(line):
                    [n, txt] = line.split('.', 1)
                    qtext = txt.rstrip() + " "
                    q = Question.objects.create(text=qtext)
                    q.save()
                    cur_quest = q.pk
                    questions[cur_quest] = q
                    cur_is_quest = True
                elif self.is_choice(line):
                    [n, txt] = line.split('.', 1)
                    num = self.char2dig(n)
                    ctext = txt.rstrip() + " "
                    c = Choice.objects.create(text=ctext, order=num, question=questions[cur_quest])
                    c.save()
                    cur_choice = c.pk
                    choices[cur_choice] = c
                    cur_is_quest = False
                else:
                    if cur_is_quest:
                        questions[cur_quest].text += line.rstrip() + " "
                        questions[cur_quest].save()
                    else:
                        choices[cur_choice].text += line.rstrip() + " "
                        choices[cur_choice].save()
            self.stdout.write("----- FINISHED -----\n")
            return None
    
        def is_question(self, arg_str):
            if self.QUEST_RE.search(arg_str):
                return True
            else:
                return False
    
        def is_choice(self, arg_str):
            if self.CHOICE_RE.search(arg_str):
                return True
            else:
                return False
    
        def char2dig(self, x):
            if x == 'a':
                return 1
            if x == 'b':
                return 2
            if x == 'c':
                return 3
            if x == 'd':
                return 4
            if x == 'e':
                return 5
    
        def is_boilerplate(self, arg_str):
            boilerplate = [u'MFT PRACTICE EXAMINATIONS',
                           u'BERKELEY TRAINING ASSOCIATES ' + u'\u00A9' + u' 2009',
                           u'BERKELEY TRAINING ASSOCIATES',
                           u'MARRIAGE AND FAMILY THERAPY',
                           u'PRACTICE EXAMINATION 41',
                           u'Page 0', u'Page 1', u'Page 2', u'Page 3', u'Page 4',
                           u'Page 5', u'Page 6', u'Page 7', u'Page 8', u'Page 9',
                           ]
            for bp in boilerplate:
                if lev.distance(bp.encode('utf-8'), arg_str.encode('utf-8')) < 4:
                    return True
            return False
    

