Django model instance full_clean method, is this right?
Question: What I want to do, is write code that will allow me to bulk-load Django object
instances from a csv file. Obviously I should check all the data first before
saving anything.

tl;dr: the full_clean() method doesn't catch an impending attempt to save None
in a field without `null=True`. Seems perverse. Is this by design, and if so
why? Django has fewer bugs than just about anything else I have ever worked
with, so "Bug!" seems most unlikely.

Full version. What I thought would work, is for each row, create an object
instance, populate fields with the data from the spreadsheet, and then invoke
the full_clean method. I.e. (in outline)

    
    
    from django.core.exceptions import ValidationError
    ...
    
    # upload a CSV file and open with a csvreader
    errors=[]
    for rownumber, row in enumerate(csvreader):
    
        o = SomeDjangoModel()
        o.somefield = row[0]  # repeated for all input data row[1] ...
    
        try:
            reason = ""
            o.full_clean()
        except ValidationError as e:
            reason = "Row:{} Reason:{}".format( rownumber, str(e))
            errors.append( reason)
            # reason, together with the row-number of the csv file, fully explains
            # what is wrong.
    
    # end of loop
    if errors:
        # display errors to the user for him to fix
    else:
        # repeat the loop,  doing .save() instead of .full_clean() 
        # and get database integrity errors trying to save Null in non-null model field.
    

Trouble is, `.full_clean()` does not catch None values in fields without
`null=True`

What should I be doing? Ideas include

  1. Wrap the whole thing in a transaction, do a batch of o.save() inside an exception handler, and roll the entire transaction back unless there were no errors. But why bother the database when probably 90% of attempts will error out in trivial ways?

  2. Feed the data in through a form, even though there is no form-level per-row interaction with the user.

  3. Manually test for None where it shouldn't be. But what else does .full_clean not check?

I can understand that ultimately the only way to catch database integrity
errors is to attempt to store the data, but why doesn't Django alone catch
None in a null=False field?

BTW This is Django 1.9.6

Added detail. This is relevant fields of the model definition

    
    
    class OrderHistory( models.Model):
        invoice_no = models.CharField( max_length=10, unique=True)         # no default
        invoice_val= models.DecimalField( max_digits=8, decimal_places=2)  # no default
        date       = models.DateField( )                                   # no default
    

and this is what is happening, done from `python manage.py shell`, to
demonstrate that the .full_clean method fails to spot a n

    
    
    >>> from orderhistory.models import OrderHistory
    >>> from datetime import date
    >>> o = OrderHistory( date=date(2010,3,17), invoice_no="21003163")
    >>> o.invoice_val=None 
    >>> o.full_clean()  # passes clean
    >>> o.save() # attempt to save this one which has passed full_clean() validation
    Traceback (most recent call last):
      File "<console>", line 1, in <module>
      File "/home/nigel/.virtualenvs/edge22/lib/python3.4/site-packages/django/db/models/base.py", line 708, in save
    force_update=force_update, update_fields=update_fields)
      File "/home/nigel/.virtualenvs/edge22/lib/python3.4/site-packages/django/db/models/base.py", line 736, in save_base
        updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
      File "/home/nigel/.virtualenvs/edge22/lib/python3.4/site-packages/django/db/models/base.py", line 820, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
      File "/home/nigel/.virtualenvs/edge22/lib/python3.4/site-packages/django/db/models/base.py", line 859, in _do_insert
    using=using, raw=raw)
      File "/home/nigel/.virtualenvs/edge22/lib/python3.4/site-packages/django/db/models/manager.py", line 122, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
      File "/home/nigel/.virtualenvs/edge22/lib/python3.4/site-packages/django/db/models/query.py", line 1039, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
      File "/home/nigel/.virtualenvs/edge22/lib/python3.4/site-packages/django/db/models/sql/compiler.py", line 1060, in execute_sql
    cursor.execute(sql, params)
      File "/home/nigel/.virtualenvs/edge22/lib/python3.4/site-packages/django/db/backends/utils.py", line 79, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
      File "/home/nigel/.virtualenvs/edge22/lib/python3.4/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
      File "/home/nigel/.virtualenvs/edge22/lib/python3.4/site-packages/django/db/utils.py", line 95, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
      File "/home/nigel/.virtualenvs/edge22/lib/python3.4/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
      File "/home/nigel/.virtualenvs/edge22/lib/python3.4/site-packages/django/db/backends/utils.py", line 64, in execute
        return self.cursor.execute(sql, params)
    django.db.utils.IntegrityError: null value in column "invoice_val" violates not-null constraint
    DETAIL:  Failing row contains (2, 21003163, , , 2010-03-17, , null, null, null, null, null, null).
    >>>
    >>> p = OrderHistory( invoice_no="21003164") # no date
    >>> p.date=None
    >>> p.full_clean()                           # this DOES error as it should
      Traceback (most recent call last):
      File "<console>", line 1, in <module>
      File "/home/nigel/.virtualenvs/edge22/lib/python3.4/site-packages/django/db/models/base.py", line 1144, in full_clean
      raise ValidationError(errors)
    django.core.exceptions.ValidationError: {'date': ['This field cannot be null.']}
    >>>
    


Answer: I have just repeated your steps in shell, and full_clean() triggers
ValidationError for None values:

    
    
    >>> from orders.models import OrderHistory
    >>> o = OrderHistory()
    >>> o.full_clean()
    Traceback (most recent call last):
      File "<console>", line 1, in <module>
      File "/Users/oz/.virtualenvs/full_clean_test/lib/python2.7/site-packages/django/db/models/base.py", line 1144, in full_clean
        raise ValidationError(errors)
    ValidationError: {'date': [u'This field cannot be null.'], 'invoice_val': [u'This field cannot be null.'], 'invoice_no': [u'This field cannot be blank.']}
    

I've tested it on fresh project with Django 1.9.6 and Python 2.7.10 on OSX and
Python 3.4.3 on Ubuntu.

Try removing all *.pyc files from your project. If that doesn't work, remove
your virtual env, create new one and reinstall your dependencies.

