Scraping website for book listings, checking if database contains relevant listings, and displaying result in a Django view
After I process the URL to get the book info, I see if we have it in our database. If we don't, I move on, that's fine.


I guess you're referring to this part of the code:


try:
    book = books.get(book_num=book_num,
                     object_type_cd_id=object_type_cd.upper())
except book.DoesNotExist:
    pass



This doesn't do what you described,
because after the pass,
the rest of the loop body is executed instead of moving on the next iteration.
Perhaps you meant to use continue there instead of pass.



Instead of having an empty if block and an else block here,
it would be better to flip the condition and have only an if block without an else:


if (BookTextVer.objects.filter(
    book=book,
    book_text_ver_type__short_name=book_type.upper()).exists()):
    pass  # matching book type exists, no processing needed
else:
    data.append({"book": object_type_cd.upper() + book_num,
                 "book_type": book_type.upper()})





The query on BookTextVer tries to match by short_name.
Is that field indexed in the database.
If not, the query will be inevitably slow,
and indexing should be a significant improvement.



The current implementation runs 1 or 2 queries for every node in tree.getchildren().
You might get better performance if you first collect all the book_num, book_type, object_type_cd values from the document,
and formulate a single database query or constant number of database queries.
That would be a major change,
and possibly you might need to forego Django's OR-mapping benefits,
but the reduced number of database queries could make a big difference.