Title: How optimize fetch and related objects loading?
Tags: python;django-models;fetch
Question: I am beginer with django, and after few hours passed to looking for a solution I finally ask for help. I would like to find the best way (performance) to retrieve/fetch data with my model. I tried a couple of things and I read documentation about select_related, prefetch_related.
The concret probleme is :
I have a list of specific_id, retrieved from a web service, and i need to load the correpondings objects (in the right order given by the list) and with some related objects.
My model :
I have a product with 0-N repositories related (FK from Repository).
This product have also 0-N sources (couple of 0-N Sensor and  0-N Platform), through sourcecontribtoproduct
class Product(models.Model):
specific_id = models.CharField(max_length=100, unique=True, blank=False, null=False)
shortname = models.CharField(max_length=100, unique=False, blank=True, null=False)
class Repository(models.Model):
product = models.ForeignKey(Product, blank=True, null=True)
name = models.CharField(max_length=200, unique=True, blank=False, null=False)
url = models.CharField(max_length=2048, blank=True, null=True)
class SourceContribToProduct(models.Model):
source = models.ForeignKey(Source)
product = models.ForeignKey(Product)
class Source(models.Model):
products = models.ManyToManyField(Product, through='SourceContribToProduct')
shortname = models.CharField(max_length=100, unique=True, blank=False, null=False)
platform = models.ForeignKey(Platform, blank=True, null=True)
sensor = models.ForeignKey(Sensor, blank=True, null=True)
class Sensor(models.Model):
shortname = models.CharField(max_length=50, unique=True, blank=False, null=False)
type = models.TextField(blank=True, null=True)
class Platform(models.Model):
shortname = models.CharField(max_length=50, unique=True, blank=False, null=False)
description = models.TextField(blank=True, null=True)
So, as I said, I need to retrieve a list of products with all Repository, Sources and Plateform loaded. Here is what I have done in my views.py
# Retrive list of objects
products = Product.objects.filter(specific_id__in=ids).prefetch_related('repository_set', 'source_set')
for p in products:
p.repositories = p.repository_set.all()
p.sources = Source.objects.filter(sourcecontribtoproduct__product_id=p.id).select_related('sensor_set', 'platform_set')
for s in p.sources:
s.sensor
s.platform
# Sort the list
products = list(products)
products.sort(key=lambda t: ids.index(t.specific_id))
What do you think of this code ? How could I optimize this treatment, cause I feel this is too long. I only have 194 products (max 5 repository related, and not more than 3 sources) and my browser need around 5 seconds to show the result. Is there a way to simply load nested / related object by default ?
Thanks for reading,

Here is the accepted answer: Well, I found out how to solve my interrogation, I don't need to iterate on objects in the view to load them. Just need to create a query set and acces in the template. Like this:
In the view.py
products=Product.objects.filter(specific_id__in=ids).prefetch_related('repository_set', 'source_set').select_related('repository_set', 'source_set')
Then in my template, iterate on :
{% for src in product.source_set.all %}
{{src.platform.shortname}} / {{src.sensor.shortname}}<br/&gt;
{% endfor %}
{% for repo in product.repository_set.all %}
{{repo.url}}%}
{% endfor %}
That's all...


{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 1, 'CommentCount': 0, 'ContentLicense': 'CC BY-SA 3.0', 'CreationDate': '2015:00:08/06/15', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<python><django-models><fetch>', 'Title': 'How optimize fetch and related objects loading?', 'ViewCount': 40, 'id': 31856625, 'user_AboutMe': None, 'user_AccountId': 6746823.0, 'user_DisplayName': 'Loh', 'user_Reputation': 13.0, 'user_Views': 9.0, 'user_id': 5198198.0}}