title: Where/how is Django ManyToManyField represented in the database?
body: <p><strong><em>UPDATE:</em></strong> <em>Just found out that the ManyToManyField is causing the admin interface to crash, when a specific album is selected. I commented them out, commented out all references to it, reran <code>makemigrations</code> and <code>migrate</code>, and now the admin interface works again...which leaves me even farther away from making this "favorite" column work &nbsp; :(  See this followup: <a href="https://stackoverflow.com/questions/25435486/why-is-django-manytomanyfield-causing-admin-interface-to-crash-why-is-no-throug">Why is Django ManyToManyField causing admin interface to crash? Why is no through table being created?</a></em></p>
  
  <hr>
  
  <p>Background: My goal is to make the "Favorite?" column in this webpage reflect the favorite albums of the currently-logged-in user, where each is either "no" or "yes", and is a clickable link to toggle the choice. (When not logged in, they will all be grey "n/a"-s.)</p>
  
  <p><img src="https://i.stack.imgur.com/1KrAm.jpg" alt="enter image description here"></p>
  
  <p>Therefore, for each album, there may be exactly <em>zero or one</em> "has favorited" entry per user. If the entry exists, they've favorited it. If it doesn't exist, they didn't.</p>
  
  <p>Here is my <code>Album</code> model, with the <code>favorited_by_users</code> many-to-many column (full <code>models.py</code> at the bottom):</p>
  
  <pre><code>class Album(models.Model):
      OFFICIALITY = (
          ('J', 'Major studio release'),
          ('I', 'Non-major official release'),
          ('U', 'Unofficial'),
      )
      title = models.CharField(max_length=70)
      description = models.TextField(max_length=500, default="", null=True, blank=True)
      pub_date = models.DateField('release date')
      officiality = models.CharField(max_length=1, choices=OFFICIALITY)
      is_concert = models.BooleanField(default=False)
      main_info_url = models.URLField(blank=False)
      thumbnail = models.FileField(upload_to=get_upload_file_name, blank=True, null=True)
  
      #virtual field to skip over the through table.
      songs = models.ManyToManyField("Song", through="AlbumSong")
  
      favorited_by_users = models.ManyToManyField(User)
  
      def __str__(self):
          return  self.title
  
      class Meta:
          #Default ordering is by release date, ascending.
          ordering = ['pub_date']
  </code></pre>
  
  <p>I originally had this <code>FavoriteAlbum</code> model, but since it has no extra information beyond the foreign keys, it was recommended that I eliminate it in favor of the above many-to-many column.</p>
  
  <pre><code>class FavoriteSongs(models.Model):
      user = models.ForeignKey(User)
      song = models.ForeignKey(Song)
  
      class Meta:
          unique_together = ('user', 'song',)
  
      def __str__(self):
          return  "user=" + str(self.user) + ", song=" + str(self.song)
  </code></pre>
  
  <p>What I need to do is a "left join" between album and user, where all albums are selected, and any favorites by the currently-logged-in user are joined to it (<code>None</code> if they haven't favorited it). I don't know what to do.</p>
  
  <p>I've also been told about the <a href="https://docs.djangoproject.com/en/1.7/ref/models/querysets/#extra" rel="nofollow noreferrer"><code>extra()</code></a> function to do this join.  The currently-working query in the view's <code>get_queryset()</code> is</p>
  
  <pre><code>return  super(AlbumList, self).get_queryset().order_by("pub_date")
  </code></pre>
  
  <p>(Full <code>views.py</code> below.) My current guess is this:</p>
  
  <p><code>return  super(AlbumList, self).get_queryset().order_by("pub_date").extra(select={"is_favorite": "favorited_by_users__id = " + str(request.user.id) })</code></p>
  
  <p>But, while this doesn't crash, the value of each <code>{{ is_favorite }}</code> in the template is <em>nothing</em> (the empty string). This makes sense since there's nothing in the database yet, but what now? I have no idea if this is the correct Django query.</p>
  
  <p>I want to add an item in the database to test this, with a manual SQL statement in postgres (not via a Django command yet), but how and where do I do this?</p>
  
  <p>I've successfully run <code>makemigrations</code> and then <code>migrate</code> with this new m2m column (and without the <code>FavoriteSongs</code> model), but I see nothing in the database that represents the is-favorite value. There's no extra column in <code>billyjoel_album</code>, and no through table akin to <code>billyjoel_favoritealbum</code>. So where/how is this data stored in the database?</p>
  
  <p>(Any other advice regarding this extra "favorite" column would be appreciated as well!)</p>
  
  <p>Thanks.</p>
  
  <hr>
  
  <p>models.py</p>
  
  <pre><code>from  django.db import models
  from  django.contrib.auth.models import User
  from  time import time
  
  def get_upload_file_name(instance, filename):
      return  "uploaded_files/%s_%s" % (str(time()).replace(".", "_"), filename)
  
  class Album(models.Model):
      OFFICIALITY = (
          ('J', 'Major studio release'),
          ('I', 'Non-major official release'),
          ('U', 'Unofficial'),
      )
      title = models.CharField(max_length=70)
      description = models.TextField(max_length=500, default="", null=True, blank=True)
      pub_date = models.DateField('release date')
      officiality = models.CharField(max_length=1, choices=OFFICIALITY)
      is_concert = models.BooleanField(default=False)
      main_info_url = models.URLField(blank=False)
      thumbnail = models.FileField(upload_to=get_upload_file_name, blank=True, null=True)
  
      #virtual field to skip over the through table.
      songs = models.ManyToManyField("Song", through="AlbumSong")
  
      favorited_by_users = models.ManyToManyField(User)
  
      def __str__(self):
          return  self.title
  
      class Meta:
          #Default ordering is by release date, ascending.
          ordering = ['pub_date']
  
  
  class Song(models.Model):
      name = models.CharField(max_length=100)
      description = models.TextField(max_length=500, default="", null=True, blank=True)
      length_seconds = models.IntegerField()
      lyrics_url = models.URLField(default="", blank=True, null=True)
      albums = models.ManyToManyField("Album", through="AlbumSong")
  
      favorited_by_users = models.ManyToManyField(User)
  
      def get_length_desc_from_seconds(self):
          if(self.length_seconds == -1):
              return  "-1"
          m, s = divmod(self.length_seconds, 60)
          h, m = divmod(m, 60)
          if(h):
              return  "%d:%02d:%02d" % (h, m, s)
          else:
              return  "%d:%02d" % (m, s)
  
      def __str__(self):
          return  self.name
  
  class AlbumSong(models.Model):
      song = models.ForeignKey(Song)
      album = models.ForeignKey(Album)
      sequence_num = models.IntegerField()
  
      class Meta:
          unique_together = ('album', 'sequence_num',)
          unique_together = ('album', 'song',)
  
      def __str__(self):
          return  str(self.album) + ": " + str(self.sequence_num) + ": " + str(self.song)
  </code></pre>
  
  <p>views.py</p>
  
  <pre><code>from  .models import Album, Song, AlbumSong
  from  datetime import datetime, timedelta
  from  django.core.context_processors import csrf
  from  django.shortcuts import render, render_to_response
  from  django.views.generic import DetailView, ListView
  from  enum import Enum
  
  def get_str_with_appended(string, between_if_str_non_empty, new_value):
      if(len(string) == 0):
          return  new_value
      else:
          return  string + between_if_str_non_empty + new_value
  
  class PrependQuestionMark(Enum):
      YES, NO = range(2)
  
  def get_url_param_string_from_params(prepend_question_mark=PrependQuestionMark.YES, **kwargs_all_params):
      param_list = ""
      for  key in iter(kwargs_all_params):
          value = kwargs_all_params[key]
          if(value is not None):
              param_list = get_str_with_appended(param_list, '&amp;', str(key) + "=" + str(value))
  
      if(len(param_list) == 0):
          return  param_list;
  
      if(prepend_question_mark == PrependQuestionMark.YES):
          return  "?" + param_list
      else:
          return  param_list
  
  class AlbumList(ListView):
      model = Album
      context_object_name = "albums"
  
      #Derived from irc/#dango/tbaxter...START
      def dispatch(self, request, *args, **kwargs):
                                               #default to asc
          self.sort_order = request.GET.get("sort_order", None)
          self.sort_item = request.GET.get("sort_item", None)
          self.csrf_token = csrf(request)["csrf_token"]
          self.logged_in_user = request.user
  
          #self.csrf_token = request.GET.get("csrf_token", None)
          return super(AlbumList, self).dispatch(request, *args, **kwargs)
  
      def get_queryset(self):
          #Item zero in both is the default
          #should be static global
          asc_desc_list = ["asc", "dsc"]
          sort_by_types = ["pub_date", "title"]
  
          if(self.sort_order is None  and  self.sort_item is None):
              #Use default ordering
              return  super(AlbumList, self).get_queryset()
  
          #Custom ordering requested
  
          sort_order = self.sort_order
          sort_item = self.sort_item
  
          if(sort_order is None  or
                  sort_order not in asc_desc_list):
              sort_order = asc_desc_list[0]
          if(sort_item is None  or
                  sort_item not in sort_by_types):
              sort_item = sort_by_types[0]
  
          order_minus = ""  if  sort_order == "asc"  else "-"
  
          return  super(AlbumList, self).get_queryset().order_by(order_minus + sort_item).extra(select={"is_favorite": "favorited_by_users__id = " + str(self.logged_in_user.id) })
  
      def get_context_data(self, **kwargs):
          context = super(AlbumList, self).get_context_data(**kwargs)
  
          context["sort_order"] = self.sort_order
          context["sort_item"] = self.sort_item
  
          context["url_params"] = get_url_param_string_from_params(
              sort_item=self.sort_item,
              sort_order=self.sort_order,
              csrf_token=self.csrf_token)
  
          return  context
  
  
  class AlbumDetail(DetailView):
      model = Album
      context_object_name = "album"
  
      def dispatch(self, request, *args, **kwargs):
                                               #default to asc
          self.sort_order = request.GET.get("sort_order", None)
          self.sort_item = request.GET.get("sort_item", None)
          self.csrf_token = csrf(request)["csrf_token"]
  
          return super(AlbumDetail, self).dispatch(request, *args, **kwargs)
  
      def get_context_data(self, **kwargs):
          #Call the base implementation first to get a context
          context = super(AlbumDetail, self).get_context_data(**kwargs)
  
          #Add in the required extra info: album_songs, ordered by
          #sequence_num
  
          #select_related is to query the database for all songs at once, here
          #in the view, to prevent the template from pin-pricking the database
          #in each for loop iteration. For large datasets, this is critical.
          context['album_songs'] = kwargs["object"].albumsong_set.order_by('sequence_num').select_related("song")
  
          context["url_params"] = get_url_param_string_from_params(
              sort_item=self.sort_item,
              sort_order=self.sort_order,
              csrf_token=self.csrf_token)
  
          return  context
  </code></pre>
  
  <p>album_list.html</p>
  
  <pre><code> {% extends "base.html" %}
   {% load bj_filters %}
   {% block title %}Billy Joel Album Browser{% endblock %}
  
   {% block sidebar %}
   &lt;UL&gt;
    &lt;LI&gt;&lt;a href="{% url 'album_list' %}{{ url_params }}"&gt;All albums&lt;/A&gt;&lt;/LI&gt;
    &lt;LI&gt;&lt;a href="/admin/"&gt;Admin&lt;/A&gt;&lt;/LI&gt;
   &lt;/UL&gt;
   {% endblock %}
  
   {% block content %}
  
   &lt;TABLE ALIGN="center" WIDTH="100%" BORDER="1" CELLSPACING="0" CELLPADDING="4" BGCOLOR="#EEEEEE"&gt;&lt;TR ALIGN="center" VALIGN="middle"&gt;
   {% if  user.is_authenticated %}
    &lt;TD&gt;My profile (&lt;a href="{% url 'accounts_logout' %}"&gt;Logout&lt;/A&gt;)&lt;/TD&gt;
   {% else %}
    &lt;TD&gt;&lt;a href="{% url 'accounts_login' %}"&gt;Login&lt;/A&gt; to view your favorites&lt;/TD&gt;
   {% endif %}
   &lt;/TR&gt;&lt;/TABLE&gt;
  
   &lt;H1&gt;Billy Joel Album Browser&lt;/H1&gt;
  
   &lt;!--
   &lt;P&gt;url_params={{ url_params }}&lt;/P&gt;
    --&gt;
  
   {% if  albums.count &gt; 0 %}
    &lt;P&gt;Officiality: &lt;IMG SRC="/static/images/major.jpg" height="20"/&gt;=Major studio release, &lt;IMG SRC="/static/images/minor.jpg" height="20"/&gt;=Official release, &lt;IMG SRC="/static/images/unofficial.jpg" height="20"/&gt;=Unofficial&lt;/P&gt;
    &lt;TABLE ALIGN="center" WIDTH="100%" BORDER="1" CELLSPACING="0" CELLPADDING="4" BGCOLOR="#EEEEEE"&gt;&lt;TR ALIGN="center" VALIGN="middle"&gt;
       &lt;TD&gt;&lt;B&gt;&lt;U&gt;&lt;a href="{% url 'album_list' %}?sort_item=title&amp;sort_order=
          {% if  sort_item == 'pub_date' %}asc{% else %}
             {{ sort_order|multival_to_str:'asc,dsc-&gt;dsc,asc,dsc' }}
          {% endif %}
       &amp;csrf_token={{ csrf_token }}"&gt;Title&lt;/A&gt;&lt;/U&gt;&lt;/B&gt;&lt;BR&gt;&lt;I&gt;&lt;FONT SIZE="-1"&gt;(click a title to view its song list)&lt;/FONT&gt;&lt;/I&gt;&lt;/TD&gt;
       &lt;TD&gt;&lt;B&gt;&lt;U&gt;&lt;a href="{% url 'album_list' %}?sort_item=pub_date&amp;sort_order=
          {% if  sort_item == 'title' %}asc{% else %}
             {{ sort_order|multival_to_str:'asc,dsc-&gt;dsc,asc,dsc' }}
          {% endif %}
       &amp;csrf_token={{ csrf_token }}"&gt;Released&lt;/A&gt;&lt;/U&gt;&lt;/B&gt;&lt;/TD&gt;
       &lt;TD&gt;Officiality&lt;/TD&gt;
       &lt;TD&gt;Concert&lt;/TD&gt;
       &lt;TD&gt;Wiki&lt;/TD&gt;
       &lt;TD&gt;Favorite?&lt;/TD&gt;
    {% for  album in albums %} &lt;!-- No colon after "albums" --&gt;
    &lt;/TR&gt;&lt;TR&gt;
       &lt;TD VALIGN="top"&gt;
          {% if  album.thumbnail %}
             &lt;img src="/static/{{ album.thumbnail }}" width="25"/&gt;
          {% else %}
             &lt;img src="/static/images/white_block.jpg" width="25"/&gt;
          {% endif %}
          &amp;nbsp; &lt;a href="/albums/get/{{ album.id }}{{ url_params }}"&gt;{{ album.title }}&lt;/a&gt;
          {% if  album.description %}
             &lt;BR/&gt;&lt;FONT SIZE="-1"&gt;&lt;I&gt;{{ album.description|truncatewords:10 }}&lt;/I&gt;&lt;/FONT&gt;
          {% endif %}
       &lt;TD&gt;{{ album.pub_date|date:"m/y" }}&lt;/TD&gt;
       &lt;TD&gt;&lt;IMG SRC="/static/images/{{ album.officiality|multival_to_str:"J,I,U-&gt;major,minor,unofficial,broken_image"}}.jpg" height="20"/&gt;&lt;/TD&gt;
       &lt;TD&gt;{{ album.is_concert|yesno:"Yes,No" }}&lt;/TD&gt;
       &lt;TD&gt;&lt;A HREF="{{ album.main_info_url }}"&gt;Wiki&lt;/A&gt;&lt;/TD&gt;
       &lt;TD&gt;&lt;I&gt;n/a {{ is_favorite }}&lt;/I&gt;&lt;/TD&gt;
    {% endfor %}
    &lt;/TR&gt;&lt;/TABLE&gt;
   {% else %}
    &lt;P&gt;&lt;I&gt;There are no albums in the database.&lt;/I&gt;&lt;/P&gt;
   {% endif %}
  
   {% endblock %}
  </code></pre>