title: Storing images and thumbnails on s3 in django
body: <p>I'm trying to get my images thumbnailed and stored on s3 using django-storages, boto, and sorl-thumbnail. I have it working, but it's very slow, even with small images. I don't mind it being slow when I save the form and upload the images to s3, but I'd like it to display the image quickly after that.</p>
  
  <p>The answer to this SO question explains that the thumbnail won't be created until first access, but that you can use get_thumbnail() to create it beforehand. </p>
  
  <p><a href="https://stackoverflow.com/questions/5562942/django-s3-boto-sorl-thumbnail-suggestions-for-optimisation">Django + S3 (boto) + Sorl Thumbnail: Suggestions for optimisation</a></p>
  
  <p>I'm doing that, and now it seems that all entries into the thumbnail_kvstore table are created when uploading the image, rather than when it is displayed. </p>
  
  <p>The problem is that the page displaying the image is still really slow. Looking at the logging panel in the debug toolbar, it looks like there is still lots of communication with s3. It seems like after the image and thumbnails are uploaded and cached, page should render quickly without communicating with s3. </p>
  
  <p>What am I doing wrong? Thanks!</p>
  
  <p><strong>Update</strong>: weak hack seems to have gotten it working, but I'd love to know how to do this properly:</p>
  
  <p><a href="https://github.com/asciitaxi/sorl-thumbnail/commit/545cce3f5e719a91dd9cc21d78bb973b2211bbbf" rel="noreferrer">https://github.com/asciitaxi/sorl-thumbnail/commit/545cce3f5e719a91dd9cc21d78bb973b2211bbbf</a></p>
  
  <p><strong>Update</strong>: more information for @sorl</p>
  
  <p>I'm working with 2 views:</p>
  
  <p>ADD VIEW: In this view I submit the form to create the model with the image in it. The image is uploaded to s3. In a post_save signal, I call get_thumbnail() to generate the thumbnail before it's needed:</p>
  
  <pre><code>im = get_thumbnail(instance.image, '360x360')
  </code></pre>
  
  <p>DISPLAY VIEW: In this view I display the thumbnail generated in the add view:</p>
  
  <pre><code>    {% thumbnail object.image "360x360" as im %}
      &lt;img src="{{ im.url }}" width="{{ im.width }}" height="{{ im.height }}"&gt;
      {% endthumbnail %}
  </code></pre>
  
  <p>Without the patch:</p>
  
  <p>ADD VIEW: creates 3 entries in the kvstore table, accesses the cache 10 times (6 sets, 4 gets), logging tab of debug toolbar says "establishing HTTP connection" 12 times</p>
  
  <p>DISPLAY VIEW: still just 3 entries in the kvstore table, just 1 get from cache, but debug toolbar says "establishing HTTP connection" 3 times still</p>
  
  <p>With only the change on line 122:</p>
  
  <p>ADD VIEW: same as above, except the logging only says "establishing HTTP connection" 2 times
  DISPLAY VIEW: same as above, except the logging only says "establishing HTTP connection" 1 time</p>
  
  <p>Also adding the change on line 118:</p>
  
  <p>ADD VIEW: same as above, but now we are down to 2 "establishing HTTP connection" messages
  DISPLAY VIEW: same as above, with no logging messages at all</p>
  
  <p><strong>UPDATE</strong>: It looks like storage._setup() is called twice, and storage.url() is called once. Based on the timing, I'd say each one makes connections to s3:</p>
  
  <pre><code>1304711315.4
  _setup
  1304711317.84
  1304711317.84
  _setup
  1304711320.3
  1304711320.39
  _url
  1304711323.66
  </code></pre>
  
  <p>This seems to be reflected by the boto logging, which says "establishing HTTP connection" 3 times.</p>