How would I go about downloading a file from a submitted link then reuploading to my server for streaming?
Question: I'm working on a project where a user can submit a link to a sound file hosted
on another site through a form. I'd like to download that file to my server
and make it available for streaming. I might have to upload it to Amazon S3.
I'm doing this in Django but I'm new to Python. Can anyone point me in the
right direction for how to do this?


Answer: Here's how I would do it:

  1. Create a model like `SoundUpload` like:
    
        class SoundUpload(models.Model):
        STATUS_CHOICES = (
            (0, 'Unprocessed'),
            (1, 'Ready'),
            (2, 'Bad File'),
        )
        uploaded_by = models.ForeignKey(User)
        original_url = models.URLField(verify_true=False)
        download_url = models.URLField(null=True, blank=True)
        status = models.IntegerField(choices=STATUS_CHOICES, default=0)
    

  2. Next create the view w/a `ModelForm` and save the info to the database.

  3. Hook up a [post-save signal](http://docs.djangoproject.com/en/dev/ref/signals/#django.db.models.signals.post_save) on the `SoundUpload` model that kicks of a [django-celery](http://pypi.python.org/pypi/django-celery) [Task](http://ask.github.com/celery/userguide/tasks.html). This will ensure that the UI responds while you're processing all the data.
    
        def process_new_sound_upload(sender, **kwargs):
       # Bury to prevent circular dependency issues.
       from your_project.tasks import ProcessSoundUploadTask
       if kwargs.get('created', False):
            instance = kwargs.get('instance')
            ProcessSoundUploadTask.delay(instance.id)
    
    
    post_save.connect(process_new_sound_upload, sender=SoundUpload)
    

  4. In the `ProcessSoundUploadTask` task you'll want to:

     * Lookup the model object based on the passed in id.
     * Using [`pycurl`](http://pycurl.sourceforge.net/) download the file to a temporary folder (w/very limitied permissions).
     * Use [`ffmpeg`](http://ffmpeg.org/) (or similar) to ensure it's a real sound file. Do any other virus style checks here (depends on how much you trust your users). If it turn out to be a bad file set the `SoundUpload`.status field to `2` (Bad File), save it, and return to stop processing the task. Perhaps send out an email here.

     * Use [`boto`](http://code.google.com/p/boto/) to upload the file to s3. See [this](http://ferrouswheel.me/2009/12/upload-a-file-to-s3-with-boto/) example.

     * Update the `SoundUpload`.download_url to be the s3 url, the status to be "processed" and save the object.
     * Do any other post-processing (sending notification emails, etc.) 

The key to this approach is using `django-celery`. Once the task is kicked off
through the post_save signal the UI can return, thus creating a very "snappy"
experience. This task gets put onto an
[AMQP](http://www.amqp.org/confluence/display/AMQP/Advanced+Message+Queuing+Protocol)
message queue that can be processed by multiple workers (dedicated EC2
instances, etc.), so you'll be able to scale without too much trouble. This
may seem like a bit overkill, but it's really not as much work as it seems.

