From time to time, there is need in web application for users to upload file or multiple files to server, whether it is image or video or some text file.
When button is pressed, form is submitted and whole content is uploaded to server. This usually works for smaller files. For bigger files this approach (few tens of mega bytes) can be inefficient due to memory consumption and time, since after you upload content to server, you still need to save it somewhere (either in database or some kind of storage).
Google Cloud Platform has product called Google Cloud Storage which is suitable (among many things) for storing uploaded user files but it can be also used for archiving data for example where size of single file can be up to 5TB.
Full code for this example is on https://github.com/zdenulo/gcs_multiple_upload, I will discuss here only the most important parts of process.
Handle files function loops over selected files and first makes request to backend url "/get_signed_url" which returns signed url from Cloud Storage to which file data will be uploaded. After that it calls uploadCallback where in upload function through Ajax PUT request, file data is uploaded to Cloud Storage. Upon upload filename and time is displayed on page and postUploadHandler is called to handle post upload process.
For backend part I created FileModel which is using Cloud Datastore to store path in Cloud Storage plus some possible other custom fields.
from request I get filename and content type, I create blob (object) in bucket, then I am using generate_signed_url method where main parameter is expiration time (in this example set to 2 hours) and explicit method PUT which is used for file uplod (otherwise GET method is used to generate signed url for reading file in Cloud Storage). I create then entity for FileModel and in response I return json data with url and key of entity. The reason why I am returning key is that after upload process is completed I can fetch entity and do some custom task with that object, for exampe trigger some background process which will process the file for example.
Cool thing with this process that it allows to upload files in parallel, which reduces total upload time.
or via web interface in console.cloud.google.com. In left menu go under Storage Section and click on Storage option and then Browse.
You need to type name for bucket (needs to be unique across whole Cloud Storage) and click Create.
Of course setting localhost as allowed domain is only good for development / testing purpose.
Next step is to create Service Account through which we will access Cloud Storage. This is done in IAM section. Under IAM section go to Service accounts and then click on Create Service Account.
You need to set up Service Account name (for display purposes), Service Account ID and set role to be Storage / Storage Object Admin. If you already have some service account which you want to use, you can assing this role to that account. Checkout Furnish a new private key option so you can download file in json format and use it in project.
Like I mentioned before we need some kind of backend which will render html page and generate signed url for bucket in Cloud Storage. Google App Engine is perfect solution since it allows quick and easy deployment of code and scales automaticaly.
Code for backend is written in Python and cliend library for Storage is used to simplify work with Cloud Storage. I am using Google App Engine Standard as backend server but this should work on any other type. In app.yaml file bucket name and service json file name and project id should be set.
Here is screenshot of requests when uploading 3 files around 65, 67, 69 MB.
This is not the only way how to upload files. Part of this process (generating signed urls) can be used for mobile applications for files upload.