Building + deploying a Neural Style Transfer app with pre-trained models
Building + deploying a Neural Style Transfer app with pre-trained models

With Flask, Python3, OpenCV, Heroku (and a bunch of other stuff)

Styled image and the uploaded image

Hello internet friends!

I feel like one of the issues I’ve faced with regards to learning software development is that many tutorials are often too technical (and not fun). This is probably because of the nature of software development but I feel like grasping technical concepts is as important as building fun mini-projects that will continue to encourage and spur us on. ༼ つ ◕_◕ ༽つ

I thought it’ll be really cool to build something simple that involves image processing and deploy it online to show off to our friends and family. So today’s tutorial will be a diagrammatically-explained walkthrough of how to build a web app that:

Allow users to upload an image and choose a style

Apply Neural Style Transfer on that image using pre-trained .t7 models

Display the uploaded image and the styled image

Directory structure of this project

A quick explanation of the directory structure above — All the files are needed except test.py. test.py was used for me to locally test and see if my neuralStyleProcess.py script was working correctly. I have 11 .t7 models in my project but you can decide how many you want (downloaded from ProGamerGov and jcjohns). Lastly, you can just ignore __pycache__ coz its auto-generated.

A visual explanation of this web app

In the diagram above, I’ve tried to only include the most important bits that’ll hopefully help you guys understand how the web app works (I called mine pyFlaskOpenCV — bad naming, I know >.<). Also, it’s missing chunks of code, and parts of it are pseudocodes (I’ve labeled them), and the ‘…’s are also NOT actual codes.

Let’s start with the app.py script. Think of this as the app’s root — we use this to set up the routes (something like the ‘control’ that tells things how to interact). This article gives a pretty good explanation of Flask. As we can see in the diagram above, render_template() will render our upload.html template. This means that our upload.html page will be displayed to us and as we can see from the diagram, there is a form with a POST method. With action=“{{url_for(‘upload’)}}”, we call def upload() and in that function, we let users upload image(s) with the following codes:

and we render complete.html.

From complete.html, we again use “url_for(‘send_processed_image’, parameter1, parameter2)” to invoke send_processed_image function that will call our python OpenCV image processing script neuralStyleProcess.py. neuralStyleProcess.py writes the processed image to file with imwrite and returns the processed image filename so send_processed_image function knows what to send from directory to complete.html.

With regard to the neuralStyleProcess.py script, my implementation is largely adapted from pyImageSearch’s tutorial. His article gives a thorough and detailed explanation of how it works. The only parts I've changed are — instead of showing the image, I did an imwrite and I removed the scaling part for the post-processing of the image.

Deploying online with Heroku

You don’t have to use Heroku, there are other options out there but because our models take up quite a bit of space (their sizes can add up), remember to choose servers that will give you a little more storage space (e.g. Heroku’s free tier worked fine for me but PythonAnywhere’s free tier didn’t).

Continuing from our previous diagram, we still need to provide Heroku with three additional files so Heroku “knows” what to do with our web app. These files are — Procfile, requirements.txt, Aptfile.

Again, here’s a diagram to better explain things:

Neon colours, much cool

A Procfile is a text file in the root directory of your application, to explicitly declare what command should be executed to start your app. In my case, here’s mine:

web: gunicorn app:app

Okay please let me rant. BECAUSE OF THIS I SPENT HOURS DEBUGGING!!! Why? Coz they apparently care a lot about spaces. So if you wrote — web:gunicorn app:app/ web:gunicorn app: app/ or any other configuration (okay i haven’t tried all, but I know its space (very) sensitive ಥ_ಥ)

Sigh, moving on…

We use Aptfile to include other additional software that we need to be installed on the dyno. What’s dyno? According to Heroku, “The containers used at Heroku are called “dynos.” Dynos are isolated, virtualized Linux containers that are designed to execute code based on a user-specified command.”

Here is what’s in mine (OpenCV requires the following libraries to run correctly):

libsm6 libxrender1 libfontconfig1 libice6

Finally, I did a

$pip freeze > requirements.txt

in my terminal to generate my requirements.txt. If you’re interested in what’s in it:

click==7.1.1 cycler==0.10.0 Flask==1.1.1 gunicorn==20.0.4 imutils==0.5.3 itsdangerous==1.1.0 Jinja2==2.11.1 kiwisolver==1.1.0 MarkupSafe==1.1.1 matplotlib==3.2.1 numpy==1.18.2 opencv-python==4.2.0.32 pyparsing==2.4.6 python-dateutil==2.8.1 six==1.14.0 Werkzeug==1.0.0

One of the trickiest things about working with libraries like OpenCV-python and NumPy is that it can be really tedious to figure out which version works together and which doesn’t. (Sometimes version 4.2.0.32 works but 4.1.0 might not.) These are the versions that work for me.

Next steps (some suggestions)

Image extension checking for the files uploaded

Resizing the uploaded image

Saving uploaded/processed image(s) to a database so data persists

Other image processing scripts

Better styling of our html pages

Adding in some indicator (e.g. loading animation) on first page load

Building something interactive on top of it (e.g. drawing with javascript etc)

Instead of choosing a pre-trained model, generate art with Generative Adversarial Network (GAN)

Aight, hope that helped someone gain a better understanding of how things work. I’m not sure if this is a better way of going through codes rather than breaking the codes into chunks and then explaining each line — I personally find that most tutorials lack diagrammatical explanations so I thought I’d present things in a way that feels most intuitive to myself — do lemme know what you guys think.

Happy creating! \ (•◡•) /
['Ran', 'Reine']
['Python', 'Web Development', 'Neural Style Transfer', 'Heroku', 'Flask']