Extending Background Image
What you're looking for basically is "Inpainting".

In Photoshop is called "Content Aware Fill".  

You should look for articles for Inpainting.

Trivial option would be using "Replication" of the border and later enforce some smoothness (Either by Gaussian Blur or even Poisson Image Editing).

P.S.
We'd be happy if you could join our Community Proposal at http://area51.stackexchange.com/proposals/86832 for Image Processing and Computer Vision.
We need more people to follow, up vote suggested questions with less than 10 up votes and suggest new questions.
Thank You.