Why is my texture synthesis algorithm only producing blocky / noisy, non-sensible output?
My end-goal is to create a script for neural style transfer, however, during writing code for said task, I stumbled upon a certain problem: the texture synthesis part of the algorithm seemed to have some problems with reproducing the artistic style. In order to solve this, I decided to create another script where I'd try to solve the task of texture synthesis using a neural network on its own.

TL;DR ... even after tackling the problem on its own, my script still produced blocky / noisy, non-sensible output.

I've tried having a look at how other people have solved this task, but most of what I found were more sophisticated solutions ("fast neural-style-transfer", etc.). Also, I couldn't find too many PyTorch implementations. 

Since I've already spent the past couple of days on trying to fix this issue and considering that I'm new to the PyTorch-Framework, I have decided to ask the StackOverflow community for help and advice. 

I use the VGG16 network for my model ... 

class VGG16(nn.Module):
    def __init__(self):
        super(VGG16, self).__init__()
        vgg_fs = models.vgg16(pretrained=True).features

        self.sl1 = nn.Sequential()
        self.sl2 = nn.Sequential()
        self.sl3 = nn.Sequential()
        self.sl4 = nn.Sequential()
        self.sl5 = nn.Sequential()

        for i in range(4):
            self.sl1.add_module(str(i), vgg_fs[i])
        for i in range(4, 9):
            self.sl2.add_module(str(i), vgg_fs[i])
        for i in range(9, 16):
            self.sl3.add_module(str(i), vgg_fs[i])
        for i in range(16, 23):
            self.sl4.add_module(str(i), vgg_fs[i])
        for i in range(23, 30):
            self.sl5.add_module(str(i), vgg_fs[i])

        for p in self.parameters():
            p.requires_grad_(False)

    def forward(self, x):
        h = self.sl1(x)
        h1 = h
        h = self.sl2(h)
        h2 = h
        h = self.sl3(h)
        h3 = h
        h = self.sl4(h)
        h4 = h
        h = self.sl5(h)
        h5 = h

        return_tuple = namedtuple('hidden_states', ['h1', 'h2', 'h3', 'h4', 'h5'])
        ret = return_tuple(h1, h2, h3, h4, h5)

        return ret


Then, I have some functions for gram-matrix computation and normalization ...

def comp_gram(f):
    (b, c, h, w) = f.shape
    f = f.view(b, c, h * w)
    g = f.bmm(f.transpose(1, 2)) 
    g = g / (c * h * w)
    return g

def norm(b):
    mean    = torch.Tensor([0.485, 0.456, 0.406]).cuda().view(3, 1, 1)
    std     = torch.Tensor([0.229, 0.224, 0.225]).cuda().view(3, 1, 1)
    return (b - mean) / std


And finally, my training-function ... 

def train(model, style_img, w=224, h=224, iters=32, lr=1):
    style = torch.from_numpy(np.array(Image.open(style_img))).cuda().float()
    style = style / 255.
    style = style.view(1, style.size()[2], *style.size()[:2])

    style_fts = model(norm(style))
    style_gms = [comp_gram(f) for f in style_fts]

    img = torch.rand(*style.size()[:2], h, w, requires_grad=True, device='cuda')
    optimizer = optim.Adam([img], lr=lr)
    mse_loss = nn.MSELoss()

    plt.ion()
    for i in range(iters):
        optimizer.zero_grad()
        actvs = model(norm(img))

        lss = 0.
        for f, gm in zip(actvs, style_gms):
            g = comp_gram(f)
            lss += mse_loss(g, gm)

        lss.backward()
        optimizer.step()

        if (i % 5 == 0) or (i == iters - 1):
            plt.title('Iter#{:04d}'.format(i))
            plt.imshow(img.detach().cpu().view(*img.shape[2:], img.shape[1]))
            plt.pause(1e-3)

        print('[iter#{:04d}]: Loss\t-&gt; {:}'.format(i, lss.item()))

    plt.ioff()

    plt.subplot(121)
    plt.title('Original')
    plt.imshow(style.cpu().view(*style.size()[2:], style.size()[1]))

    plt.subplot(122)
    plt.title('Style')
    plt.imshow(img.detach().cpu().view(*img.size()[2:], img.size()[1]))

    plt.show()


I expected the algorithm to produce the style of the painting in some way, shape or form, but that didn't happen. ‍♂️

(Since I sadly don't have enough reputation to post an image, you can find a picture of my non-sensible output here: https://mattmoony.github.io/cdn/conv-net_pytorch-style-transfer/problem.png)

I sincerely hope that you will be able to help me out and that I'm able to use this issue as a learning experience. Thank you! 

Hurrah! 

After yet another day of researching and testing, I've finally discovered the bug in my code. 

The problem doesn't lie with the training process or the model itself, but rather with the lines responsible for loading the style image. (this article helped me discover the issue)

So... I added the following two functions to my script ... 

def load_img(p):
    img = Image.open(p).convert('RGB')
    im_transform = transforms.Compose([
        transforms.ToTensor()
    ])
    img = im_transform(img).unsqueeze(0).cuda()

    return img

def to_img(t):
    img = t.cpu().clone().detach()
    img = img.numpy().squeeze()
    img = img.transpose(1, 2, 0)
    img = img.clip(0, 1)

    return img


And used them in my training function ... 

def train(...):
...
    style = load_img(style_img)
...
            if i % 5 == 0:
                plt.title('Iter#{:04d}'.format(i))
                plt.imshow(to_img(img))
                plt.pause(1e-3)
...
&lt;etc.&gt;
...


I'm glad that I was able to figure out what the problematic part of my script was and I'll do my best to avoid similar bugs in the future. 

I hope that this helps anyone experiencing similar problems or that it at least inspires them to keep on chasing for a solution. 
