Weighting influence of two neural networks on classification I'm training a model that has two neural networks. One of them is a resnet18 CNN which has as it's input images. The other one is a small one hidden layer network that has as it's input four other variables.

At this moment I concatenate the outputs of these networks in the first (and only) fully connected layer, after which datapoints are classified into three classes in the classification layer. 

I was wondering how I could design my network in such a way that the two different neural networks have a specific amount of influence on the classification. For instance, I'd like the CNN to have 60% 'say'/influence in the eventual classification, and have the other NN have 40% 'say'/influence.

I can't find an intuitive way to to this in the current structure. The only way I can imagine doing it is by separating the networks, and then weighting the losses of both the networks. Does anybody know another way to achieve this without separating the networks?
As neural networks are non-parametric, it would usually be the case that you train the combined model (as you describe it, but perhaps with additional FC-layers at the end) and let the model as a single entity learn/decide how best to combine the output of your two sub-models.

If you want to the hard-coded percentages you mention, then the most straight-forward way would be to do what you mention in the last paragraph. 

People often train the same (or slightly differing models) independently on the same dataset, then average the predictions of these models on the test set, which can improve overall performance by integer percentages.