Is the Unified Theory of Climate (Nikolov & Zeller) compatible with the AGW/GHG theory in any respect?
The paper you linked lists as its first reference Volokin and ReLlez, 2014; a paper that addresses the magnitude of the Earth's greenhouse effect. The validity of the paper you found largely hinges on the reputation of the Volokin and ReLlez, so we will investigate that one first. 

The science of Volokin and ReLlez

The traditional way to determine the 'greenhouse effect' of a planet is to compare actual planetary temperatures with Stefan-Boltzmann blackbody temperature based on the luminosity of the star, distance from star, and the albedo and emissivity of the planet. This calculation is done in Volokin and ReLlez and can also be found on Wikipedia. 

Volokin and ReLlez are arguing that the actual greenhouse effect should be measured using the albedo of a bare planet stripped of its clouds. The Earth's albedo is 0.294, but they propose a bare albedo of the Earth of between 0.122 and 0.13. They then make an argument that simple albedo calculations are not very accurate, and that a use of Holder's Inequality should be used to integrate the effects of albedo and radiation incidence angle over the exposed surface of the bare planet. Redoing the Stefan-Boltzmann expected temperature calculations with this new method, they then calculate that actual 'greenhouse effect' of the Earth is more like 90 K, instead of the 33 K normally cited. This is significant because temperature magnitude of the Earth's greenhouse effect affects the climate sensitivity factor of global warming models.

I actually think this paper is pretty interesting. The core of its mathematical arguments seems valid (though I didn't try to work any of it out myself) However, the conclusions are not very good, and not very related to much of the mathematical arguments in the body. While there is an argument to be made that calculating expected surface temperature should assume a bare planet, the authors are basically concluding that all temperature differences between the bare planet and the observed Earth should be assigned to the greenhouse effect. That just isn't supported by the information they present. The Earth's actual atmosphere and biosphere are much more dynamic than the bare model they are presenting; the formation of high-albedo clouds by equatorial rainforests,and the presence of high-albedo sand in the sunniest places on Earth are just two non-greenhouse related effects that account for temperature differences between the author's ideal bare Earth and the real Earth. 

While they consider a lot of interesting factors, I think Volokin and ReLlez are guilty of making too much of a spherical cow estimate. The author's make an argument for one thing, then make some huge leaps to an unsupported conclusion. All in all, I don't think that the author's conclusion of a 90 K greenhouse effect is valid.

The publication of the reference

Whatever its scientific merits, the paper doesn't look very good upon meta review. First off, the authors used fake names to publish the paper in an open access site. In fact, it is the same two authors of the paper you linked, except that they spelled their names backwards (Nikolov -> Volokin; Zeller -> ReLlez). Ummm...what? Also, they later retracted the paper. Also also, the journal they published in is now defunct. All in all, this is not a sign that the paper is good science. 

Back to the original paper

So the paper that you are asking about (Nikolov and Zeller, 2017) is based on a paper by the same authors which has not-very-good conclusions, and was generally published in a shady manner. That's a big black mark at the very outset. 

This paper is pretty long and detailed, and I admit I did not read all of it. But the kill shot for me was on the very first page:


  We  began  our  study  with  the  premise  that  processes 
  controlling   the  Global  Mean  Annual  near-surface  Temperature 
  (GMAT)  of  Earth   are  also  responsible  for  creating  the 
  observed  pattern  of  planetary   temperatures across the Solar
  System. Thus, our working hypothesis was  that  a  general  physical 
  model  should  exist,  which  accurately  describes   equilibrium 
  GMATs  of  planets  using  a  common  set  of  drivers.


That is great, but we have already gone over how these authors are attempting to oversimplify a bare planet approach for predicting expected temperatures. The predictor variables they are using are stellar irradiance, blackbody reference temperature, greenhouse gas partial pressure and density, atmospheric pressure and density, and reference pressure for greenhouse fluids. We can see the authors are once again ignoring the effects of cloud formation and the way the biosphere affects albedo, amongst other things.

On a methodology note, the authors use a regression model to try to get some useful results. As a statistician (not a planetary scientist), this strikes me as a poor choice. Regression analysis of complex systems is a good way to get insight into what factors are important in affecting this system. I think they can be very useful for predicting trends, but I do not believe they are good for providing predictions that are accurate by physical science standards. 

For example, I'm writing a paper right now that uses regression analysis to predict subway ridership at each station on a subway network. This is a very complex system (like planetary atmospheres), and our goal is to get error scores of 0.2 or so. That is great for predicting subway ridership, not so much for planetary temperatures. An error score of 0.2 is like predicting a temperature to be 300 K but it is actually 360 K. That's not good enough. So I just don't know what the authors are really trying to accomplish using a regression approach, especially with so few predictor variables. The physics behind the interaction of the predictor variables is pretty well known, so there isn't a lot of value if determining the correlations of the predictor values to the planet surface temperature; it is as if they are using the regression to test if the physics is correct. Finally, the data set is way too small to be using a regression analysis. Their only available data points are the planets and moons in our own solar system. 

Conclusion

Well I wrote a lot there, and hopefully it is all comprehensible. All in all, there are some interesting bits to Nikolov and Zeller's approach, but I think they are over simplifying some things and are using a non-physical science applicable methodology to obtain questionable conclusions.

Honestly, I don't think that anyone should be treating their research pejoratively (I'm looking at you Andrew Jon Dodds in the comments). It is interesting, and it is certainly novel (from everything I've read). It just doesn't have great conclusions. And that's fine! Most research probably doesn't lead anywhere useful. At least they tried something. People seem to be mad at them just because the results are contrary to the general climate change consensus. Research should be appraised on its merits (or lack thereof) and not on whether its conclusions match preconceptions.