Why is 人口 in this sentence 多, not 大?
Interesting question. I never think that 人口多(少) should be said as 人口大(小) in English way. As you said, 人口 seems meaning people more in Chinese sense. So we should say 人口多/人口少.


  Can I say "中国是世界上人口最大的国家"?


No, it's quite weird.


  俄罗斯是世界上国土最大的国家. I put 大 here. Is it correct? Would 多 be correct?


Yes, 大 is correct, 多 is not. 国土 usually refer to the area, i.e. the antional territory area, then we should say 大(小), not 多(少).

As a supplementary, 规模 (scope/scale) could be described with 大/小, so you can say "中国是世界上人口规模最大的国家".

Another supplementary, 人口 belongs to a kind of usage that combines a noun and quantifier to make a 集合名词 (collective noun). E.g. 车辆, 人口, 纸张, 船只, 布匹, 马匹.