4 Layer PCB Altium - 2nd Layer Ground, others Thermal
Obvisouly I will need vias which go only from 1st Layer to 2nd (to ground). Is that maybe too expensive to manufacture ?


This is called a "blind via" and they are more expensive than regular via's, typically you don't need them though. 



(This is a 6 layer board it seems, but imagine it's 4)

In the image above you have a "4" layer board. The blind via connects layer 1 to layer 2, but doesn't go to layer 3 or 4. But take a look at the "through via", it connects Layer 1 and Layer 4, but not 2 or 3. This ends up looking like a small hole on the inner layers that don't connect. Your board design software accounts for this when you add a via, removing some copper around the layers that it doesn't connect to.


  I am only concerning about my connectors: the pins are soldered on the bottom. Normally THD are like vias, so it also has a connection on the top layer. Which I can connect then to a local ground plane which goes with vias down to the 2nd Layer. Is that alright ?


THD are kind of like via's, with no internal plane connections (usually, sometimes they do). You can safely solder these without shorting out your internal planes. You don't need to worry about "local ground planes", if the through-hole connects to a net with the same name as the internal layer, like GND, it will connect internally.