Should I remove this gray layer on freshly bought steel?
Looks like mill scale.  It's normally removed prior to riveting, welding and painting.  It is metal oxide but it's a pretty good rust barrier in the short term.  Painting over mill scale works well... For awhile... But when it fails, the mill scale turns against you; being moist and trapped under paint, it rusts progressively under the paint, making ever larger blisters with heavy rust under them. 

Knock it off prior to applying a good metal primer and paint.    Absolute ideal is grit (”sand") blast to SP10 near white metal, then a mil-spec 2-part epoxy strontium chromate primer and appropriate topcoat.  That can last 50 years, so it's what you use if you're NASA.   A weaker surface prep will significantly reduce performance.  A weaker primer will reduce performance somewhat.