Are we naturally born right or left handed or ambidextrous?
tl;dr

Yes!

Is handedness determined before birth?

Without thinking of whether handedness is genetically or environmentally defined (see below), is handedness decided before birth?

Hepper et al. 1991 and DeVries et al. 2001 show that handedness can be inferred from from fetal position and hand usage. Typically, looking at what thumb is used for sucking is good indication of handedness.

Is handedness heritable?

The question 


  [A]re we naturally born right or left handed?


can sometimes be understood as 


  Is handedness genetically determined?


or in more concrete terms as


  Is handedness heritable in humans?


If it is not obvious to you that this is your question, then you probably don't understand the concept of heritability! Then you should have a look at Why is a heritability coefficient not an index of how “genetic” something is? first and then come back here to read the results.

There have been a large number of studies on the question, unfortunately most of them were very small studies with small sample size. Medland et al. 2008 is pretty large scale twin study which report a heritability of 0.24.

In other words, yes handedness is heritable. To be more accurate, 24% of the total variance in handedness is explain by additive genetic variance. Yet again in other words.

The cooperation-competition hypothesis

On the evolution of the handedness in he human population you might want to hear about the cooperation-competition hypothesis. For that, you should have a look at this TED-ed video. As you watch this video, you'll probably be interested in reading the wikipedia article on frequency-dependent selection.

Thanks to @Mockingbird for pointing out to this video.