What causes the Shape (convex or non-convex) of the cost function?
The cost function is convex if its Second Order Derivative is positive semidefinite (i.e. $\geq0$ ). 

But this definition depends on the function with respect to which you take the derivative. This convexity changes when we are talking about Neural Networks, as in that case, our derivatives are taken with respect to the weights.

Refer: Why Cross Entropy is convex, but not necessarily in neural nets and Why RMSE is convex but not necessarily in neural nets