Cost Function on non linear noisy data I was curious to see if one can use a cost function on a set of data points to find the "optimial minimum" solution for any given set of data. 

I know for a regular set of data that is clustered symmetrically following a straight regression line is easy to find the proper cost function 

Example:



but what if the data makes a funky shape like that of x^2 or 3*x^3 or etc...

As you can see that the the second graph is a lot more noisy than the first. 



Then what do you do? is it the same process or is it different?
Can you find an optimal Minimum for any set of data points? 

I know the cost function is using the residuals to form its best fit but I was just curious if it can do with any set of data.
I don't think the second picture necessarily has more noise, it's just less linear than the first one. If you are doing regression (which is the case in both your problems) you can use the same loss function to optimize your family of functions that you are using to fit your data. The only difference is that with certain function families gradient descent might not get to a global optimum but will get stuck in a local one. The loss function only determines how to penalize certain errors (based on the residuals).