Any efficient way to build non-linear regression model for polynomial features? I am trying to understand how crime frequency affect house price in certain area. To do so, I started with Chicago crime data and zillow real estate data. I want to understand the relation between house price and crime frequency and top 5 crimes in certain areas. Initially, I build up model for this specification, but it wasn't very meaningful to me. Can anyone enlighten me what should I do? any efficient approach to train regression model for potential relation between house price and crime frequency in certain areas? any heuristic idea to move forward?

example data snippet:

here is the merged data that includes annual house price and top crime type in certain areas:



Here is reproducible example data snippet

my attempt

so here is my attempt to fit regression model with above reproducible example data:

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler
import pandas as pd

regDF = pd.read_csv('exampleDF')

X_feats = regDF.drop(['Avg_Price_2012'], axis=1)
y_label = regDF['Avg_Price_2012'].values

sc_x = StandardScaler()
sc_y = StandardScaler()
X = sc_x.fit_transform(X_feats)
#y= sc_y.fit_transform(y_label)
y = sc_y.fit_transform(y_label .reshape(-1,1)).flatten()
regModel = LinearRegression()
regModel.fit(X, y)
regModel.coef_


but to me, above model wasn't that efficient and needs to be done something more. I think I have to use non linear regression model for those polynomial features, and I am not sure to get this done.

Can anyone point me out how to build correct model for house price prediction over type of crimes and frequencies in certain areas? any idea? Thanks

Goal:

I want to build regression model to predict house price based on crime frequencies and types in certain areas. How can I get modeling the relationship between house price and crimes in certain areas? any thoughts?
You are probably finding yourself in one of the most interesting problems in data science, the part which is more art than science.

I will give you some ideas that can give you hints on how to solve this problem:


Prices, Salaries and other variables who have information on "accumulables" many times have a distribution which is skewed to the left (many individuals have a little, a few have a lot), what is reccomended to do is to take logarithm of it. Your new variable should be $Ln(Y)$, with this, you will close the gap between areas with greater avg_price and areas with lower avg_price. When that happens you find a less-skewed-normal-like distribution of your $Y$ variable.
The idea of taking Logarithm also applies to the $X$ variables you have (because crimes also accumulate in certain areas).
The Standard Scalling is not necessary when you run a linear regression, because the relativeness of the variable is not influencial in the regression:


The regression $Y = \alpha_0 + \alpha_1X_1+...+\alpha_nX_n$ (no
    scalled) is mathematically equivalent to $Y = \beta_0 +
\beta_1Z_1+...+\beta_nZ_n$ (scalled)


If you want to use other models, you data seems suitable for it, maybe a regression tree or XGBoost might work well on your problem.


I would bet that getting Logarithm in the avg_price, in some exogenous variables and not scalling would get better results for you.

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler
import pandas as pd

regDF = pd.read_csv('exampleDF')

X_feats = regDF.drop(['Avg_Price_2012'], axis=1)
y_label = regDF['Avg_Price_2012'].values

X = log(X_feats)
y = log(y_label.reshape(-1,1)).flatten()
regModel = LinearRegression()
regModel.fit(X, y)
regModel.coef_