I am currently working with regression analysis in combination with language analysis on online reviews. I would like to give you a short description of my task and what I want to achieve with it, in order to maybe find a proper solution.

I wanted to find possible correlations with the helpfulness score given by the users. For this I used scaled\_HS as my dependent variable `y`. For my variable `x` I used 63 language features from the `LIWC manual`. The helpfulness score got scaled by the maximum number of views, to give ratings such as 500 thumbs up and 1000 views a bigger weight than 2 thumbs up and 4 views. Below is a small description of my dataframe:

```
Rating  Spoiler     HS  WC  Tone    WPS     Dic     function.   ppron   i   ...     swear   netspeak    assent  nonflu  filler  QMark   Exclam  keywords    mistakes    scaled_HS
0   1   0   0.37    150     25.77   16.67   90.67   46.67   5.33    1.33    ...     0.00    0.00    0.00    0.0     0.0     0.00    4.00    0.000000    0.186667    0.017273
1   1   1   0.16    131     1.55    26.20   89.31   58.02   6.87    0.76    ...     0.00    0.00    0.76    0.0     0.0     3.05    0.00    0.000000    0.137405    0.026339
2   1   0   0.24    133     39.19   33.25   92.48   50.38   3.01    2.26    ...     0.00    0.00    0.00    0.0     0.0     0.00    0.00    0.000000    0.097744    0.044079 

```

After applying simple linear regression with the `Pearsonr` function I found out that some features were significant but had very low coefficients, except `HS` but I forgot to delete the column, because I have the `scaled HS` anyways:

```
       Coefficients            p-value
Rating (-0.06201207484873128, 0.015435587631467379)
Spoiler (0.059553066433509855, 0.020030343178780382)
HS (0.4200389620235087, 3.095697207704496e-66)
Review_length (0.0611843321063948, 0.016866533916445086)
WC (0.0629356733799715, 0.013965934404277043)
Tone (-0.07394090951501098, 0.0038637998296509337)
shehe (0.07131791547159314, 0.005331048300074106)
conj (-0.05033621584191379, 0.04937650321109507)
compare (-0.06621954834327438, 0.009690693447463867)
number (0.05913351896038223, 0.02092332389500587)
quant (-0.06552319095849933, 0.010484851019228214)
posemo (-0.05931337090970264, 0.020536361646821617)
negemo (0.05131647584743782, 0.04510762970526194)
male (0.06826773827085715, 0.007656218956918288)
death (0.05074141927289752, 0.047572524702827156)
QMark (0.053348318103051344, 0.03724269305864121)

```

After checking the `mean` and `std` of my dependent variable y I found out that my data is dispersed:

```
    scaled_HS
mean 0.01242697466467958
std  0.02743090432486186

```

So I thought that a `negative binomial regression` would fit much better than a linear regression model. After using the `statsmodel` package I got the following table:

```
                 Generalized Linear Model Regression Results                  
==============================================================================
Dep. Variable:              scaled_HS   No. Observations:                 1525
Model:                            GLM   Df Residuals:                     1461
Model Family:        NegativeBinomial   Df Model:                           63
Link Function:                    log   Scale:                          1.0000
Method:                          IRLS   Log-Likelihood:                -93.260
Date:                Tue, 07 Feb 2023   Deviance:                       21.239
Time:                        07:03:00   Pearson chi2:                     33.7
No. Iterations:                     6                                         
Covariance Type:            nonrobust                                         
===============================================================================
                  coef    std err          z      P>|z|      [0.025      0.975]
-------------------------------------------------------------------------------
Rating         -0.0589      0.109     -0.540      0.589      -0.273       0.155
Spoiler        -0.0572      0.593     -0.096      0.923      -1.220       1.106
HS              4.5688      1.316      3.471      0.001       1.989       7.149
WC           5.122e-05      0.001      0.036      0.972      -0.003       0.003
Tone           -0.0018      0.016     -0.114      0.909      -0.033       0.029
WPS            -0.0028      0.010     -0.272      0.785      -0.023       0.017
Dic            -0.0205      0.075     -0.272      0.786      -0.169       0.127
function.      -0.0671      0.101     -0.661      0.509      -0.266       0.132
ppron          -0.0324      0.388     -0.084      0.933      -0.793       0.728
i               0.0527      0.393      0.134      0.893      -0.718       0.823
you             0.1103      0.466      0.237      0.813      -0.802       1.023
shehe           0.1647      0.491      0.335      0.737      -0.798       1.127
they            0.0642      0.457      0.141      0.888      -0.831       0.960
ipron           0.0665      0.123      0.540      0.590      -0.175       0.308
adverb          0.0802      0.124      0.647      0.518      -0.163       0.323
conj           -0.0281      0.118     -0.238      0.812      -0.259       0.203
negate          0.0495      0.238      0.209      0.835      -0.416       0.515
adj             0.0353      0.150      0.235      0.814      -0.258       0.329
compare        -0.0563      0.189     -0.298      0.766      -0.426       0.314
interrog       -0.0231      0.253     -0.091      0.927      -0.518       0.472
number          0.0035      0.161      0.022      0.983      -0.312       0.319
quant          -0.1101      0.176     -0.626      0.531      -0.455       0.235
affect         -0.0286      1.014     -0.028      0.978      -2.015       1.958
posemo         -0.0059      1.008     -0.006      0.995      -1.981       1.970
negemo          0.0553      1.046      0.053      0.958      -1.996       2.106
anx            -0.0961      0.517     -0.186      0.853      -1.109       0.917
anger          -0.0627      0.397     -0.158      0.875      -0.841       0.716
sad            -0.2089      0.463     -0.451      0.652      -1.116       0.698
family          0.0265      0.468      0.057      0.955      -0.891       0.944
friend         -0.3726      0.875     -0.426      0.670      -2.087       1.341
female         -0.0805      0.347     -0.232      0.817      -0.761       0.600
male           -0.0277      0.313     -0.088      0.930      -0.642       0.587
insight        -0.0469      0.207     -0.226      0.821      -0.453       0.359
cause          -0.1160      0.221     -0.525      0.600      -0.549       0.317
discrep        -0.0251      0.241     -0.104      0.917      -0.497       0.447
tentat         -0.0234      0.179     -0.131      0.896      -0.375       0.328
certain        -0.0242      0.205     -0.118      0.906      -0.425       0.377
differ         -0.0384      0.166     -0.231      0.817      -0.364       0.287
see            -0.0170      0.188     -0.090      0.928      -0.386       0.352
feel           -0.0078      0.325     -0.024      0.981      -0.645       0.630
bio            -0.0860      0.293     -0.294      0.769      -0.660       0.488
sexual          0.1453      0.695      0.209      0.834      -1.217       1.507
affiliation     0.0709      0.265      0.267      0.789      -0.448       0.590
achieve         0.0106      0.205      0.052      0.959      -0.390       0.412
power          -0.0958      0.175     -0.547      0.584      -0.439       0.247
reward          0.0172      0.176      0.098      0.922      -0.328       0.363
risk            0.0496      0.412      0.120      0.904      -0.757       0.857
focuspast       0.0047      0.117      0.040      0.968      -0.224       0.234
focusfuture     0.0510      0.201      0.254      0.800      -0.343       0.445
time           -0.0120      0.107     -0.112      0.911      -0.221       0.197
home            0.0394      0.399      0.099      0.921      -0.742       0.820
money          -0.0196      0.541     -0.036      0.971      -1.080       1.041
relig           0.0978      0.749      0.131      0.896      -1.369       1.565
death           0.0169      0.480      0.035      0.972      -0.924       0.958
informal        0.0847      1.507      0.056      0.955      -2.869       3.038
swear          -0.1255      1.613     -0.078      0.938      -3.288       3.037
netspeak       -0.0516      1.524     -0.034      0.973      -3.038       2.935
assent          0.0018      1.562      0.001      0.999      -3.060       3.064
nonflu         -0.1514      1.590     -0.095      0.924      -3.268       2.965
filler         -0.2321      1.896     -0.122      0.903      -3.948       3.484
QMark           0.0523      0.256      0.204      0.838      -0.449       0.554
Exclam          0.0518      0.149      0.349      0.727      -0.240       0.343
keywords       -2.5026     26.795     -0.093      0.926     -55.019      50.014
mistakes       -4.1954      7.054     -0.595      0.552     -18.020       9.629

```

My question would be: Why is this regression model showing poorer findings than the linear regression model? My y variable is in range of `0` and `1` and the `std` > `mean`. Thank you for your help!