def _correlate(series: pd.Series, correlation_value: int, seed: int = 0):
  """Generates a correlated random variables from a given series."""
  # https://stats.stackexchange.com/questions/38856/how-to-generate-correlated-random-numbers-given-means-variances-and-degree-of
  np.random.seed(seed)
  value_error_term = 1 - correlation_value**2
  error_terms = np.random.normal(0, value_error_term**0.5, len(series))
  return series * correlation_value + error_terms


np.random.seed(18)
data = pd.DataFrame(np.random.normal(0, 1, (10000, 6)))
data[0] = (data[0] >= 0.0).astype(int)

data['constant'] = 1
data['var1'] = data[0]
data['var2'] = data[1]
data['var3'] = data[2]
data['collinear_var2a'] = data['var2']
data['collinear_var2b'] = _correlate(data['var2'], correlation_value=0.99)
data['random1'] = data[3]
data['random2'] = data[4]
data['random3'] = data[5]
data['target'] = (
    data['var1'] * 0.1 +
    data['var2'] * 5.0 +
    data['var3'] * -0.5 +
    (np.random.rand(len(data))-0.5) # Adding Noise
    )

"""
Explanation: Simulate some data
Let's create some random simulated data, where the target variables will be
y = var1 * 0.1 + var2 * 5.0 + var3 * -0.5 + e
To complicate the problems, we're going to add adding the following type of variables:

constant: Variable with no variation that carries no information.
collinear: Collinear variables which are highly correlated. In our example we'll have an highly correlated variable and an identical copy to var2.
random: Random variables with no relationship with the target variable.
End of explanation
"""


_ = data.plot.scatter('var2', 'collinear_var2b')
_ = data.plot.scatter('var2', 'collinear_var2a')

"""
Explanation: These are the collinear variables introduced and their relationship with var2.
This show that var2 and collinear_var2a are identical, while var2 and collinear_var2b are highly correlated.
End of explanation
"""


inference_data = data_preparation.InferenceData(
    initial_data=data[[
        'constant',
        'collinear_var2a', 'collinear_var2b',
        'var1', 'var2', 'var3',
        'random1', 'random2', 'random3',
        'target'
        ]],
    target_column='target')
inference_data.data

"""
Explanation: Modelling
Preparing our InferenceData container.
End of explanation
"""


naive_model = models.InferenceRidge(alpha=100)
naive_model.fit(inference_data, raise_on_data_error=False)
naive_model.get_results()[['effect']]

"""
Explanation: Fitting a model with no data preparation.
We can fit a model directly, setting the parameter raise_on_data_error=False to avoid InferenceDataError exception to be risen. This will skip any data check and you'll be notified with InferenceDataWarning warnings that some additionals step may be required before proceed with modelling.
Results
These shows that the collinear variables are confounding when trying to recover the
real effect. Also notice that a set of Warnings are triggered to inform you that the data was not prepared for modelling.
End of explanation
"""


inference_data.address_low_variance(threshold=0, drop=True)

"""
Explanation: Reminding that our equaltion for y is
y = var1 * 0.1 + var2 * 5.0 + var3 * -0.5 + e
When looking at the recovered effect from our model assigned to variables var2, collinear_var2a and collinear_var2b aren't quiet right - this because they are multi-collinear. The expected 5.0 effect has been spread across all of them with different proportions.
Note that a strong regularization has been used in the Ridge regression (alpha=1000) to emphatize the collinarity issue.
Preparing the data correctly before fitting a model.
Before modelling, we want to:

Remove low-variance variables, in our case the constant variable.
Remove collinear variables, in our case removing two the var2.

Note that we should also control for external factors, which are not present in this example.
Data preparations steps in a flow-chart.
The follow flow-chart summarizes the data preparation steps to make sure to minimize issues while attempting to recover the real effect.

Addressing low variance variables
The method address_low_variance(threshold=0, drop=True) in InferenceData identifies low variances variables and have an option to drop them. When variance threshold is zero, means the variable has no variation and is a constant value for all the observations.
After running the method, the constant column in the InferenceData has been remove from the dataframe.
End of explanation
"""


inference_data.address_collinearity_with_vif(vif_method='sequential',
                                             vif_threshold=10,
                                             drop=True)

"""
Explanation: Addressing Collinearity with Variance Inflation Factor (VIF)
In the example above, collinear columns are interfering with recovering the real effect. The method address_collinearity_with_vif(address_collinearity_with_vif(vif_method='sequential', vif_threshold=10, drop=True) uses VIF. to identify columns that are collinear and option to drop them and retain only one.
End of explanation
"""


less_naive_model = models.InferenceRidge(alpha=100)
less_naive_model.fit(inference_data, raise_on_data_error=False)
less_naive_model.get_results()[['effect']]

"""
Explanation: Collinearity is always tricky to address, the options usually are:
* Iteratively eliminate collinear variables one at time until collinearity is removed
* Create casual graphs to map dependencies between variables and include one per collinear group
* Remove all collinear variables
In our example var2, collinear_var2a and collinear_var2b are all collinear with each others. We've decided to went for option one and because the elimination process in VIF is random, collinear_var2b was selected. Unfortunately, there is no way know that var2 should be retain instead of collinear_var2b with this approach.
Results
With data better suited for recovering the real effects, these are more much closer to the real one. Fitting bootstrap and permutation test will also give you tools to reject or accept theses.
End of explanation
"""


less_naive_model.fit_bootstrap(50, n_jobs=1, verbose=False)
less_naive_model.get_results()

less_naive_model.permutation_test(50, n_jobs=1, verbose=False)
less_naive_model.get_results()

"""
Explanation: Reminding that our equaltion for y is
y = var1 * 0.1 + var2 * 5.0 + var3 * -0.5 + e
When looking at the recovered effect collinear_var2b, this is much closer to what it's supposed to be.
End of explanation
"""


