# Phi_K Correlation Analyzer Library¶

Phi_K is a new and practical correlation coefficient based on several refinements to Pearson’s hypothesis test of independence of two variables.

The combined features of Phi_K form an advantage over existing coefficients. First, it works consistently between categorical, ordinal and interval variables. Second, it captures non-linear dependency. Third, it reverts to the Pearson correlation coefficient in case of a bi-variate normal input distribution. These are useful features when studying the correlation matrix of variables with mixed types.

The presented algorithms are easy to use and available through this public Python library: the correlation analyzer package. Emphasis is paid to the proper evaluation of statistical significance of correlations and to the interpretation of variable relationships in a contingency table, in particular in case of low statistics samples.

For example, the Phi_K correlation analyzer package has been used to study surveys, insurance claims, correlograms, etc. For details on the methodology behind the calculations, please see our publication.

## Documentation¶

The entire Phi_K documentation including tutorials can be found at read-the-docs.

## Check it out¶

The Phi_K library requires Python 3 and is pip friendly. To get started, simply do:

$pip install phik  or check out the code from out GitHub repository: $ git clone https://github.com/KaveIO/PhiK.git
\$ pip install -e PhiK/


where in this example the code is installed in edit mode (option -e).

You can now use the package in Python with:

import phik


Congratulations, you are now ready to use the PhiK correlation analyzer library!

## Quick run¶

As a quick example, you can do:

import pandas as pd
import phik
from phik import resources, report

# open fake car insurance data

# Pearson's correlation matrix between numeric variables (pandas functionality)
df.corr()

# get the phi_k correlation matrix between all variables
df.phik_matrix()

# get the significance matrix (expressed as one-sided Z)
# of the hypothesis test of each variable-pair dependency
df.significance_matrix()

# contingency table of two columns
cols = ['mileage','car_size']
df[cols].hist2d()

# normalized residuals of contingency test applied to cols
df[cols].outlier_significance_matrix()

# show the normalized residuals of each variable-pair
df.outlier_significance_matrices()

# generate a phik correlation report and save as test.pdf
report.correlation_report(df, pdf_file_name='test.pdf')
{"ft_lang_label":"__label__en","ft_lang_prob":0.7744734,"math_prob":0.671825,"size":3019,"snap":"2019-51-2020-05","text_gpt3_token_len":665,"char_repetition_ratio":0.114759535,"word_repetition_ratio":0.0,"special_character_ratio":0.2080159,"punctuation_ratio":0.14503817,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.9916229,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2019-12-12T16:26:14Z\",\"WARC-Record-ID\":\"<urn:uuid:e49fbd79-1027-43d2-88c7-2bd36a44fa40>\",\"Content-Length\":\"16830\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:4bc02086-08b6-402d-b580-40d6fd62a7b5>\",\"WARC-Concurrent-To\":\"<urn:uuid:5f905828-d0df-40b0-982a-5c0344f4a86c>\",\"WARC-IP-Address\":\"104.208.221.96\",\"WARC-Target-URI\":\"https://phik.readthedocs.io/en/latest/?badge=latest\",\"WARC-Payload-Digest\":\"sha1:WJ5ZGNSQD3SETTFYGZ2ILAC7SLXIKWFB\",\"WARC-Block-Digest\":\"sha1:Y74WY4WW2BECQKVTN4MB4PGBVRMS5WBO\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2019/CC-MAIN-2019-51/CC-MAIN-2019-51_segments_1575540544696.93_warc_CC-MAIN-20191212153724-20191212181724-00081.warc.gz\"}"}