# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# + [markdown] id="J6rnhr2Xs5Vs"
# <h1 style="text-align:center;font-size:30px;" > Quora Question Pairs </h1>

# + [markdown] id="o9fciGc7s5Vu"
# <h1> 1. Business Problem </h1>

# + [markdown] id="LRzmxjKxs5Vw"
# <h2> 1.1 Description </h2>

# + [markdown] id="1nlaIYe9s5Vx"
# <p>Quora is a place to gain and share knowledge—about anything. It’s a platform to ask questions and connect with people who contribute unique insights and quality answers. This empowers people to learn from each other and to better understand the world.</p>
# <p>
# Over 100 million people visit Quora every month, so it's no surprise that many people ask similarly worded questions. Multiple questions with the same intent can cause seekers to spend more time finding the best answer to their question, and make writers feel they need to answer multiple versions of the same question. Quora values canonical questions because they provide a better experience to active seekers and writers, and offer more value to both of these groups in the long term.
# </p>
# <br>
# > Credits: Kaggle 
#

# + [markdown] id="wdWP5SdFs5Vy"
# __ Problem Statement __
# - Identify which questions asked on Quora are duplicates of questions that have already been asked. 
# - This could be useful to instantly provide answers to questions that have already been answered. 
# - We are tasked with predicting whether a pair of questions are duplicates or not. 

# + [markdown] id="34hYn911s5V0"
# <h2> 1.2 Sources/Useful Links</h2>

# + [markdown] id="Hv6fd7txs5V7"
# 1. The cost of a mis-classification can be very high.
# 2. You would want a probability of a pair of questions to be duplicates so that you can choose any threshold of choice.
# 3. No strict latency concerns.
# 4. Interpretability is partially important.

# + [markdown] id="cW_MVIlps5WQ"
# <h1>3. Exploratory Data Analysis </h1>

# + id="sNzZdmBJs5WS" colab={"base_uri": "https://localhost:8080/", "height": 17} executionInfo={"status": "ok", "timestamp": 1640228227455, "user_tz": -330, "elapsed": 4014, "user": {"displayName": "<NAME>", "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64", "userId": "15716323643841165579"}} outputId="ca5220dd-b086-44db-c004-b7f8ee00bfa1"
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from subprocess import check_output
# %matplotlib inline
import plotly.offline as py
py.init_notebook_mode(connected=True)
import plotly.graph_objs as go
import plotly.tools as tls
import os
import gc

import re
from nltk.corpus import stopwords
#import distance
from nltk.stem import PorterStemmer
from bs4 import BeautifulSoup

# + [markdown] id="Ffhe1d2m2ckT"
# DataSet Download

# + [markdown] id="TlSZs3h8JSvC"
# https://www.kaggle.com/c/quora-question-pairs/data?select=train.csv.zip

# + id="F1br-oewc90o" colab={"base_uri": "https://localhost:8080/"} executionInfo={"status": "ok", "timestamp": 1640228240809, "user_tz": -330, "elapsed": 2678, "user": {"displayName": "<NAME>", "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64", "userId": "15716323643841165579"}} outputId="bd5ce309-e118-4163-9e3e-ab3ed80edbd0"
# !wget --header="Host: storage.googleapis.com" --header="User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" --header="Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9" --header="Accept-Language: en-US,en;q=0.9" --header="Referer: https://www.kaggle.com/" "https://storage.googleapis.com/kagglesdsdata/competitions/6277/323734/train.csv.zip?GoogleAccessId=<EMAIL>&Expires=1640402232&Signature=HKWlWk36tV2Rr7kcqNp9q9ajO4X8F6%2FDdwydqWmpqnrUoWIBkolkoUjXAAZEw2l3ngTweM9h%2BCDyMR5Ec%2FWEq%2BVG1EjTbHEhMOisIgph9%2F0NlCpv9%2B8f6qIaYERtII4JAY7Lpw0zFjyLIsPPuSDBRUSN4v413Z11EPQELNkCQEn6AiPwX2enPx44i%2FjW0iWrTVnixpk9V4PC4nfXU0%2BbVrm3D0%2BS%2Bf1%2B%2FpNu7eDDXRCUEI7UVlmTnv9%2FTHfZdUJumKvCoAEe%2Bud7MjPyZlbvnZv7ZJuEIOYGf5WFFNyOO0YwedahnJgAXRjY%2FA1wJaQItvJNXov0SoGv0V6TkemcIw%3D%3D&response-content-disposition=attachment%3B+filename%3Dtrain.csv.zip" -c -O 'train.csv.zip'
# !unzip train.csv

# + [markdown] id="__T8jddGs5Wc"
# <h2> 3.1 Reading data and basic stats </h2>

# + id="ifM_s9rvs5Wd" colab={"base_uri": "https://localhost:8080/"} executionInfo={"status": "ok", "timestamp": 1640228243186, "user_tz": -330, "elapsed": 2391, "user": {"displayName": "<NAME>", "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64", "userId": "15716323643841165579"}} outputId="793c149d-9165-42f8-bfa1-3d88c0ecaf10"
df = pd.read_csv("/content/train.csv")

print("Number of data points:",df.shape[0])

# + id="34zXGW8gs5Wj" colab={"base_uri": "https://localhost:8080/", "height": 206} executionInfo={"status": "ok", "timestamp": 1640228243192, "user_tz": -330, "elapsed": 26, "user": {"displayName": "<NAME>", "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64", "userId": "15716323643841165579"}} outputId="b838ecbc-f657-4eeb-b5d5-9aa1c38f6a3f"
df.head()

# + colab={"base_uri": "https://localhost:8080/"} id="5imGFPLZnAvh" executionInfo={"status": "ok", "timestamp": 1640228480870, "user_tz": -330, "elapsed": 92150, "user": {"displayName": "<NAME>", "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64", "userId": "15716323643841165579"}} outputId="a4fe2240-5d97-48e3-b1e9-45f014a89f5b"
from google.colab import drive
drive.mount('/content/drive')

# + id="mx4DFwMns5Wp" colab={"base_uri": "https://localhost:8080/"} executionInfo={"status": "ok", "timestamp": 1640228480874, "user_tz": -330, "elapsed": 53, "user": {"displayName": "<NAME>", "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64", "userId": "15716323643841165579"}} outputId="7c5ee626-120e-4c5a-cc18-ed29fc3a41d5"
df.info()

# + [markdown] id="HHHTGTzws5Ww"
# We are given a minimal number of data fields here, consisting of:
#
# - id:  Looks like a simple rowID
# - qid{1, 2}:  The unique ID of each question in the pair
# - question{1, 2}:  The actual textual contents of the questions.
# - is_duplicate:  The label that we are trying to predict - whether the two questions are duplicates of each other.

# + [markdown] id="ZulqVzTDs5Wx"
# <h3> 3.2.1 Distribution of data points among output classes</h3>
# - Number of duplicate(smilar) and non-duplicate(non similar) questions

# + id="YHp64yNjs5Wx" colab={"base_uri": "https://localhost:8080/", "height": 294} executionInfo={"status": "ok", "timestamp": 1640228480876, "user_tz": -330, "elapsed": 45, "user": {"displayName": "<NAME>", "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64", "userId": "15716323643841165579"}} outputId="4495e9bd-555a-4067-d3a1-9990f97cd383"
df.groupby("is_duplicate")['id'].count().plot.bar()

# + id="-usI2K2bs5W4" outputId="756b4734-4498-4f43-d7c0-6e049a7d630a" colab={"base_uri": "https://localhost:8080/"} executionInfo={"status": "ok", "timestamp": 1640228486684, "user_tz": -330, "elapsed": 359, "user": {"displayName": "<NAME>", "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64", "userId": "15716323643841165579"}}
print('~> Total number of question pairs for training:\n   {}'.format(len(df)))

# + id="YiPia6Pjs5W_" outputId="5c2d7dac-5f0e-4c45-daaa-10c6cb4dbb50" colab={"base_uri": "https://localhost:8080/"} executionInfo={"status": "ok", "timestamp": 1640228488664, "user_tz": -330, "elapsed": 7, "user": {"displayName": "<NAME>", "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64", "userId": "15716323643841165579"}}
print('~> Question pairs are not Similar (is_duplicate = 0):\n   {}%'.format(100 - round(df['is_duplicate'].mean()*100, 2)))
print('\n~> Question pairs are Similar (is_duplicate = 1):\n   {}%'.format(round(df['is_duplicate'].mean()*100, 2)))

# + [markdown] id="wGX03QVRs5XF"
# <h3> 3.2.2 Number of unique questions </h3>

# + id="VOKa6aU2s5XG" colab={"base_uri": "https://localhost:8080/"} executionInfo={"status": "ok", "timestamp": 1640228491640, "user_tz": -330, "elapsed": 1774, "user": {"displayName": "<NAME>", "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64", "userId": "15716323643841165579"}} outputId="cd140dfe-89b1-4b4b-9bf4-9e169c243d84"
qids = pd.Series(df['qid1'].tolist() + df['qid2'].tolist())
unique_qs = len(np.unique(qids))
qs_morethan_onetime = np.sum(qids.value_counts() > 1)
print ('Total number of  Unique Questions are: {}\n'.format(unique_qs))
print(len(np.unique(qids)))

print ('Number of unique questions that appear more than one time: {} ({}%)\n'.format(qs_morethan_onetime,qs_morethan_onetime/unique_qs*100))

print ('Max number of times a single question is repeated: {}\n'.format(max(qids.value_counts()))) 

q_vals=qids.value_counts()

q_vals=q_vals.values

# + id="plcvbd4Cs5XM" colab={"base_uri": "https://localhost:8080/", "height": 480} executionInfo={"status": "ok", "timestamp": 1640228505104, "user_tz": -330, "elapsed": 903, "user": {"displayName": "<NAME>", "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64", "userId": "15716323643841165579"}} outputId="ce2504f4-cfea-46f7-9955-28d847284071"
x = ["unique_questions" , "Repeated Questions"]
y =  [unique_qs , qs_morethan_onetime]

plt.figure(figsize=(10, 6))
plt.title ("Plot representing unique and repeated questions  ")
sns.barplot(x,y)
plt.show()

# + [markdown] id="G-CwGaMms5XS"
# <h3>3.2.3 Checking for Duplicates </h3>

# + id="YCiDBHm5s5XT" outputId="20097698-af98-4efb-a5b2-e0302e42ccfe" colab={"base_uri": "https://localhost:8080/"} executionInfo={"status": "ok", "timestamp": 1640228508950, "user_tz": -330, "elapsed": 361, "user": {"displayName": "<NAME>", "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64", "userId": "15716323643841165579"}}
#checking whether there are any repeated pair of questions

pair_duplicates = df[['qid1','qid2','is_duplicate']].groupby(['qid1','qid2']).count().reset_index()

print ("Number of duplicate questions",(pair_duplicates).shape[0] - df.shape[0])

# + [markdown] id="iaHTnnt8s5XX"
# <h3> 3.2.4 Number of occurrences of each question </h3>

# + id="dPZwk-C8s5Xa" outputId="c6522fee-609d-4294-9031-effcfc130928" colab={"base_uri": "https://localhost:8080/", "height": 656} executionInfo={"status": "ok", "timestamp": 1640228515327, "user_tz": -330, "elapsed": 3576, "user": {"displayName": "<NAME>", "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64", "userId": "15716323643841165579"}}
plt.figure(figsize=(20, 10))

plt.hist(qids.value_counts(), bins=160)

plt.yscale('log', nonposy='clip')

plt.title('Log-Histogram of question appearance counts')

plt.xlabel('Number of occurences of question')

plt.ylabel('Number of questions')

print ('Maximum number of times a single question is repeated: {}\n'.format(max(qids.value_counts()))) 

# + [markdown] id="h_WdYxlYs5Xj"
# <h3> 3.2.5 Checking for NULL values </h3>

# + id="r0x1gR2fs5Xk" colab={"base_uri": "https://localhost:8080/"} executionInfo={"status": "ok", "timestamp": 1640228524588, "user_tz": -330, "elapsed": 626, "user": {"displayName": "<NAME>", "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64", "userId": "15716323643841165579"}} outputId="513fb927-ebe5-4688-ec0e-836703201747"
#Checking whether there are any rows with null values
nan_rows = df[df.isnull().any(1)]
print (nan_rows)

# + [markdown] id="CCYmufv6s5Xo"
# - There are two rows with null values in question2 

# + id="yLBRyACgs5Xp" outputId="c4add345-336a-4959-fc3d-5341d129256b" colab={"base_uri": "https://localhost:8080/"} executionInfo={"status": "ok", "timestamp": 1640228529217, "user_tz": -330, "elapsed": 376, "user": {"displayName": "<NAME>", "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64", "userId": "15716323643841165579"}}
# Filling the null values with ' '
df = df.fillna('')
nan_rows = df[df.isnull().any(1)]
print (nan_rows)

# + [markdown] id="l9Qcl5xfs5Xs"
# <h2>3.3 Basic Feature Extraction (before cleaning) </h2>

# + [markdown] id="RRzvPYzGs5Xu"
# Constructing a few features like:
#  - ____freq_qid1____ = Frequency of qid1's
#  - ____freq_qid2____ = Frequency of qid2's 
#  - ____q1len____ = Length of q1
#  - ____q2len____ = Length of q2
#  - ____q1_n_words____ = Number of words in Question 1
#  - ____q2_n_words____ = Number of words in Question 2
#  - ____word_Common____ = (Number of common unique words in Question 1 and Question 2)
#  - ____word_Total____ =(Total num of words in Question 1 + Total num of words in Question 2)
#  - ____word_share____ = (word_common)/(word_Total)
#  - ____freq_q1+freq_q2____ = sum total of frequency of qid1 and qid2 
#  - ____freq_q1-freq_q2____ = absolute difference of frequency of qid1 and qid2 

# + colab={"base_uri": "https://localhost:8080/"} id="xAMHtwQpsHHE" executionInfo={"status": "ok", "timestamp": 1640228537572, "user_tz": -330, "elapsed": 379, "user": {"displayName": "<NAME>", "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64", "userId": "15716323643841165579"}} outputId="1cc3ffc1-2b85-45d3-a6c5-4cc4568adb4f"
 df['question1']

# + id="Iq4DZ-rYs5Xv" outputId="ee74d7a2-da6f-4546-fb1f-16ac542a4fbf" colab={"base_uri": "https://localhost:8080/", "height": 704} executionInfo={"status": "ok", "timestamp": 1640228843889, "user_tz": -330, "elapsed": 37387, "user": {"displayName": "<NAME>", "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64", "userId": "15716323643841165579"}}
df['freq_qid1'] = df.groupby('qid1')['qid1'].transform('count') 
df['freq_qid2'] = df.groupby('qid2')['qid2'].transform('count')
df['q1len'] = df['question1'].str.len() 
df['q2len'] = df['question2'].str.len()
df['q1_n_words'] = df['question1'].apply(lambda row: len(str(row).split(" ")))
df['q2_n_words'] = df['question2'].apply(lambda row: len(str(row).split(" ")))

def normalized_word_Common(row):
    w1 = set(map(lambda word: word.lower().strip(), row['question1'].split(" ")))
    w2 = set(map(lambda word: word.lower().strip(), row['question2'].split(" ")))    
    return 1.0 * len(w1 & w2)
df['word_Common'] = df.apply(normalized_word_Common, axis=1)

def normalized_word_Total(row):
    w1 = set(map(lambda word: word.lower().strip(), row['question1'].split(" ")))
    w2 = set(map(lambda word: word.lower().strip(), row['question2'].split(" ")))    
    return 1.0 * (len(w1) + len(w2))
df['word_Total'] = df.apply(normalized_word_Total, axis=1)

def normalized_word_share(row):
    w1 = set(map(lambda word: word.lower().strip(), row['question1'].split(" ")))
    w2 = set(map(lambda word: word.lower().strip(), row['question2'].split(" ")))    
    return 1.0 * len(w1 & w2)/(len(w1) + len(w2))
df['word_share'] = df.apply(normalized_word_share, axis=1)

df['freq_q1+q2'] = df['freq_qid1']+df['freq_qid2']
df['freq_q1-q2'] = abs(df['freq_qid1']-df['freq_qid2'])

df.to_csv("/content/drive/MyDrive/case studies/Docs_QUORA/df_fe_without_preprocessing_train.csv", index=False)

df.head()

# + [markdown] id="-zLujovVs5X3"
# <h3> 3.3.1 Analysis of some of the extracted features </h3>

# + [markdown] id="zRIFQTkCs5X3"
# - Here are some questions have only one single words.

# + id="jSS0X82Ds5X5" outputId="1916d296-f2a5-4333-aac8-0ff9ee10f196" colab={"base_uri": "https://localhost:8080/"} executionInfo={"status": "ok", "timestamp": 1640228586037, "user_tz": -330, "elapsed": 391, "user": {"displayName": "<NAME>", "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64", "userId": "15716323643841165579"}}
print ("Minimum length of the questions in question1 : " , min(df['q1_n_words']))

print ("Minimum length of the questions in question2 : " , min(df['q2_n_words']))

print ("Number of Questions with minimum length [question1] :", df[df['q1_n_words']== 1].shape[0])
print ("Number of Questions with minimum length [question2] :", df[df['q2_n_words']== 1].shape[0])

# + [markdown] id="kFzTIHW3s5YB"
# <h4> 3.3.1.1 Feature: word_share </h4>

# + id="s4rwGLFDs5YD" outputId="40bffb42-8a04-48f3-86ec-4d6968852066" colab={"base_uri": "https://localhost:8080/", "height": 657} executionInfo={"status": "ok", "timestamp": 1640228597760, "user_tz": -330, "elapsed": 5370, "user": {"displayName": "<NAME>", "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64", "userId": "15716323643841165579"}}
plt.figure(figsize=(12, 8))

plt.subplot(1,2,1)
sns.violinplot(x = 'is_duplicate', y = 'word_share', data = df[0:])

plt.subplot(1,2,2)
sns.distplot(df[df['is_duplicate'] == 1.0]['word_share'][0:] , label = "1", color = 'red')
sns.distplot(df[df['is_duplicate'] == 0.0]['word_share'][0:] , label = "0" , color = 'blue' )
plt.show()

# + [markdown] id="RcwMI4xps5YJ"
# - The distributions for normalized word_share have some overlap on the far right-hand side, i.e., there are quite a lot of questions with high word similarity
# - The average word share and Common no. of words of qid1 and qid2 is more when they are duplicate(Similar)

# + [markdown] id="K0AbOS65s5YL"
# <h4> 3.3.1.2 Feature: word_Common </h4>

# + id="_mCFvztcs5YM" outputId="efec6061-e542-4392-c853-f8ee72af7873" colab={"base_uri": "https://localhost:8080/", "height": 657} executionInfo={"status": "ok", "timestamp": 1640228607466, "user_tz": -330, "elapsed": 5342, "user": {"displayName": "<NAME>", "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64", "userId": "15716323643841165579"}}
plt.figure(figsize=(12, 8))

plt.subplot(1,2,1)
sns.violinplot(x = 'is_duplicate', y = 'word_Common', data = df[0:])

plt.subplot(1,2,2)
sns.distplot(df[df['is_duplicate'] == 1.0]['word_Common'][0:] , label = "1", color = 'red')
sns.distplot(df[df['is_duplicate'] == 0.0]['word_Common'][0:] , label = "0" , color = 'blue' )
plt.show()

# + [markdown] id="9Ej1ouEVs5YR"
# <p> The distributions of the word_Common feature in similar and non-similar questions are highly overlapping </p>

StarcoderdataJupyterScriptsDedupFiltered