diff --git "a/ai-medical-chatbot-master/3-Modeling/3_2-Clustering.ipynb" "b/ai-medical-chatbot-master/3-Modeling/3_2-Clustering.ipynb" new file mode 100644--- /dev/null +++ "b/ai-medical-chatbot-master/3-Modeling/3_2-Clustering.ipynb" @@ -0,0 +1,1248 @@ +{ + "cells": [ + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Clustering\n", + "\n", + "We use a simple k-means algorithm to demonstrate how clustering can be done. Clustering can help discover valuable, hidden groupings within the data. The dataset is created in the [Obtain_dataset Notebook](Obtain_dataset.ipynb)." + ] + }, + { + "cell_type": "code", + "execution_count": 433, + "metadata": {}, + "outputs": [], + "source": [ + "# imports\n", + "import numpy as np\n", + "import pandas as pd\n", + "from ast import literal_eval\n", + "# load data\n", + "datafile_path = \"../2-Data/dialogues_embededd.pkl\"\n", + "df = pd.read_pickle(datafile_path)" + ] + }, + { + "cell_type": "code", + "execution_count": 434, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + " | Description | \n", + "Patient | \n", + "Doctor | \n", + "combined | \n", + "n_tokens | \n", + "embedding | \n", + "
---|---|---|---|---|---|---|
255916 | \n", + "What medicine is best for taking out pores and... | \n", + "Hi, I am (Simran) 28 years (Female) hight 5/3 ... | \n", + "Hi simran, your problem can be best sort out u... | \n", + "Description: What medicine is best for taking ... | \n", + "282 | \n", + "[-0.2568769, 0.14872037, 0.17243277, 0.2426784... | \n", + "
255917 | \n", + "What causes non conception despite unprotected... | \n", + "Hi, i am living with my partner from last six ... | \n", + "do serum tsh and serum prolactin den do hsg on... | \n", + "Description: What causes non conception despit... | \n", + "195 | \n", + "[-0.058888793, 0.084498726, -0.05540138, 0.021... | \n", + "
255918 | \n", + "How long after getting chicken pox is it safe ... | \n", + "i am 25 yrs old. last year in June i had an IU... | \n", + "Hi, I think you should keep a gap of 3 months ... | \n", + "Description: How long after getting chicken po... | \n", + "254 | \n", + "[-0.12680525, -0.01062898, -0.25460148, 0.0511... | \n", + "
255919 | \n", + "How to remove unwanted hair without any side e... | \n", + "hi doctor! i m 22 years old and i want to remo... | \n", + "Hello, which part of the body do you have thes... | \n", + "Description: How to remove unwanted hair witho... | \n", + "110 | \n", + "[0.12641467, -0.1543769, 0.42004266, 0.1722529... | \n", + "
255920 | \n", + "I am 15. Can i shave my pubic hair? | \n", + "sorry,im just curious because im 15 and when i... | \n", + "haha aw. Yes your supposed to shave, trim it f... | \n", + "Description: I am 15. Can i shave my pubic hai... | \n", + "223 | \n", + "[-0.06219541, 0.020706447, 0.38799724, -0.0219... | \n", + "
... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "
256911 | \n", + "Why is hair fall increasing while using Bontre... | \n", + "I am suffering from excessive hairfall. My doc... | \n", + "Hello Dear Thanks for writing to us, we are he... | \n", + "Description: Why is hair fall increasing while... | \n", + "211 | \n", + "[-0.17113408, 0.10835318, 0.33148944, -0.06146... | \n", + "
256912 | \n", + "Why was I asked to discontinue Androanagen whi... | \n", + "Hi Doctor, I have been having severe hair fall... | \n", + "hello, hair4u is combination of minoxid... | \n", + "Description: Why was I asked to discontinue An... | \n", + "154 | \n", + "[-0.24637492, 0.031407423, -0.05137701, -0.301... | \n", + "
256913 | \n", + "Can Mintop 5% Lotion be used by women for seve... | \n", + "Hi..i hav sever hair loss problem so consulted... | \n", + "HI I have evaluated your query thoroughly you... | \n", + "Description: Can Mintop 5% Lotion be used by w... | \n", + "191 | \n", + "[-0.32340947, 0.3667281, 0.3651925, -0.0989788... | \n", + "
256914 | \n", + "Is Minoxin 5% lotion advisable instead of Foli... | \n", + "Hi, i am 25 year old girl, i am having massive... | \n", + "Hello and Welcome to ‘Ask A Doctor’ service.I ... | \n", + "Description: Is Minoxin 5% lotion advisable in... | \n", + "232 | \n", + "[-0.18737659, 0.12219846, 0.2365137, 0.1126744... | \n", + "
256915 | \n", + "Are Biotin supplements need to reduce severe h... | \n", + "iam having hairfall for a decade.. but fews we... | \n", + "you did'nt mention about thyroid problem ...us... | \n", + "Description: Are Biotin supplements need to re... | \n", + "213 | \n", + "[-0.032349, -0.050617322, 0.30625877, 0.201994... | \n", + "
1000 rows × 6 columns
\n", + "\n", + " | Description | \n", + "Patient | \n", + "Doctor | \n", + "combined | \n", + "n_tokens | \n", + "embedding | \n", + "
---|---|---|---|---|---|---|
255916 | \n", + "What medicine is best for taking out pores and... | \n", + "Hi, I am (Simran) 28 years (Female) hight 5/3 ... | \n", + "Hi simran, your problem can be best sort out u... | \n", + "Description: What medicine is best for taking ... | \n", + "282 | \n", + "[-0.2568769, 0.14872037, 0.17243277, 0.2426784... | \n", + "
255917 | \n", + "What causes non conception despite unprotected... | \n", + "Hi, i am living with my partner from last six ... | \n", + "do serum tsh and serum prolactin den do hsg on... | \n", + "Description: What causes non conception despit... | \n", + "195 | \n", + "[-0.058888793, 0.084498726, -0.05540138, 0.021... | \n", + "
255918 | \n", + "How long after getting chicken pox is it safe ... | \n", + "i am 25 yrs old. last year in June i had an IU... | \n", + "Hi, I think you should keep a gap of 3 months ... | \n", + "Description: How long after getting chicken po... | \n", + "254 | \n", + "[-0.12680525, -0.01062898, -0.25460148, 0.0511... | \n", + "
255919 | \n", + "How to remove unwanted hair without any side e... | \n", + "hi doctor! i m 22 years old and i want to remo... | \n", + "Hello, which part of the body do you have thes... | \n", + "Description: How to remove unwanted hair witho... | \n", + "110 | \n", + "[0.12641467, -0.1543769, 0.42004266, 0.1722529... | \n", + "
255920 | \n", + "I am 15. Can i shave my pubic hair? | \n", + "sorry,im just curious because im 15 and when i... | \n", + "haha aw. Yes your supposed to shave, trim it f... | \n", + "Description: I am 15. Can i shave my pubic hai... | \n", + "223 | \n", + "[-0.06219541, 0.020706447, 0.38799724, -0.0219... | \n", + "
... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "
256911 | \n", + "Why is hair fall increasing while using Bontre... | \n", + "I am suffering from excessive hairfall. My doc... | \n", + "Hello Dear Thanks for writing to us, we are he... | \n", + "Description: Why is hair fall increasing while... | \n", + "211 | \n", + "[-0.17113408, 0.10835318, 0.33148944, -0.06146... | \n", + "
256912 | \n", + "Why was I asked to discontinue Androanagen whi... | \n", + "Hi Doctor, I have been having severe hair fall... | \n", + "hello, hair4u is combination of minoxid... | \n", + "Description: Why was I asked to discontinue An... | \n", + "154 | \n", + "[-0.24637492, 0.031407423, -0.05137701, -0.301... | \n", + "
256913 | \n", + "Can Mintop 5% Lotion be used by women for seve... | \n", + "Hi..i hav sever hair loss problem so consulted... | \n", + "HI I have evaluated your query thoroughly you... | \n", + "Description: Can Mintop 5% Lotion be used by w... | \n", + "191 | \n", + "[-0.32340947, 0.3667281, 0.3651925, -0.0989788... | \n", + "
256914 | \n", + "Is Minoxin 5% lotion advisable instead of Foli... | \n", + "Hi, i am 25 year old girl, i am having massive... | \n", + "Hello and Welcome to ‘Ask A Doctor’ service.I ... | \n", + "Description: Is Minoxin 5% lotion advisable in... | \n", + "232 | \n", + "[-0.18737659, 0.12219846, 0.2365137, 0.1126744... | \n", + "
256915 | \n", + "Are Biotin supplements need to reduce severe h... | \n", + "iam having hairfall for a decade.. but fews we... | \n", + "you did'nt mention about thyroid problem ...us... | \n", + "Description: Are Biotin supplements need to re... | \n", + "213 | \n", + "[-0.032349, -0.050617322, 0.30625877, 0.201994... | \n", + "
1000 rows × 6 columns
\n", + "\n", + " | Description | \n", + "Patient | \n", + "Doctor | \n", + "combined | \n", + "n_tokens | \n", + "embedding | \n", + "Cluster | \n", + "distance | \n", + "
---|---|---|---|---|---|---|---|---|
255916 | \n", + "What medicine is best for taking out pores and... | \n", + "Hi, I am (Simran) 28 years (Female) hight 5/3 ... | \n", + "Hi simran, your problem can be best sort out u... | \n", + "Description: What medicine is best for taking ... | \n", + "282 | \n", + "[-0.2568769, 0.14872037, 0.17243277, 0.2426784... | \n", + "7 | \n", + "3.389531 | \n", + "
255917 | \n", + "What causes non conception despite unprotected... | \n", + "Hi, i am living with my partner from last six ... | \n", + "do serum tsh and serum prolactin den do hsg on... | \n", + "Description: What causes non conception despit... | \n", + "195 | \n", + "[-0.058888793, 0.084498726, -0.05540138, 0.021... | \n", + "5 | \n", + "3.848544 | \n", + "
255918 | \n", + "How long after getting chicken pox is it safe ... | \n", + "i am 25 yrs old. last year in June i had an IU... | \n", + "Hi, I think you should keep a gap of 3 months ... | \n", + "Description: How long after getting chicken po... | \n", + "254 | \n", + "[-0.12680525, -0.01062898, -0.25460148, 0.0511... | \n", + "5 | \n", + "3.287058 | \n", + "
255919 | \n", + "How to remove unwanted hair without any side e... | \n", + "hi doctor! i m 22 years old and i want to remo... | \n", + "Hello, which part of the body do you have thes... | \n", + "Description: How to remove unwanted hair witho... | \n", + "110 | \n", + "[0.12641467, -0.1543769, 0.42004266, 0.1722529... | \n", + "6 | \n", + "3.875464 | \n", + "
255920 | \n", + "I am 15. Can i shave my pubic hair? | \n", + "sorry,im just curious because im 15 and when i... | \n", + "haha aw. Yes your supposed to shave, trim it f... | \n", + "Description: I am 15. Can i shave my pubic hai... | \n", + "223 | \n", + "[-0.06219541, 0.020706447, 0.38799724, -0.0219... | \n", + "6 | \n", + "3.841674 | \n", + "
... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "
256911 | \n", + "Why is hair fall increasing while using Bontre... | \n", + "I am suffering from excessive hairfall. My doc... | \n", + "Hello Dear Thanks for writing to us, we are he... | \n", + "Description: Why is hair fall increasing while... | \n", + "211 | \n", + "[-0.17113408, 0.10835318, 0.33148944, -0.06146... | \n", + "6 | \n", + "2.765943 | \n", + "
256912 | \n", + "Why was I asked to discontinue Androanagen whi... | \n", + "Hi Doctor, I have been having severe hair fall... | \n", + "hello, hair4u is combination of minoxid... | \n", + "Description: Why was I asked to discontinue An... | \n", + "154 | \n", + "[-0.24637492, 0.031407423, -0.05137701, -0.301... | \n", + "6 | \n", + "3.314942 | \n", + "
256913 | \n", + "Can Mintop 5% Lotion be used by women for seve... | \n", + "Hi..i hav sever hair loss problem so consulted... | \n", + "HI I have evaluated your query thoroughly you... | \n", + "Description: Can Mintop 5% Lotion be used by w... | \n", + "191 | \n", + "[-0.32340947, 0.3667281, 0.3651925, -0.0989788... | \n", + "6 | \n", + "3.229499 | \n", + "
256914 | \n", + "Is Minoxin 5% lotion advisable instead of Foli... | \n", + "Hi, i am 25 year old girl, i am having massive... | \n", + "Hello and Welcome to ‘Ask A Doctor’ service.I ... | \n", + "Description: Is Minoxin 5% lotion advisable in... | \n", + "232 | \n", + "[-0.18737659, 0.12219846, 0.2365137, 0.1126744... | \n", + "6 | \n", + "2.536337 | \n", + "
256915 | \n", + "Are Biotin supplements need to reduce severe h... | \n", + "iam having hairfall for a decade.. but fews we... | \n", + "you did'nt mention about thyroid problem ...us... | \n", + "Description: Are Biotin supplements need to re... | \n", + "213 | \n", + "[-0.032349, -0.050617322, 0.30625877, 0.201994... | \n", + "6 | \n", + "3.123004 | \n", + "
1000 rows × 8 columns
\n", + "