# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# <html>
# <table width="100%" cellspacing="2" cellpadding="2" border="1">
# <tbody>
# <tr>
# <td valign="center" align="center" width="25%"><img src="../../media/decartes.jpg"
# alt="DeCART Icon" width="128" height="171"><br>
# </td>
# <td valign="center" align="center" width="75%">
# <h1 align="center"><font size="+3">DeCART Summer School<br>
# for<br>
# Biomedical Data Science</font></h1></td>
# <td valign="center" align="center" width="25%"><img
# src="../../media/U_Health_stacked_png_red.png" alt="Utah Health
# Logo" width="128" height="134"><br>
# </td>
# </tr>
# </tbody>
# </table>
# <br>
# </html>
#
#
# # Working with Textual Data
# #### &copy; <NAME>, Ph.D.
#
#
# ### To get the latest version of this lesson, execute the cell below

# %matplotlib inline

import pymysql
import pandas as pd
import getpass
from textblob import TextBlob

conn = pymysql.connect(host="mysql",
                       port=3306,user="jovyan",
                       passwd=getpass.getpass("Enter MySQL passwd for jovyan"),db='mimic2')
cursor = conn.cursor()

# ### Use Pandas and SQL to create a dataframe with the following:
# * subject_id
# * hospital admission id
# * text of the radiology report
# * Limit the number of reports to 10000

rad_data = \ ### newline
pd.read_sql("""SELECT noteevents.subject_id, 
                      noteevents.hadm_id,
                      noteevents.text 
               FROM noteevents
               WHERE noteevents.category = 'RADIOLOGY_REPORT' LIMIT 10000""",conn)
rad_data.head(5)

rad_data.shape

print(rad_data['text'][0])


# ## Section Splitting
#
# <img src="http://medicine.utah.edu/dbmi/images/faculty/Chapman,Wendy_240x360.jpg"
# alt="<NAME>" width="128">
#
# Clinical reports are often divided into sections. Physicians are taught to write their reports in the SOAP format: subjective information, objective information, assessment, and plan, and sections often reflect this process. Information each setion may be formatted differently. For example, subjective information is often narrative, telling a story of the patient's description of their past medical history and their current issues. The objective section may be much more telegraphic with shortened descriptions of measurements like blood pressure and heart rate, such as "120/80 88". You may need different NLP techniques to accurately extract information out of different sections. Sentence splitting, for example, may be different in subjective versus objective sections. Radiology reports are not in the SOAP format, but section identification can still be very important, because in these reports the radiologist describes what she sees on the radiographic image then sometimes summarizes her impression of what the observations may represent, such as "the infiltrate could be consistent with pneumonia." For some tasks, you may only be interested in the impression, which has more weight than the reasoning that occurs in the Findings section.

# ## Exercise
#
# Write a function that returns the impression section of a report. The function will take two arguments:
#
# 1. The text to search for the impression section is the first positional argument.
# 1. A list of phrases to use to identify the impression section.
#
# Not every report will have a section labeled "IMPRESSION". You can consider "INTERPRETATION" and "CONCLUSIONS" as synonyms for "IMPRESSION"
#
# If you cannot identify the impresison section (or equivalent), return an empty string.
#
# #### Hints
#
# * Use a for loop  to iterate across
# * Use the find method of a string to identify where in the string the impression section occurs
# * Use slicing to grab the impression section
#

# +

def find_impression(report, section = None):
    if section == None:
        section = ['IMPRESSION:', 'INTERPRETATION:', 'CONCLUSIONS:']
    for s in section:
        index = report.find(s)
        if index != -1:
            return report[index:]
    return ""
            

print(find_impression(rad_data['text'][100]))

# -

type(print(find_impression(rad_data['text'][100])))

# ## Exercise 
#
# Use ``apply()`` to create a new column in the ``rad_data`` named "IMPRESSIONS" with the string identified with ``find_impression()``.

rad_data.head() ##, impressions, interpretation, conclusion %debug

rad_data['IMPRESSION'] = rad_data['text'].apply(lambda x: find_impression(x))

rad_data['IMPRESSION'] = rad_data.apply(lambda x: find_impression(x['text']), axis = 1)

rad_data.head()

# ## How else might we need to break up a report?
#
# * How would I break a report into sentences?
# * How would I break a report into words?

# ## Group Exercise
#
# ### Create a single string with all the reports
#
# #### Hints, etc.
# * Use List Comprehension
# * Use string joins
# * Iterate over the rows of the data frame
# * Use TextBlob to get words

imp = [row["IMPRESSION"] for _, row in rad_data[:5].iterrows()] ### throw away row number
 

d =[row["IMPRESSION"] for _, row in rad_data[:5]]

rad_data[:5]

help(rad_data.iterrows())

len(imp)

imps = []
for _, row in rad_data[:5].iterrows():
    imps.append(row["IMPRESSION"])

all_text =''
for impression in imp:
    all_text = all_text + impression

len(all_text)

all_text =" ".join([row["IMPRESSION"] for _, row in rad_data.iterrows()])

all_text

len(all_text)

from wordcloud import WordCloud
import matplotlib.pyplot as plt

wc = WordCloud().generate(all_text)

plt.imshow(wc)
plt.axis("off")



StarcoderdataJupyterScriptsDedupFiltered