""" ![image.png](attachment:image.png) """ """ https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge/tasks?taskId=572 """ """ <h1>Introduction</h1> <p>As the number of reported coronavirus (COVID-19) cases and deaths continue to climb across the U.S. and around the world, it is crucial for health providers to be aware of what has been published about best practices in medical care relating to COVID-19. Having access to the latest research can help prevent future spread of the virus and potentially save lives. This document provides a way to access the most relevant data to answer the question “What has been published about medical care?”</p> """ """ <h2>Subtasks</h2> """ """ <p>The question “What has been published about medical care?” is composed of the 16 subtasks below.</p> <ol> <li>Resources to support skilled nursing facilities and long-term care facilities.</li> <li>Mobilization of surge medical staff to address shortages in overwhelmed communities.</li> <li>Age-adjusted mortality data for Acute Respiratory Distress Syndrome (ARDS) with/without other organ failure – particularly for viral etiologies.</li> <li>Extracorporeal membrane oxygenation (ECMO) outcomes data of COVID-19 patients.</li> <li>Outcomes data for COVID-19 after mechanical ventilation adjusted for age.</li> <li>Knowledge of the frequency, manifestations, and course of extrapulmonary manifestations of COVID-19, including, but not limited to, possible cardiomyopathy and cardiac arrest.</li> <li>Application of regulatory standards (e.g., EUA, CLIA) and ability to adapt care to crisis standards of care level.</li> <li>Approaches for encouraging and facilitating the production of elastomeric respirators, which can save thousands of N95 masks.</li> <li>Best telemedicine practices, barriers and facilitators, and specific actions to remove/expand them within and across state boundaries.</li> <li>Guidance on the simple things people can do at home to take care of sick people and manage disease.</li> <li>Oral medications that might potentially work.</li> <li>Use of AI in real-time health care delivery to evaluate interventions, risk factors, and outcomes in a way that could not be done manually.</li> <li>Best practices and critical challenges and innovative solutions and technologies in hospital flow and organization, workforce protection, workforce allocation, community-based support resources, payment, and supply chain management to enhance capacity, efficiency, and outcomes.</li> <li>Efforts to define the natural history of disease to inform clinical care, public health interventions, infection prevention control, transmission, and clinical trials.</li> <li>Efforts to develop a core clinical outcome set to maximize usability of data across a range of trials.</li> <li>Efforts to determine adjunctive and supportive interventions that can improve the clinical outcomes of infected patients (e.g. steroids, high flow oxygen).</li> </ol> """ """ <h2>Relevant, Simplified and Accessible Results</h2> <p>Within this document, the most relevant research for each of the 16 subtasks above is presented in a simple, easy to use results table which contains the following:</p> <ul> <li>a short summary of the results for each subtask</li> <li>the top five documents related to each subtask</li> <li>a brief two sentence summary for each of the top five documents for each subtask</li> </ul> <p>For convenience, each of the top five documents may be immediately accessed by clicking on the document title within the results table.</p> """ """ <h1>Methodology</h1> <p> There are tens of thousands of scholarly articles related to COVID-19 and similar coronaviruses in the COVID-19 Open Research Dataset (CORD-19). CORD-19 is growing rapidly each day as researchers learn more about the virus. Because CORD-19 contains the most comprehensive bank of coronavirus research data available, it is the resource used for this text and data mining solution. Numerous algorithms were tested to determine which would provide the most relevant query results to assist researchers. It was determined that a combination of Python’s BM25 ranking function and Word2vec would be used to obtain the most accurate and relevant results. </p> <h2>Text Summarization With BM25 and Word2vec</h2> <p> Two models, BM25 and Word2vec, were used to analyze the text and produce the desired query results for each of the 16 subtasks. Python's BM25 (Best Matching) ranking function ranks a set of documents according to their relevance in the given search query. The query performs the following actions: - Filters documents to include only those published between 2012-2020 to ensure data relevance - Searches the combined fields of full text+title+abstract to produce more accurate results Word2vec summarizes CORD-19 by making predictions about a word's meaning based on its past appearance in the text. A Skip Gram Word2vec model was employed where the current word predicts the surrounding window of context words and weighs nearby context words more heavily than more distant context words. The Skip Gram model, though it runs more slowly than other models, is better for handling infrequent words. </p> """ """ <h1>Process Flow</h1> <p> Below is the overall process flow that is used to obtain the final output that best answers the question "What Has Been Published About Medical Care?" <ol> <li>Train the BM25 model against the list of clean text (full text+title+abstract) from all the prefiltered documents.</li> <li> Construct the question key words which consist of: original question + list of customized synonyms for the questions + extra key words + common medical key words. </li> <li> Apply the subtask's key words against the corpus to obtain the list of top ranked documents.</li> <li> Apply text summarization for each top ranked document to obtain the top ranked sentences that match the question's key words. </li> <li> Rank the top sentences for each document to produce the overall top ranked sentences for the subtask.</li> <li> Display the top two sentences from all the top ranked documents as the overall summary.</li> <li> Present the final results from the top ranked documents and the text summarization in an HTML table format.</li> </ol> """ """ ![image.png](attachment:image.png) """ """ <h1>Libraries</h1> <p>This project was created with common open source libraries used for Python and machine learning.</p> """ # Following librariers are not included in Kaggle and require install !pip install rank_bm25 !pip install langdetect # # Standard Python Libraries # import csv import gc import glob import heapq import json import pickle import os import re import string import sys # # Libraries licensed under BSD # import numpy as np import pandas as pd from IPython.display import HTML from sklearn.manifold import TSNE from sklearn.feature_extraction.text import CountVectorizer from sklearn.feature_extraction.text import TfidfTransformer from sklearn.metrics.pairwise import cosine_similarity # # Libraries licensed Apache 2.0 # import nltk from nltk.stem import WordNetLemmatizer from nltk.corpus import stopwords from nltk.tag import pos_tag from nltk.tokenize import word_tokenize from nltk.corpus import wordnet as wn from langdetect import detect # # Libraries licensed under LGPL # import gensim from gensim.models import KeyedVectors # # Libraries liceensed under MIT # import ipywidgets as widgets from ipywidgets import interact, interact_manual from rank_bm25 import BM25Okapi %%javascript // Disable scrolling in jupyter notebook IPython.OutputArea.prototype._should_scroll = function(lines) { return false; } """ <h2>Text Cleaning and Data Loading Utility Functions</h2> <p>The text is cleaned before performing data modeling to allow processing to proceed in a uniform manner. The text is lemmatized (converted to its base form) to ensure accurate text analysis. The text is normalized by converting it to lowercase and removing punctuation and digits. Text in languages other than English are removed.</p> """ DATA_PATH = '/kaggle/input/CORD-19-research-challenge' W2V_PATH = '/kaggle/input/covid19-w2v/' stopwords = stopwords.words('english') lemmatizer = WordNetLemmatizer() def sent_to_words(sentences): #for sentence in sentences: yield(gensim.utils.simple_preprocess(str(sentences), deacc=True)) def remove_stopwords(texts): return [[word for word in doc if word not in stopwords] for doc in texts] def clean_text(x): formatted_article_text = re.sub('[^a-zA-Z]', ' ', x) doc = re.sub(r'\s+', ' ', formatted_article_text) words = list(sent_to_words(doc)) words_cleaned = remove_stopwords(words) tokens = [lemmatizer.lemmatize(w.strip()) for w in words_cleaned[0]] return tokens def split_word(text, separator): tokenized_word = [text.split(separator) for doc in text] return tokenized_word def get_year(date): # expected input format yyyy-mm-dd # case when date=NaN, passed in value is fload if isinstance(date, float): yr="" elif len(date)==0: yr="" else: x = split_word(date, "-") yr = x[0][0] return yr def detect_lang(text): try: portion=text[0:400] lang=detect(portion) except Exception: lang=None return lang """ <h1>Load and Parse Data from Kaggle</h1> <p>Data is received from Kaggle in a zipped file which must be unzipped to be usable. Once the file is unzipped, Python is used to generate a data frame so the data may be easily accessed.</p> """ # Input data files are available in the "../input/" directory. # will list all files under the input directory # class to be used to fetch json records class FileReader: def __init__(self, file_path): with open(file_path) as file: content = json.load(file) self.paper_id = content['paper_id'] self.abstract = [] self.body_text = [] # Abstract for entry in content['abstract']: self.abstract.append(entry['text']) # Body text for entry in content['body_text']: self.body_text.append(entry['text']) self.abstract = '\n'.join(self.abstract) self.body_text = '\n'.join(self.body_text) def __repr__(self): return f'{self.paper_id}: \n{self.abstract[:400]}...\n{self.body_text[:400]}...' def doi_url(d): return f'http://{d}' if d.startswith('doi.org') else f'http://doi.org/{d}' def load_data_from_kaggle(): file_counter = 0 outer_loop = True for dirname, _, filenames in os.walk(DATA_PATH): for filename in filenames: file_counter += 1 print(filename) if file_counter > 10: outer_loop = False break else: continue break # Any results you write to the current directory are saved as output. root_path = '/kaggle/input/CORD-19-research-challenge/' all_json = glob.glob(f'{root_path}/**/*.json', recursive=True) filename = f'{root_path}metadata.csv' print(f'Metadata Filename: {filename}') len(all_json) # load in metadata # meta_data = pd.read_csv(filename, dtype={ 'doi': str, 'pubmed_id': str, 'Microsoft Academic Paper ID': str }) print(len(meta_data)) meta_data.head(2) # working with relevent fields # meta_data = meta_data[['sha','title','doi','abstract','publish_time','authors']] meta_data.abstract = meta_data.abstract.fillna(meta_data.title) print(f'record count: {len(meta_data)}') meta_data.head(2) # remove older years meta_data['publish_time_year'] = meta_data['publish_time'].str[:4] meta_data = meta_data[meta_data['publish_time_year'] >= '2012'] meta_data = meta_data.reset_index(drop=True) # removal of null and duplicate # duplicate_paper = ~(meta_data.title.isnull() | meta_data.abstract.isnull() | meta_data.doi.isnull()) & (meta_data.duplicated(subset=['title', 'abstract'])) meta_data = meta_data[~duplicate_paper].reset_index(drop=True) len(meta_data) meta_data.doi = meta_data.doi.fillna('').apply(doi_url) first_row = FileReader(all_json[0]) print(first_row) # Load the data into dataframe # dict_ = {'paper_id': [], 'abstract': [], 'body_text': []} counter = 0 for idx, entry in enumerate(all_json): if idx % (len(all_json) // 10) == 0: print(f'Processing index: {idx} of {len(all_json)}') try: content = FileReader(entry) dict_['paper_id'].append(content.paper_id) dict_['abstract'].append(content.abstract) dict_['body_text'].append(content.body_text) except: counter+= 1 dataframe = pd.DataFrame(dict_, columns=['paper_id', 'abstract', 'body_text']) print(f'Total records rejected due to wrong structure: {counter}') dataframe.head() # perform join between metadata and json files # left = meta_data[['sha','title','doi','abstract','publish_time','authors']] right = dataframe[['paper_id','body_text']] dataset = pd.merge(left, right, left_on='sha', right_on='paper_id', how='left') print(f'dataset ->: {len(dataframe)}') print(f'left ->: {len(left)}') print(f'right ->: {len(right)}') print(f'final ->: {len(dataset)}') del left del right del meta_data del dataframe # fill 'body_text' with 'title' where it's null dataset.body_text = dataset.body_text.fillna(dataset.title) # remode 'paper_id' dataset = dataset.drop(['paper_id'], axis = 1) # count of records len(dataset) # considering usefull columns dataset = dataset.rename(columns={"sha": "document_id", "body_text": "full_text"}) dataset = dataset[['title','doi','abstract','authors','full_text','publish_time']] dataset.head(2) # Utilize English only documents dataset['language'] = dataset['full_text'].apply(lambda x: detect_lang(x)) dataset = dataset[dataset['language'] == 'en'] dataframe = dataset del dataset gc.collect() return dataframe # Load directly from Kaggle - will do this after Legal review # df_orig_data = load_data_from_kaggle() """ <h1>Dataset Cleansing</h1> <p>The dataset is cleaned further, lemmatized, and tokenized.</p> """ clean_text_list = [] for index, data in df_orig_data.iterrows(): if isinstance(data['title'], float): title = "" else: title = data['title'] if isinstance(data['abstract'], float): abstract = "" else: abstract = data['abstract'] if isinstance(data['full_text'], float): full_text = "" else: full_text = data['full_text'] text_to_clean = title + " " + abstract + " " + full_text clean_text_list.append(clean_text(text_to_clean)) """ <h2>Define the Subtasks with Helper Words to be Used as Queries on the BM25 Model</h2> <p>Task Info: https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge/tasks?taskId=572</p> """ """ Each of the 16 subtasks that help answer the overarching question "What has been published about medical care" are further defined to help achieve better search results. Synonyms and additional key words are provided with each subtask to further refine the results. """ # # List of questions as defined by the task # task_ques_list = ['Resources to support skilled nursing facilities and long term care facilities.', \ 'Mobilization of surge medical staff to address shortages in overwhelmed communities', \ 'Age-adjusted mortality data for Acute Respiratory Distress Syndrome (ARDS) with/without other organ failure – particularly for viral etiologies', \ 'Extracorporeal membrane oxygenation (ECMO) outcomes data of COVID-19 patients', 'Outcomes data for COVID-19 after mechanical ventilation adjusted for age', \ 'Knowledge of the frequency, manifestations, and course of extrapulmonary manifestations of COVID-19, including, but not limited to, possible cardiomyopathy and cardiac arrest.', \ 'Application of regulatory standards (e.g., EUA, CLIA) and ability to adapt care to crisis standards of care level.', \ 'Approaches for encouraging and facilitating the production of elastomeric respirators, which can save thousands of N95 masks.', \ 'Best telemedicine practices, barriers and faciitators, and specific actions to remove/expand them within and across state boundaries.', \ 'Guidance on the simple things people can do at home to take care of sick people and manage disease', \ 'Oral medications that might potentially work', \ 'Use of AI in real-time health care delivery to evaluate interventions, risk factors, and outcomes in a way that could not be done manually', \ 'Best practices and critical challenges and innovative solutions and technologies in hospital flow and organization, workforce protection, workforce allocation, community-based support resources, payment, and supply chain management to enhance capacity, efficiency, and outcome', \ 'Efforts to define the natural history of disease to inform clinical care, public health interventions, infection prevention control, transmission, and clinical trials', \ 'Efforts to develop a core clinical outcome set to maximize usability of data across a range of trials', \ 'Efforts to determine adjunctive and supportive interventions that can improve the clinical outcomes of infected patients (e.g. steroids, high flow oxygen)'] # # For each question, additional topic related words are provided to hone in optimal results # common_key = 'coronavirus SARS-CoV-2 2019-nCov COVID-19 COVID19 Novel Coronavirus SARS-CoV-2' extra_ques1='HHSC CMS HWS CDC HHAC DSHA AMDA LTCFs SPICE Isolate Isolation pandemic disinfectants cleaning supplies no-touch thermometers shortages contingency plans repurpose alternative machines specialized beds nurse skill resource diagnostics tools effectiveness health care PPE coronavirus elder care elderly disabled elderly personnel ill residents supplies staffing postmortem education training planning checklist self-assess strengths weaknesses preparedness mandatory requirements standards important areas risk assessments measures infection control protocol PACE personal protective equipment (PPE) face masks eye protection gowns gloves Centers for Medicare & Medicaid Services (CMS) guidance Programs All-Inclusive Care protocols high-risk population multiple chronic illnesses' extra_ques2='pandemic medical personnel mobilize intensive care unit (ICU) critical care unit (CCU) surge capacity registered nurses critically ill critical acute care telephone triage reactivate licenses reactivation states barriers child care options coverage expanding onboard surging demand mobilize widespread strict criteria for admissions postpone accelerate discharge alternative delivery platforms essential repurpose upskill rapid training reassignment shiftwork low risk settings final year medical nursing students relocating recruit ' extra_ques3='percentage % virus age years old died respiratory heart kidney renal lung liver breathing coughing infection MOF multiple organ failure sepsis distal organs organ crosstalk brain-lung communication acute kidney injury (AKI) extra-pulmonary organ failure cardiovascular neurologic function cirrhosis mechanical ventilation hypoxemia ventilator pressures renal function septic shock acidosis heart failure CHF cardiac disease BMI circulating inflammatory mediators SISR' extra_ques4='age percentage % virus respiratory breathing coughing infection clinical management guidelines acute respiratory distress syndrome (ARDS) severe ARDS refractory medical management benefit conventional treatment supported venovenous restricted specialised complex therapies therapeutic interventions planning preparedness judicious resource allocation training personnel adhering adherence strict infection control measures crucial components action plan specialist centres critically ill staff equipment Venovenous (VV) extracorporeal membrane oxygenation temporary mechanical assistance device prolonged cardiopulmonary support cardiopulmonary bypass machinelife-threatening respiratory failure mortality data hypoxemia organ support Diffuse pulmonary edema hyaline membrane formation mechanical ventilation' extra_ques5='virus outcome, results, respirators, ventilators adjust adjustments, age demographics, mortality, prognosis acute hypoxemic respiratory failure HFNC NIV intubation pneumonia ARDS ' extra_ques6='impact impacts frequency COVID-19 cardiomyopathy cardiac arrest fever dry cough fatigue renal failure sepsis septic shock AKI kidney failure multiorgan dysfunction (MOF) gastrointestinal diarrhea vomiting cardiac heart dehydration virus ' extra_ques7='Application regulatory adapt crisis standards level allocation scarace resources Emergency Use Authorization (EUA), EUA authority, EAU, Anti-Malaria Drugs Clinical Laboratory Improvement Amendments (CLIA) testing performed humans United States conventional contingency crisis indicator trigger duty care proportionality fairness PPE personal protective equipment healthcare coalition cohorting strategies (ECMO) conservation conserve prepare stockpile materials anticipate oxygen delivery devices substitute adapt alternative re-use medical triage re-allocate sterilize ventilators medications clinic spaces alternative staffing workforce space supplies N95 masks shortages plan contingency reassignment shiftwork low risk settings final year medical nursing students recruit CDC crisis standard CSC Institute Medicine tiered proactive strategies strategy surge plans capacity pandemic' extra_ques8='incentives partnerships emergency Defense Production Act (DPA) EO Executive Order supply chain advances, ehancement facilitatating easier ease make making production manufacture manufacturing producing building fabrication assembly creation design personal protective equipment PPE difficulty shortage medical resources critical care response responses companies pivot 3D-printing new startups cloth designers private sector pandemic' extra_ques9 ='monitoring patient Check symptoms surge capacity Telehealth Travel Healthcare HIPAA Access to healthcare Online Virtual visit Frequently Asked Questions Assessment Screening Videoconference Medical History On-call Doctor Appointment Telehealth surge capacity State Borders HIPAA Virtual visit Videoconference Online doctor Cameras Abulatory Care Access to Healthcare Actuators American Telemedicine Association Appointment Clinical Applications Control Professional Questionnaire Diagnosis Digital Stethoscope E-Health Evolving Health Family Doctor Government Bureaucracy Guidelines Online Test Results Immune System Immunocompromised Medical History Medical Records Nurses Outpatient Paramedics Physiotherapist Prescription Protocol Quarantine Remote Presence Solution Screening Self-Care Sensors Tele-Case Management Tele-Consulting Tele-Diagnosis Tele-education Tele-facilitator Telehealth Tele-Monitoring Tele-Reminder Triage Underlying Conditions Urgent Care Videoconference Appointment visit Vital Signs' extra_ques10='CDC Doctor nursing homes Home care Video conference Patient Self-Isolation Hydration Disinfect Sterilize Alcohol Hand Sanitizer Antibacterial Limit Contact Separate Utensils Separate Laundy Hot water Shower Gloves Mask N-95 Support Severe Moderate Shortness of Breath Self-Care Supportive Hand wash Soap Mask Isolation Temperature check Hydration Clean 911 Emergency water Humidifier Isolate isolating OTC Medication SOB Utensils Sterilize Vaporizer Ventilate Bath Warm non-pharmaceutical countermeasures Shower Soap Social distancing Breathing Therapy Clean Disinfect Doctor Drink Plenty Fluids Remedies bathroom bedroom bed Compresses inhalers humidifiers rest breathing treatments acetaminophen tylenol antipyretics isolation nutrition vitamins cough drops' extra_ques11='fever cough body aches malaise inflammation sore throat TB antimalaria antibiotics pneumonia drug antihistaminic corticosteroids beta antagonists t-cell blockers fda food drug administration (FDA) drugs treatment acute exacerbation maintenance medication over the counter (OTC) quick relievers biologic drugs spray tablet capsule injection children medication medication compressor ventilator malaria alternative pneumonia Bronchitis Allergy Alternative Alveolates Anti-inflammatory Anti-malaria Antiviral Blood Count Bronchial Tubes Bronchodilators Anti-viral Immunization Immunotherapy Inflammation Longer Acting Inhaler Lymphocytes Oxygen Therapy Plasma Platelets Protein Red Blood Cells Rescue Inhaler Scarce Medicine Steroid T-Cell Blockers Treatment Vaccines chloroquine phosphate hydroxychloroquine sulfate analgesics azithromycin Antiviral Remdesivir Niclosamide Nitazoxanide anthelmintic aniparasitic Afatinib Avigan emergency-use authorization (EUA) discovery development Gero' extra_ques12='pandemic NLP regression analysis artificial intelligence machine learning learning–based models big data thermal sensors database ct scan parse electrocardiogram ECG Digital Health Learning Collaborative DHLC electronic health record(s) HER software clinical trials medical device SaMD automation recommendations visualization information shared decision challenges limitations adoption maintenance algorithms scalable storage silos integration integrate ntegrated transfers population parent ethical equity inclusivity inclusive population-representative datasets developers implementers biase biases inequities experience USE-CASE Gero domain transparency trust HEALTH CARE SYSTEM HUMAN FACTORS IMPLEMENTATION SCIENCE practice framework ethical inclusivity software development developers implementation science human–computer interaction information technology (IT) governance strategy psychosocial risk factors benefits deployers deployment deploying regulation liability real-world transparency solutions protection privacy consent intellectual property (IP) technologies developmental processes transformative disruptive advances unintended consequences better health improved care experience clinician lower cost health internet of things IOT accurately predicted severe respiratory disease laboratory demographic radiological findings decision-support tool capabilities predictive analytics patterns age gender liver enzyme alanine aminotransferase (ALT) reported myalgia hemoglobin levels ARDS accuracy respiratory distress dataset tracking monitoring real time tracking controlling biomedical research molecular modeling pharmacological databases genomic effects interventions multi-factor learning data-driven mining scientific literature past present future assessing mitigating monitoring hospital beds tracking availability recognizing recognition signs symptoms' extra_ques13 ='emergency preparedness strategies T-T-T framework triage test treat Virtual care Telehealth minimizing risk exposure fever clinics drive-thu testing isolation protocol protocols iPads iPad tablet remote patient monitoring strategy communication training PPE focus facts science access to care infection control protocol drive-through new technological solutions reinforce teams FEMA urgent clean recycle Emergency Use Authorization (EUA) HSS CDC vendors data-informed decisions maintaining expanding facility capacity testing safety Staffing Supporting Staff shortages Critical Supplies Materials Logistic Support Ventilators increased cost decreased revenue CDC stockpile misinformation education workloads non-traditional conserving reusing training ventilators childcare laundry nasal swabs viral transfer media reagents exposure unable limitations improvised scarcity challenges extended Strain availability unnecessarily stays hampered grocery services hotel accommodations ambulatory care telehealth alternate anesthesia machines renting leasing affiliated affiliation reassignment licensed professionals settings modalities eligible reimbursement flexibility financial assistance faster increased Medicare payments loans grants Hospitals centralized communication public information evidence-based guidance reliable data predictive models central repository COVID-19-related specialized staffing severe shortages waits test results limited shortages uncertainty medical equipment disrupted increased demand Strategic National Stockpile poorquality products high-prices traveling specialized staff providers respiratory therapist therapists quarantine operations anxiety emotional specialized beds specialized beds nurses disinfectants cleaning supplies no-touch thermometers shortages contingency plans repurpose alternative machines specialized beds ethical allocation liability developing revising guidelines financial viability workforce shortages financial nightmare financial nightmare insufficient Federal State local inconsistency extraordinary non-traditional sources disruptions Conservation non-medical-grade full-feature ventilators hotlines Social Distancing Restricting Access temporary walls temporary critical care units incident command centers line credit fitting converting sharing ' extra_ques14='designing design studies impact impacts interventions chemotherapeutic environmental initiation expression Identification biomarkers disease progression important indicators drug targets surrogate outcomes processes evolve evolution chronological periods subjects patients epidemiological statistical issues limitations challenges designs population samples infectious and chronic conditions protocols treatments medications vaccines testing self isolation social distancing mask N95 PPE hand washing gloves Multi-state schematic disease progression disease–free death initiation expression Epidemiological Studies Risk asssessment factors transmissibility asymptomatic period rapid risk assessment pandemic preparedness plans public awareness social distancing communication PPE healthcare workers surgical masks gowns aprons hand hygiene alcohol sanitizer surfaces copper metal plastic cardboard respiratory droplets cough management airborne ECDC guidelines non-pharmaceutical countermeasures School day care measures closure playbook playbooks' extra_ques15='drug studies COMET Initiative focus development framework outcome measurement domains definitions measurement instruments timing awareness attention outcomes measured standardization reporting consideration issues checklist implementation disease intervention interventions management geographical COS protocols research tests vaccines medicines R&D research consensus strategic scientific collaborations actions priortities prioritization virus natural history transmission diagnostics animal environmental research origin management measures human-animal interface epidemiological studies characterization infection prevention control health care workers protection protect candidate therapeutics ethical ethics social sciences outbreak response critical control emergency diagnostics care innovations innovate improve survival recovery evaluate adjunctive supportive therapies quick evaluation steroids high flow oxygen protective equipment infection prevention control measures health care community settings transmissibility susceptible susceptibility investigational therapeutics “Master Protocols” drug diagnostics funders communication interaction samples Epidemiological characterisation management characterization outbreak response physical psychological health guidelines survey diagnostics ' extra_ques16='IVAA oxygen therapy conservative fluid management empiric antimicrobials monitor corticosteroids RCTs clinical trials NSAIDS nonsteroidal anti-inflammatory drugs (NSAIDS) (ACEi) (ARBs) mechanical ventilation high nasal flow oxygen (HFNO) non-invasive ventilation (NIV) intubation lung-protective ventilation stategies anti pyretics hospitalization prevent complications medical trials clinical trials collaboration ' extra_key_list = [] extra_key_list.append(extra_ques1) extra_key_list.append(extra_ques2) extra_key_list.append(extra_ques3) extra_key_list.append(extra_ques4) extra_key_list.append(extra_ques5) extra_key_list.append(extra_ques6) extra_key_list.append(extra_ques7) extra_key_list.append(extra_ques8) extra_key_list.append(extra_ques9) extra_key_list.append(extra_ques10) extra_key_list.append(extra_ques11) extra_key_list.append(extra_ques12) extra_key_list.append(extra_ques13) extra_key_list.append(extra_ques14) extra_key_list.append(extra_ques15) extra_key_list.append(extra_ques16) """ <h2>Synonym Functions</h2> <p>The BM25 and Word2vec queries also use a set of predefined key words and Word Net synonyms to select the top 10 sentences in each document. A query runs the original subtask against a common synonym key and a preselected synonym. Then the two sentences that are the most similar to the query are extracted to generate a summary.</p> <p>These functions were utilized to facilitate in the creation of the synonym lists. These functions are not currently being utilized when the notebook is ran.</p> """ def tag(Sentence): Sentence = Sentence.lower() words = word_tokenize(Sentence) words = pos_tag(words) return words def paraphraseable(tag): return tag.startswith('NN') or tag == 'VB' or tag.startswith('JJ') def pos(tag): if tag.startswith('NN'): return wn.NOUN elif tag.startswith('V'): return wn.VERB def synonyms(word, tag): lemma_lists = [ss.lemmas() for ss in wn.synsets(word, pos(tag))] lemmas = [lemma.name() for lemma in sum(lemma_lists, [])] return (lemmas) def synonymIfExists(Sentence): for (word, t) in tag(Sentence): if paraphraseable(t): syn = synonyms(word, t) if syn: if len(syn) > 1: yield syn continue yield syn def get_synonym(Sentence): syn_list = [] synList_unique = [] return_syn_list = [] for x in synonymIfExists(Sentence): for word in x: if word not in syn_list: syn_list.append(word) #Get unique list of synonym words for word in syn_list: if word not in synList_unique: synList_unique.append(word) seperator = ' ' return_syn_list = converttostr(synList_unique, seperator).replace('_',' ') return return_syn_list def converttostr(input_seq, seperator): # Join all the strings in list final_str = seperator.join(input_seq) return final_str """ <h2> Customized Synonym List </h2> <p> The synonym list is generated from the above code and is manually edited to ensure that is applicable to the subtasks and domain for the tasks specified. </p> """ # # For each subtask, additional synonym related words are provided to hone in optimal results # syn_ques1 ='resource resourcefulness imagination support back up back endorse indorse hold sustain hold up confirm corroborate substantiate affirm subscribe defend patronize patronise patronage keep going digest endure stand tolerate brook abide suffer put up facility installation adeptness adroitness deftness quickness readiness hanker long yearn retentive recollective tenacious farseeing farsighted foresighted foresightful prospicient term condition full term terminal figure care attention aid tending caution precaution forethought concern fear charge tutelage guardianship maintenance upkeep' syn_ques2 ='mobilization mobilisation militarization militarisation rush spate surge upsurge billow checkup medical checkup medical examination medical exam medical health check staff faculty stave address turn to speak direct call cover treat handle deal deficit shortage shortfall dearth overwhelm sweep overcome overtake interests community' syn_ques3 ='death rate fatality data information datum data point acute incisive suffering adult wet white lung early unsuccessful etiology aetiology virus' syn_ques4 ='tissue layer information datum point patient affected role patient role results outcome consequence period' syn_ques5 ='result resultant final outcome termination consequence effect event issue upshot information datum data point mechanical mechanically skillful ventilation ventilating breathing external respiration respiration historic period years old ' syn_ques6 ='cognition relative frequency absolute manifestation materialization materialisation expression demonstrating study trend course action path track form grade limited restrict restrain trammel limit bound confined circumscribe specify set determine define fix circumscribed modified special possible potential myocardiopathy apprehension catch ' syn_ques7 ='practical covering program applications programme diligence regulative standard criterion measure monetary ability accommodate adjust conform care attention aid tending caution precaution concern fear charge maintain maintenance crisis degree grade level tier stage point level modify horizontal layer apply' syn_ques8 ='approach attack plan access approach path advance production produce manufacture product output yield respirator inhalator salvage relieve preserve carry through pull bring aside make unnecessary necessary deliver redeem spare economize economise keep hold open conserve generate' syn_ques9 ='best better good full estimable honorable respectable beneficial just upright adept expert practiced proficient skillful skilful dear near dependable safe secure right effective in effect in force serious honest easily considerably substantially intimately advantageously comfortably practice pattern exercise drill practice session recitation praxis barrier roadblock particular specific action activity activeness military action natural process natural action action mechanism legal action action at law state province nation country land commonwealth body politic Department of State boundary limit virtual medical roadblocks challenges falcilitate specified grow cut collaborate collaboration ' syn_ques10 ='guidance counsel counseling counselling direction steering simple elemental uncomplicated unproblematic bare unsubdivided things thing matter make do perform execute fare make out come get along cause practice practise exercise suffice answer serve act behave manage arrange set home place dwelling domicile abode habitation dwelling house home family household nursing rest occupy use up lead direct conduct guide get hold assume acquire adopt take read convey choose select pick out accept have fill consider deal look at necessitate ask postulate need require involve call demand remove take away withdraw consume ingest take submit admit learn study claim exact pack charter engage subscribe contain hold contract care attention aid tending caution precaution forethought concern fear charge tutelage guardianship maintenance upkeep sick vomit purge cast sick puke throw up ill nauseated nauseous queasy sick tired pale pallid' syn_ques11 ='oral medicine medication medicament medicinal drug act function operate aleviate influence act solve figure out turn mouth PO treatment treat drug study trial potential candidate candidates ' syn_ques12 ='use usage utilization utilisation function purpose role usance manipulation Artificial Intelligence health wellness care attention aid tending caution precaution forethought concern maintenance upkeep delivery bringing transfer rescue deliverance measure evaluate assess appraise value pass judgment judge intervention intercession interference interposition treatment hazard endangerment danger risk infection exposure factor component constituent element result resultant final result outcome termination consequence effect event issue upshot manner mode style way fashion means direction path exist equal constitute represent make up comprise follow' syn_ques13 ='practice pattern exercise practice session recitation praxis critical vital decisive challenge advanced forward-looking innovative modern innovational groundbreaking solution answer result resolution technology hospital infirmary flow flowing flow rate rate of flow organization organisation arrangement system administration governance establishment constitution formation work force workforce manpower hands men protection protective covering protective cover security shelter trade protection allocation support reinforcement reenforcement documentation keep sustenance supporting accompaniment backup financial support funding backing financial resource resourcefulness provision provisions strand management direction enhance heighten raise capacity capability content resultant final result outcome termination consequence effect event issue superior favored increase proven tested issues increase' syn_ques14 ='attempt effort endeavor endeavour try campaign cause crusade drive movement specify define delineate delimit delimitate set determine fix limit natural cancel instinctive account chronicle story inform clinical care attention aid tending caution precaution forethought concern fear charge maintenance upkeep public world health wellness intervention intercession interference interposition treatment infection contagion transmission prevention bar control condition dominance ascendance ascendence ascendancy ascendency restraint command mastery controller transmittal transmitting transmittance transmission system test trial runtryout' syn_ques15 ='attempt effort endeavor endeavour cause drive movement develop evolve acquire produce get originate arise rise uprise build up explicate formulate train prepare educate modernize modernise make grow break essence effect clinical result resultant final result outcome termination consequence event issue maximize maximise useableness usability data information data point scope range reach compass ambit chain image range of a function test trial run trial run tryout ' syn_ques16 ='attempt effort endeavor endeavour try campaign cause crusade drive movement determine find find out ascertain shape mold influence regulate set specify define fix limit decide settle square check see watch learn intervention intercession interference interposition treatment care better improve amend ameliorate meliorate clinical result resultant final result outcome termination consequence effect event issue upshot infect taint septic infected patient affected role patient role steroid eminent flowing flow rate effectiveness ' synonym_list = [] synonym_list.append(syn_ques1) synonym_list.append(syn_ques2) synonym_list.append(syn_ques3) synonym_list.append(syn_ques4) synonym_list.append(syn_ques5) synonym_list.append(syn_ques6) synonym_list.append(syn_ques7) synonym_list.append(syn_ques8) synonym_list.append(syn_ques9) synonym_list.append(syn_ques10) synonym_list.append(syn_ques11) synonym_list.append(syn_ques12) synonym_list.append(syn_ques13) synonym_list.append(syn_ques14) synonym_list.append(syn_ques15) synonym_list.append(syn_ques16) """ <h3>BERT Tested but Not Used</h3> <p>The queries were also tested using the BERT algorithm (Bidirectional Encoder Represenations from Transformers) to extract the top keywords from the list of top 5 documents that were selected with BM25 and Doc2vec. However, it was determined that the output did not meet the objective as BERT was unable to accurately identify the key words in the data summary.</p> <p>In addition, using BERT had a negative impact on the processing time. For example, for the 16 subtasks that each have the top 5 documents selected (16x5 for a total of 80 documents), BERT took an average of 2 minutes to run per document. 80 documents X 2 minutes each = 160 minutes or 3 hours to produce the targeted output. A three hour processing time was deemed excessive for this task.</p> """ """ <h2>Pros and Cons of the Selected Methodology</h2> <p>Word2vec is the selected methodology. Word2vec finds the top two sentences from each document that are the most similar to the query. The query sentence is first converted into a vector by averaging the sum of all the word embeddings of the words in the sentence. It is then compared to all the sentences in the top documents (returned by the BM25 model) using cosine distance. The top two sentences with the highest score are selected. Additional words are added to each query using the Word Net synonyms and some manual supervision. Pros of using Word2vec: <ul> <li>Ease and flexibility of model training.</li> <li>Large contained vocabulary size.</li> <li>Mapping between the target word to its context word implicitly embeds the sub-linear relationship into the vector space of words.</li> </ul> Cons of using Word2vec: <ul> <li>Cannot learn the representation for words not contained in the model.</li> <li>Cannot separate some opposite word pairs. For example, “good” and “bad” are usually located very close to each other in the vector space which may limit the performance of word vectors in NLP tasks like sentiment analysis.</li> </ul> """ def question_to_vec(question, embeddings_model, dim=300): """ question: a string embeddings_model: dict where the key is a word and a value is its' embedding dim: size of the representation result: vector representation for the question """ sentence_embedding = np.zeros((dim,)) if question == "": return sentence_embedding count = 0 words = clean_text(question) for word in words: if word in embeddings_model.wv.vocab: sentence_embedding += embeddings_model.wv[word] count +=1 if len(words) > 1: sentence_embedding = sentence_embedding / count return sentence_embedding def rank_candidates(question, candidates, top_100_index, embeddings, dim=300): """ question: a string candidates: a list of strings (candidates) which we want to rank top_100_index: the index of the strings embeddings: word 2 vec embeddings dim: dimension of the current embeddings result: a list of pairs (initial position in the list, question) """ question_embedding = question_to_vec(question, embeddings, dim) candidate_scores = [] for idx, candidate in zip(top_100_index,candidates): candidate_embedding = question_to_vec(candidate, embeddings, dim) cos_dist = 1 - np.dot(question_embedding, candidate_embedding )/(np.linalg.norm(question_embedding)*np.linalg.norm(candidate_embedding)) candidate_scores.append((cos_dist, idx, candidate)) candidate_scores = sorted(candidate_scores, key = lambda x: x[0]) result = [(cos_dist,idx) for (cos_dist, idx, candidate) in candidate_scores] return result def key_sent(Text,query,embedding,n_sentences = 10): sentences = nltk.sent_tokenize(Text) tmp = rank_candidates(query, sentences,range(len(sentences)), embedding, dim = 300) top_n = [x[1] for x in tmp[:n_sentences]] return " \n\n".join([sentences[x] for x in top_n]) def get_highlights(Text,query,n_sentences = 10): sentences = nltk.sent_tokenize(Text) doc_sent = [clean_text(x) for x in sentences] bm25_sent = BM25Okapi(doc_sent) tokenized_query = list(set(clean_text(query))) top_10 = bm25_sent.get_top_n(tokenized_query, sentences, n=n_sentences) return " \n\n".join(top_10) def get_answer_w2v_df(text_df, question_key, num_sentence): key_sent_df = pd.DataFrame(columns=['Key_Sentences']) key_sent_df['Key_Sentences'] = [get_highlights(text,question_key,n_sentences=10) for text in text_df] answer = key_sent(" ".join(key_sent_df['Key_Sentences']),question_key,w2v_300,n_sentences = num_sentence) return answer def get_answer_w2v_txt(text, question_key, num_sentence): key_sentence = key_sent(text,question_key,w2v_300,10) answer = key_sent(key_sentence,question_key,w2v_300,n_sentences = num_sentence) return answer with open(W2V_PATH + 'ericsson_covid19_w2v_300_2.pickle','rb') as handle: w2v_300 = pickle.load(handle) handle.close() """ <h1>Build BM25 Model on the Cleaned Corpus</h1> <h4>Model info: https://pypi.org/project/rank-bm25/</h4> """ tokenized_corpus = clean_text_list bm25= BM25Okapi(tokenized_corpus) """ <h1>Putting it All Together!!</h1> <p>The model runs the queries to get the best data to produce the final output.</p> """ top_n = 3 # Number of top ranked documents to display final_docs = pd.DataFrame(columns=['Ques_num', 'Rank', 'Question', 'Title', 'Authors', 'Published_Date', 'Link', \ 'Ques_Summary', 'Summary']) # Loop through the question list for i in range(len(task_ques_list)): # create query string using the question + synonym + additional key words question = task_ques_list[i] question_key = question + " " + synonym_list[i] + " " + extra_key_list[i] + " " + common_key # clean and tokenize the query tokenized_query = list(set(clean_text(question_key))) # Utlize bm25 to score the documents based on the question, sort the list based on the score doc_scores = bm25.get_scores(tokenized_query) doc_tuple = [(doc_score,idx) for idx,doc_score in enumerate(doc_scores)] doc_tuple_sorted = sorted(doc_tuple, key=lambda x: x[0], reverse=True) top_100_index = [x[1] for x in doc_tuple_sorted[:100]] # Take the top 3 result_df = df_orig_data.iloc[top_100_index].head(top_n)[['authors','title','full_text','abstract', 'doi', 'publish_time']] # Query summarization of the individual question Ques_Ans_w2v = get_answer_w2v_df(result_df['full_text'], question_key, 1) # For each document reference, query summary j=0 for index, row in result_df.iterrows(): j=j+1 Doc_Ans_w2v = get_answer_w2v_txt(row['full_text'],question_key, 1) final_docs = final_docs.append({'Ques_num': i+1, 'Rank': j, 'Question': question, 'Title': row['title'], \ 'Authors': row['authors'], 'Published_Date': row['publish_time'], \ 'Link': row['doi'], 'Ques_Summary': Ques_Ans_w2v, \ 'Summary': Doc_Ans_w2v },ignore_index=True) """ # Results Visualization """ """ Once the data is processed, the findings are displayed in a simple and interactive HTML format so the most relevant data my be easily accessed and referenced. The Results Visualization section contains the comprehensive output generated from the queries and is comprised of the following: - each of the 16 subtasks - a single sentence that summarizes the findings - a table which contains interactive links to the top three documents that best address the subtask, author, publication date, and a summary composed of the top sentence in the document The subtask displays first with a summary of the results. Beneath the summary is the results table. Each of the 16 questions has its own results table with four headings across the top and three rows below. **Headings** <p>The four headings are as follows:</p> - *Title* - Clickable link which provides direct access to the document - *Authors* - Name of the first author with the remaining authors replaced with "et al" - *Published Data* - Calendar year, month and day when the article was pulished - *Summary* - Key points from the document extracted from the top three sentences in the document **Rows** - Each of the three rows in the results table contains one of the three top research documents extracted from CORD-19 based on the specified search criteria. """ # # Function to take an URL string and text string and generate a href for embedding # def href(val1,val2): return '<a href="{}" target="_blank">{}</a>'.format(val1,val2) # # Function to add column width for a particular column within the HTML string # def setColWidth(html, col, width): html = re.sub('<th>'+ col, '<th width="'+ width+ '%">'+col, html) return html # Function to replace additional authors with 'et al.' def etal(val): if isinstance(val, float): return_val = " " else: if ';' in val: if ',' in val: return_val = re.sub(',.*', ' et al.', val) else: return_val = re.sub(';.*', ' et al.', val) else: return_val = " " return return_val def setCaption(html, caption): html = re.sub('mb-0">', 'mb-0"><caption>' + caption + '</caption>', html) return html def format_answer(val): val = val.replace("\n","") val = val.replace("https:","") val = val.replace("http:","") val = val.replace("www.","") return val # # Function to generate HTML string # def createHtmlTable(df,prefix): # CSS string to justify text to the left css_str ='<style>.dataframe thead tr:only-child th {text-align: left;}.dataframe thead th {text-align: left;}.dataframe tbody tr th:only-of-type {vertical-align: middle;}.dataframe tbody tr th {vertical-align: top;}.dataframe tbody tr td {text-align: left;}.dataframe caption {display: table-caption;text-align: left;font-size: 12px;color: black;font-weight: bold;}</style>' # Create a new Title column and combines the URL link to allow the user to open the source document in another tab df['Title'] = df.apply(lambda row : href (row['Link'], row['Title']), axis=1) df['Authors'] = df.apply(lambda row : etal (row['Authors']), axis=1) df['Summary'] = df.apply(lambda row : format_answer (row['Summary']), axis=1) # Generate HTML table string html_str = df[[ 'Title', 'Authors', 'Published_Date', 'Summary' ]].to_html(render_links=True, index=False, classes="table table-bordered table-striped mb-0") html_str = html_str + '<hr>' # Set table caption html_str = setCaption (html_str, 'Top Published Documents') # Perform a few adjustments on the HTML string to make it even better html_str = re.sub('&lt;', '<', html_str) html_str = re.sub('&gt;', '>', html_str) html_str = setColWidth(html_str, 'Title', '31') html_str = setColWidth(html_str, 'Authors', '13') html_str = setColWidth(html_str, 'Published_Date', '11') html_str = setColWidth(html_str, 'Summary', '45') # Return the final HTML table string for display return css_str + prefix + html_str # #Function to generate HTML Q&A + table that can be displayed # pd.set_option('mode.chained_assignment', None) def create_html_per_question(df, ques_list): i=0 for ques in ques_list: i=i+1 new_section = 1 prefix = '<h4>#' + str(i) + ': ' + ques + '</h4>' df = (final_docs.loc[final_docs['Question'] == ques]) for index, data in df.iterrows(): if data['Question'] == ques and new_section == 1: answer_w2v = data['Ques_Summary'] prefix = prefix + '<p>' + answer_w2v + '</p>' new_section = 0 small_table = final_docs.loc[final_docs["Question"] == ques] display(HTML(createHtmlTable(small_table,prefix))) """ <hr> <h1><center><font color='darkred'>WHAT HAS BEEN PUBLISHED ABOUT MEDICAL CARE?</font></center></h1> <hr> """ create_html_per_question(final_docs, task_ques_list) """ <h1>A Bonus View of the Results</h1> <p>An additional interactive way to view the final output is available, if desired. To use this option, run the Notebook (this takes approximately 40 minutes). After running the Notebook, a drop-down menu becomes available in the results table to select any of the 10 subtasks. Select the desired subtask to view the top answer for that subtask.</p> """ pd.set_option('mode.chained_assignment', None) @interact def show_articles_filter(column=['Question'], x=task_ques_list): prefix = '<h4>' + x + '</h4>' df = (final_docs.loc[final_docs[column] == x]) for index, data in df.iterrows(): if data['Question'] == x: answer_w2v = data['Summary'] break; prefix = prefix + '<p>' + answer_w2v + '</p>' return(HTML(createHtmlTable(final_docs.loc[final_docs[column] == x],prefix))) """ <h1>Ericsson Team Members</h1> <ul> <li>Anderson, Michael</li> <li>Anderson, Pamela</li> <li>Bhaskar, Rishi</li> <li>Channabasappa, Karthik</li> <li>Foster, Patrice</li> <li>Gill, Tarandeep</li> <li>Gonzalez, Carlos</li> <li>Hoque, Mahfujul</li> <li>Jalli, Bhavika</li> <li>Jeronymo, Vitor</li> <li>Magpali, Duane</li> <li>Montoya, Gloria</li> <li>Prince, Becky</li> <li>Pulyala, Indira</li> <li>Sabapathy, Rajiva</li> <li>Sengupta, Pranab</li> <li>Singla, Vishal</li> <li>Teerdhala, Uma</li> <li>Trinh, Maria</li> <li>Umradia, Amay</li> <li>Villamizar, Gustavo</li> </ul> <h5>“Ericsson, the world’s leading telecommunications company, cares about doing good. This task was completed as part of our Ericsson for Good program, which allows our 90,000 employees to contribute to their communities.”</h5> """ """ <h2>Licensing</h2> <h4>PSF LICENSE AGREEMENT FOR PYTHON 3.8.2</h4> 1. This LICENSE AGREEMENT is between the Python Software Foundation ("PSF"), and the Individual or Organization ("Licensee") accessing and otherwise using Python 3.8.2 software in source or binary form and its associated documentation. 2. Subject to the terms and conditions of this License Agreement, PSF hereby grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python 3.8.2 alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright © 2001-2020 Python Software Foundation; All Rights Reserved" are retained in Python 3.8.2 alone or in any derivative version prepared by Licensee. 3. In the event Licensee prepares a derivative work that is based on or incorporates Python 3.8.2 or any part thereof, and wants to make the derivative work available to others as provided herein, then Licensee hereby agrees to include in any such work a brief summary of the changes made to Python 3.8.2. 4. PSF is making Python 3.8.2 available to Licensee on an "AS IS" basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON 3.8.2 WILL NOT INFRINGE ANY THIRD PARTY RIGHTS. 5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON 3.8.2 FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON 3.8.2, OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. 6. This License Agreement will automatically terminate upon a material breach of its terms and conditions. 7. Nothing in this License Agreement shall be deemed to create any relationship of agency, partnership, or joint venture between PSF and Licensee. This License Agreement does not grant permission to use PSF trademarks or trade name in a trademark sense to endorse or promote products or services of Licensee, or any third party. 8. By copying, installing or otherwise using Python 3.8.2, Licensee agrees to be bound by the terms and conditions of this License Agreement. <h4>The 3-Clause BSD License</h4> Note: This license has also been called the "New BSD License" or "Modified BSD License". See also the 2-clause BSD License. Copyright <YEAR> <COPYRIGHT HOLDER> Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. #### NumPy license Copyright © 2005-2020, NumPy Developers. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. Neither the name of the NumPy Developers nor the names of any contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. #### GNU LESSER GENERAL PUBLIC LICENSE Version 2.1, February 1999 Copyright (C) 1991, 1999 Free Software Foundation, Inc. 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. [This is the first released version of the Lesser GPL. It also counts as the successor of the GNU Library Public License, version 2, hence the version number 2.1.] Preamble The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public Licenses are intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. This license, the Lesser General Public License, applies to some specially designated software packages--typically libraries--of the Free Software Foundation and other authors who decide to use it. You can use it too, but we suggest you first think carefully about whether this license or the ordinary General Public License is the better strategy to use in any particular case, based on the explanations below. When we speak of free software, we are referring to freedom of use, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish); that you receive source code or can get it if you want it; that you can change the software and use pieces of it in new free programs; and that you are informed that you can do these things. To protect your rights, we need to make restrictions that forbid distributors to deny you these rights or to ask you to surrender these rights. These restrictions translate to certain responsibilities for you if you distribute copies of the library or if you modify it. For example, if you distribute copies of the library, whether gratis or for a fee, you must give the recipients all the rights that we gave you. You must make sure that they, too, receive or can get the source code. If you link other code with the library, you must provide complete object files to the recipients, so that they can relink them with the library after making changes to the library and recompiling it. And you must show them these terms so they know their rights. We protect your rights with a two-step method: (1) we copyright the library, and (2) we offer you this license, which gives you legal permission to copy, distribute and/or modify the library. To protect each distributor, we want to make it very clear that there is no warranty for the free library. Also, if the library is modified by someone else and passed on, the recipients should know that what they have is not the original version, so that the original author's reputation will not be affected by problems that might be introduced by others. Finally, software patents pose a constant threat to the existence of any free program. We wish to make sure that a company cannot effectively restrict the users of a free program by obtaining a restrictive license from a patent holder. Therefore, we insist that any patent license obtained for a version of the library must be consistent with the full freedom of use specified in this license. Most GNU software, including some libraries, is covered by the ordinary GNU General Public License. This license, the GNU Lesser General Public License, applies to certain designated libraries, and is quite different from the ordinary General Public License. We use this license for certain libraries in order to permit linking those libraries into non-free programs. When a program is linked with a library, whether statically or using a shared library, the combination of the two is legally speaking a combined work, a derivative of the original library. The ordinary General Public License therefore permits such linking only if the entire combination fits its criteria of freedom. The Lesser General Public License permits more lax criteria for linking other code with the library. We call this license the "Lesser" General Public License because it does Less to protect the