case studywhat disease is thatfigure first wikipedia disease list"list of diseases ( - )figure shows the first entries of what you're afterthe diseases themselves diseaselistarray[ links it' time to index the diseases once they're indexedboth data entry and data preparation are effectively overas shown in the following listing listing indexing diseases from wikipedia the checklist is an array containing an array of allowed first characters if disease doesn' complyskip it looping through disease lists checklist [[" "," "," "," "," "," "," "," "," "," "][" "],[" "],[" "],[" "],[" "],[" "],[" "],[" "][" "],[" "],[" "],[" "],[" "],[" "],[" "],[" "][" "],[" "],[" "],[" "],[" "],[" "],[" "],[" "],[" "],[" "]doctype 'diseasesfor diseaselistnumberdiseaselist in enumerate(diseaselistarray)document for disease in diseaselist linkstype you'll tryindex if disease[ in checklist[diseaselistnumberand disease[ : !="list"looping through currentpage wikipedia page(diseaselists of links for client index(index=indexnameevery disease list doc_type=doctype,id diseasebody={"name"disease"title":currentpage title "fulltext":currentpage content}first check if it' except exception,ea diseasethen index it print str(ebecause each of the list pages will have links you don' needcheck to see if an entry is disease you indicate for each list what character the disease starts withso you check for this additionally you exclude the links starting with "listbecause these will pop up once you get to the list of diseases the check is rather naivebut the cost of having few unwanted entries is rather low because the search algorithms will exclude irrelevant results once you start querying for each disease you index the disease name and the full text of the page the name is also used as its index idthis is useful