!wget https://raw.githubusercontent.com/gwsb-istm-6212-fall-2016/syllabus-and-schedule/master/projects/project-01/women.txt

!cat women.txt | grep -oE '\w{{2,}}' \
   | grep -e "Jo\|Beth\|Meg\|Amy" \
   | tr '[:upper:]' '[:lower:]' \
   | sort | uniq -c | sort -rn

"""
Explanation: Problem 01 - solution
The following are my approach to solving these problems.  Note that there may be more than one approach to each, and we could even debate the exact solutions.
Problem 1 - Word Counts
Part A - Part A. Characters in Little Women
How many times are each of the following characters mentioned by name in the text of Little Women: Jo, Beth, Meg, Amy
End of explanation
"""


!wget https://raw.githubusercontent.com/gwsb-istm-6212-fall-2016/syllabus-and-schedule/master/projects/project-01/romeo.txt

"""
Explanation: As we can see from the output of the command above, by the straight wording of the question, there are exactly 1,355 mentions of Jo, 683 of Meg, 645 of Amy, and 459 of Beth in Little Women.  If we were to assume that diminutive or nickname forms might count as well, we might add mentions of "Megs", "Bethy", and "Meggy" to these counts, for example.  For the purposes of this solution, however, I assume that these are not required, because the text might need to be consulted directly by someone familiar with the novel to determine which nicknames are valid, which seems beyond the scope of the assignment.
Part B - Juliet and Romeo in Romeo and Juliet
How many times do each of the characters Juliet and Romeo have speaking lines in Romeo and Juliet? Keep in mind that this is the text of a play.
End of explanation
"""


!cat romeo.txt | grep "Rom" | head -25

"""
Explanation: First we must recall -- as the problem highlights -- that this text is that of a play.  Because of this, we cannot simply count mentions of "Romeo," as we might accidentally inflate the count due to mentions of this character, for example, by other characters in their speaking lines.  Instead, we must first look for a patter that indicates Romeo's speaking lines specifically.
End of explanation
"""


!cat romeo.txt | grep "Jul" | head -25

"""
Explanation: In this brief sample, we can see title lines and metadata that include mention of Romeo, and both stage directions ("Enter Romeo") and spoken lines that include his name.  What stands out, though, is that lines spoken by Romeo appear to be delineated by "Rom.", so we can search for this specific pattern.  Let's verify that the same should hold true for mentions of Juliet.
End of explanation
"""


!cat romeo.txt | grep -w "Rom\." \
   | grep -oE '\w{{2,}}\.' \
   | grep "Rom" \
   | sort | uniq -c | sort -rn

!cat romeo.txt | grep -w "Jul\." \
   | grep -oE '\w{{2,}}\.' \
   | grep "Jul" \
   | sort | uniq -c | sort -rn

"""
Explanation: We see that the pattern seems to hold for both.  I will assume that matches of the exact characters "Rom." and "Jul." indicate the start of a speaking line for one or the other characters, and will explicitly count only those lines.
End of explanation
"""


!wget https://raw.githubusercontent.com/gwsb-istm-6212-fall-2016/syllabus-and-schedule/master/projects/project-01/2016q1.csv.zip

!unzip 2016q1.csv.zip

!head -5 2016q1.csv | csvlook

!csvcut -n 2016q1.csv

!csvcut -c5 2016q1.csv | tail -n +2 | csvsort | uniq -c | sort -rn | head -10

"""
Explanation: The two pipelines above indicate that Romeo has 163 speaking lines, while Juliet has only 117.  To match the specific case with a trailing ., the first regular expressions in both above cases use the -w flag to denote a word match and the escape sequence \. to match the literal trailing period.  The second regular expressions include this literal at the end of the match sequence as well, with the trailing literal period in '\w{{2,}}\.' requiring that the match include the period at the end.
Problem 2 - Capital Bikeshare
Part A - Station counts
Which 10 Capital Bikeshare stations were the most popular departing stations in Q1 2016?
End of explanation
"""


!csvcut -c7 2016q1.csv | tail -n +2 | csvsort | uniq -c | sort -rn | head -10

"""
Explanation: As we can see in the above results, the top ten starting stations in this time period were led by Columbus Circle / Union Station with over 13,000 rides, followed by Dupont Circle and the Lincoln Memorial and the rest as listed.
In the pipeline above, tail -n +2 ensures we skip the header line before the sort process begins.
Which 10 were the most popular destination stations in Q1 2016?
End of explanation
"""


!csvgrep -c5 -m "Columbus Circle / Union Station" 2016q1.csv | head

"""
Explanation: The above results show us very similar numbers for destination stations during the same time period, with the first four stations unchanged and led again by Union Station with over 13,000 rides.  Thomas Circle appears to be a more prominent start station than end station, as does Eastern Market, which does not even make the top ten destination stations.
Part B - bike counts
For the most popular departure station, which 10 bikes were used most in trips departing from there?
In this part, we will use csvgrep to select only the required stations - Union Station, in both cases.
End of explanation
"""


!csvcut -c5,8 2016q1.csv | csvgrep -c1 -m "Columbus Circle / Union Station" | head

!csvcut -c5,8 2016q1.csv \
    | csvgrep -c1 -m "Columbus Circle / Union Station" \
    | csvcut -c2 \
    | tail -n +2 \
    | sort | uniq -c | sort -rn | head -12

"""
Explanation: We can further limit the columns used to cut down on the data flowing through the pipe.
End of explanation
"""


!csvcut -c7,8 2016q1.csv \
    | csvgrep -c1 -m "Columbus Circle / Union Station" \
    | csvcut -c2 \
    | tail -n +2 \
    | sort | uniq -c | sort -rn | head -15

"""
Explanation: Above are the most commonly used bikes in trips departing from Union Station, led by bike number W22227.  As we might expect it appears that the distribution seems rather uniform.  Note that because several bikes had exactly 15 trips starting from Union Station, the list includes the top twelve bikes, rather than the top ten.
Which 10 bikes were used most in trips ending at the most popular destination station?
End of explanation
"""


!wget https://raw.githubusercontent.com/gwsb-istm-6212-fall-2016/syllabus-and-schedule/master/projects/project-01/simplefilter.py

!cp simplefilter.py split.py

"""
Explanation: Above are the most commonly used bikes in trips arriving at Union Station, let by bike number W00485.  It is interesting to note that bike W22227, the top departing bike, is in second place, but bike W00485, the top arriving bike, does not appear in the top ten departing bikes.  In any case these also seem at first glance to be uniformly distributed.  Again, the list is expanded, this time to fifteen bikes, to account for the tie at exactly fifteen trips.
Problem 3 - Filters
Part A - split and lowercase filters
Write a Python filter than replaces grep -oE '\w{2,}' to split lines of text into one word per line, and write an additional Python filter to replace tr '[:upper:]' '[:lower:]' to transform text into lower case.
With your two new filters, repeat the original pipeline, and substitute your new filters as appropriate. You should obtain the same results.
End of explanation
"""


!cat women.txt \
    | grep -oE '\w{{1,}}' \
    | tr '[:upper:]' '[:lower:]' \
    | sort \
    | uniq -c \
    | sort -rn \
    | head -10

"""
Explanation: The file split.py is modified from the template to split lines of text into one word per line.  To demonstrate this, we can compare the original pipeline with a new pipeline with split.py substituting for the first grep command.
End of explanation
"""


!chmod +x split.py

"""
Explanation: We can ignore the broken pipe and related errors as the output appears to be correct.
Next, we repeat the pipeline with split.py substituted:
End of explanation
"""


!grep -n '' split.py

!cat women.txt \
    | ./split.py \
    | tr '[:upper:]' '[:lower:]' \
    | sort \
    | uniq -c \
    | sort -rn \
    | head -10

"""
Explanation: Examining the filter script below, the key line, #14, removes trailing newlines, splits tokens by the space (' '), and removes words that are not entirely alphabetical.
End of explanation
"""


!cat women.txt | grep -oE '\w{{2,}}' | head -25

!cat women.txt | ./split.py | head -25

"""
Explanation: Almost the exact words listed appear in nearly the same order, but with lower counts for each.  We can examine the output of each command to see if there are obvious differences:
End of explanation
"""


!head -3 women.txt

"""
Explanation: We can see straight away on the first few lines that there is a difference.  Let's look at the text itself:
End of explanation
"""


!grep -n '' split.py

!cat women.txt | ./split.py | head -25

"""
Explanation: Three obvious issues jump out.  First, the initial "The" is elided; it is not clear why.  Next, "Women" is removed, perhaps due to the trailing comma, which will cause the token to fail the isalpha() test.  Also, "Alcott" is removed, perhaps having to do with its position at the end of the line.
We can update the filter to use Python's regular expression model and a similar expression, \w{1,} to find all matches more intelligently.  Here the regular expression is prepared in line 13 and used in line 18.
End of explanation
"""


!cat women.txt \
    | ./split.py \
    | tr '[:upper:]' '[:lower:]' \
    | sort \
    | uniq -c \
    | sort -rn \
    | head -10

"""
Explanation: This looks much better.  We can try the full pipeline again:
End of explanation
"""


!cp simplefilter.py lowercase.py

"""
Explanation: This looks to be an exact match.
End of explanation
"""


!chmod +x lowercase.py

!grep -n '' lowercase.py

"""
Explanation: The filter lowercase.py is modified from the template to lowercase incoming lines of text.
End of explanation
"""


!head women.txt | ./lowercase.py

"""
Explanation: Note that the only line aside from the comments that changes in the above script is line #12, which adds the lower() to the print statement.
End of explanation
"""


!cat women.txt \
    | grep -oE '\w{{1,}}' \
    | ./lowercase.py \
    | sort \
    | uniq -c \
    | sort -rn \
    | head -10

"""
Explanation: This looks correct, so we'll first attempt to replace the original pipeline's use of tr with lowercase.py:
End of explanation
"""


!cat women.txt \
    | ./split.py \
    | ./lowercase.py \
    | sort \
    | uniq -c \
    | sort -rn \
    | head -10

"""
Explanation: Looks good so far, we are seeing the exact same counts.  To address the problem's challenge, we finally replace both filters at once.
End of explanation
"""


!wget http://www.textfixer.com/resources/common-english-words.txt

!head common-english-words.txt

"""
Explanation: This completes Problem 3 - Part A.
Part B - stop words
Write a Python filter that removes at least ten common words of English text, commonly known as "stop words". Sources of English stop word lists are readily available online, or you may generate your own list from the text.
We begin by acquiring a common list of English stop words, gathered from the site http://www.textfixer.com/resources/common-english-words.txt as linked from the Wikipedia page on stop words.
End of explanation
"""


!cp simplefilter.py stopwords.py

!chmod +x stopwords.py

!grep -n '' stopwords.py

"""
Explanation: Next we copy the template filter script as before, renaming it appropriately.
End of explanation
"""


!head women.txt | ./split.py | ./lowercase.py | ./stopwords.py

"""
Explanation: The key changes in stopwords.py from the template are line #13, which imports the list of stopwords, and line #20, which checks whether an incoming word is in the stopword list.  Note also that in line #19 the removal of a trailing newline occurs before checking for stopwords.
The assumption that incoming text will already be split into one word per line and lowercased is stated explicitly in the first comment, lines #6-7.
End of explanation
"""


!cat women.txt \
    | ./split.py \
    | ./lowercase.py \
    | ./stopwords.py \
    | sort \
    | uniq -c \
    | sort -rn \
    | head -25

"""
Explanation: This appears to be correct.  Let's put it all together:
End of explanation
"""


!cat women.txt \
    | grep -oE '\w{{1,}}' \
    | ./lowercase.py \
    | ./stopwords.py \
    | sort \
    | uniq -c \
    | sort -rn \
    | head -25

"""
Explanation: This would seem to be correct - we see the names we looked for earlier appearing near the top of the list, and common stop words are indeed removed - however the list starts with odd "words", in "t", "s", "m", and "ll".  Is it possible that these are occurences of contractions?  We can check a few different ways.  First, let's see if our split.py is causing the problem:
End of explanation
"""


!cat women.txt \
    | ./split.py \
    | ./lowercase.py \
    | grep --context=2 -oE '^t$' \
    | head -20

!grep -i "we haven't got" women.txt

"""
Explanation: No, the results are exactly the same.  Instead, we'll need to look for occurrences of "t" and "s" by themselves.  The --context option to grep might help us here, pointing out surrounding text to search for in the source.
End of explanation
"""


!cat women.txt \
    | ./split.py \
    | ./lowercase.py \
    | grep --context=2 -oE '^s$' \
    | head -20

!grep -i "amy's valley" women.txt

"""
Explanation: Aha, it does appear that the occurences of a bare "t" are from contractions.  Let's repeat with "s", which might occur in possessives.
End of explanation
"""


!cat women.txt \
    | ./split.py \
    | ./lowercase.py \
    | ./stopwords.py \
    | grep -v -oE '^s|t|m|ll$' \
    | sort \
    | uniq -c \
    | sort -rn \
    | head -25

"""
Explanation: There we have it - the counts from above were correct, and we could eliminate "t" and "s" from consideration with a grep -v, and we can further assume that the "ll" and "m" occurences are also from contractions, so we'll remove them as well.
End of explanation
"""


!wget https://raw.githubusercontent.com/gwsb-istm-6212-fall-2016/syllabus-and-schedule/master/projects/project-01/texts.zip

!unzip -l texts.zip | head -5

!mkdir all-texts

!unzip -d all-texts texts.zip

!time ls all-texts/*.txt \
    | parallel --eta -j+0 "grep -oE '\w{1,}' {} | tr '[:upper:]' '[:lower:]' | grep -v -oE '^s|t|m|l|ll|d$' | ./stopwords.py >> all-words.txt"

"""
Explanation: Here we have a final count.  It is interesting to note that these counts of character names (Jo, Meg, etc.) are slightly different from before, perhaps due to punctuation handling, but it seems beyond the scope of the question to answer it precisely.
Extra credit - parallel stop words
Use GNU parallel to count the 25 most common words across all the 109 texts in the zip file provided, with stop words removed.
End of explanation
"""


!wc -l all-words.txt

!time sort all-words.txt | uniq -c | sort -rn | head -25

"""
Explanation: In the above line, I've limited the word size to one character, removed common contractions, and piped the overall result through the new stopwords.py Python filter.
End of explanation
"""


