!mkdir part1

"""
Explanation: Exercise 2 - shell, pipes, and csvkit
In this exercise, we'll review a few shell commands and explore working with pipes and csvkit.
You will need to fill in commands whenever prompted.  Please replace the text with your solution.
Remember to submit your completed .ipynb file to Blackboard and to add/commit it to your Git repository and push it to GitHub.
Part 1 - shell commands, redirection, and pipes
Basic shell commands and redirection
Create a directory called part1 using mkdir.
End of explanation
"""


!mv part1 partone
!ls

"""
Explanation: Rename part1 to partone using mv.
End of explanation
"""


!ls > filelist.txt

!cat filelist.txt

"""
Explanation: Create a file named filelist.txt using the output from ls and the output redirector &gt;.
End of explanation
"""


!ls >> filelist.txt
!cat filelist.txt

!ls > filelist.txt
!cat filelist.txt

"""
Explanation: Append to filelist.txt using the output appending redirector &gt;&gt;.  Note the difference between the single &gt; and double &gt;&gt;.
End of explanation
"""


# YOUR CODE HERE
raise NotImplementedError()

"""
Explanation: What's the difference between &gt; and &gt;&gt;?
End of explanation
"""


# YOUR CODE HERE
raise NotImplementedError()

import os
assert 'mydirectory' in os.listdir('.')

"""
Explanation: Your turn
Complete the following tasks in the cells provided.  All the tests in the testing cells (with the assert statements) below should pass without error - be sure to execute those as well, and if you see errors, fix your answer and try testing again until there are no errors.
Create a directory called mydirectory.
End of explanation
"""


# YOUR CODE HERE
raise NotImplementedError()

assert 'myfiles.txt' in os.listdir('mydirectory')

myfiles = open('mydirectory/myfiles.txt').read()
assert 'exercise-02.ipynb' in myfiles

"""
Explanation: Using ls and output redirection, create a file called myfiles.txt in the directory mydirectory that contains the list of files in the current directory.
End of explanation
"""


assert 'myfiles.txt' not in os.listdir('mydirectory')

"""
Explanation: Clean up the directory you just created by removing its contents (the file you created) using rm.
End of explanation
"""


# YOUR CODE HERE
raise NotImplementedError()

assert 'mydirectory' not in os.listdir('.')

"""
Explanation: Now remove the directory itself using rmdir.
End of explanation
"""


!wget https://www.gutenberg.org/cache/epub/2500/pg2500.txt

"""
Explanation: Filters and pipes
Let's look at something a little more interesting.  Download the text of Herman Hesse's Siddhartha from Project Gutenberg:
End of explanation
"""


!mv pg2500.txt siddhartha.txt

"""
Explanation: Note: sometimes Project Gutenberg restricts access.  If that creates an error for you, you should be able to wget the same file from our class repository on GitHub at the url https://github.com/gwsb-istm-6212-fall-2016/syllabus-and-schedule/raw/master/exercises/pg2500.txt.
However you get the file, let's rename it to something easier to remember.
End of explanation
"""


!head siddhartha.txt

!tail siddhartha.txt

"""
Explanation: head and tail are very useful.  They let you take a quick peek at the start and end of files.
End of explanation
"""


!grep copyright siddhartha.txt

"""
Explanation: grep is one of the most useful filters.  It lets you search for and match lines that contain specific expressions.  For example, to find mentions of "copyright":
End of explanation
"""


!grep -n copyright siddhartha.txt

"""
Explanation: Notice anything that those lines have in common?
Let's add a little more information by including the -n flag to add matching line numbers.
End of explanation
"""


!grep -n river siddhartha.txt | head

"""
Explanation: Now let's look for any mention of "river".  This will match a lot of text, so we'll just take the first 10 matching lines by piping the output from grep into head.
End of explanation
"""


!grep river siddhartha.txt | wc

"""
Explanation: How many lines contain "river"?  We can count by piping into the word count tool wc.
End of explanation
"""


!grep river siddhartha.txt | wc -l

"""
Explanation: That's 109 matching lines, containing 1365 words and 7979 characters.  If you just wanted the lines by themselves, use wc -l:
End of explanation
"""


!grep time siddhartha.txt | wc -l

!grep -i time siddhartha.txt | wc -l

"""
Explanation: What if we want to match both upper- and lower-case text?  Use grep -i:
End of explanation
"""


# YOUR CODE HERE
raise NotImplementedError()

%sc h_other = head -1 other-lines.txt
assert "other" in h_other

%sc t_other = tail -1 other-lines.txt
assert "other" in t_other

"""
Explanation: Your turn
How many lines in Siddhartha contain "other" (just lower-case)?  Start by using grep to extract lines that match the word "other" in siddhartha.txt and redirecting it to a file called other-lines.txt.
End of explanation
"""


# YOUR CODE HERE
raise NotImplementedError()

assert "127" in _

"""
Explanation: Now count up the lines in the file you created using wc.
End of explanation
"""


!cat siddhartha.txt | grep -oE '\w{2,}' | head -10

"""
Explanation: Your answer should be 127!
Counting words with grep
By piping commands together we can do a lot of powerful things right at the command line.  Let's create a count of the most commonly occurring words in Siddhartha.  To do that, we could write a Python script that just counts words, but with the command line shell tools we only need to put a proper pipeline together and we can often accomplish tasks like this in one line.
First we need to split up the text lines into a word per line.  There are grep flags for that!
End of explanation
"""


!cat siddhartha.txt | grep -oE '\w{2,}' | sort | head -10

"""
Explanation: Now we need to sort them and count the unique tokens.  sort solves the first problem.
End of explanation
"""


!cat siddhartha.txt | grep -oE '\w{2,}' | sort | uniq -c | head -25

"""
Explanation: And uniq -c solves the second problem.
End of explanation
"""


!cat siddhartha.txt | grep -oE '\w{2,}' | tr '[:upper:]' '[:lower:]' | sort | uniq -c | head -25

"""
Explanation: But there's a catch... do you see it?
We need to convert all the words down into lower case so that we are correctly counting unique words.  There's another command, tr, for that.
End of explanation
"""


!cat siddhartha.txt | grep -oE '\w{2,}' | tr '[:upper:]' '[:lower:]' | sort | uniq -c | sort | head -10

"""
Explanation: ...and if we want to know only the top 10 words in Siddhartha, we need to sort the output.
End of explanation
"""


!cat siddhartha.txt | grep -oE '\w{2,}' | tr '[:upper:]' '[:lower:]' | sort | uniq -c | sort -n | head -10

"""
Explanation: But that sorts by character, not number.  Fortunately, sort -n does what we want.
End of explanation
"""


!cat siddhartha.txt | grep -oE '\w{2,}' | tr '[:upper:]' '[:lower:]' | sort | uniq -c | sort -rn | head -10

"""
Explanation: But that's the wrong end of the list!  Two ways to fix that:  (a) use tail instead of head; (b) use sort -rn, which will sort in reverse order.  Let's try the latter.
End of explanation
"""


# YOUR CODE HERE
raise NotImplementedError()

assert 'pg11.txt' in os.listdir('')

"""
Explanation: Your turn
Download Alice in Wonderland from http://www.gutenberg.org/cache/epub/11/pg11.txt (or https://github.com/gwsb-istm-6212-fall-2016/syllabus-and-schedule/raw/master/exercises/pg11.txt if the first url doesn't work).
End of explanation
"""


# YOUR CODE HERE
raise NotImplementedError()

assert 'alice.txt' in os.listdir('')

"""
Explanation: Now rename pg11.txt to alice.txt.
End of explanation
"""


!cat alice.txt | grep -oE '\w{2,}' | sort | uniq -c | head -25

"""
Explanation: Take a look at the next cell.  Will it find the top 25 unique words in Alice in Wonderland successfully?
End of explanation
"""


# YOUR CODE HERE
raise NotImplementedError()

"""
Explanation: Describe what needs to be done to the previous cell to get it to work correctly.  Describe it using words, explaining the issues, rather than using shell commands!
End of explanation
"""


# YOUR CODE HERE
raise NotImplementedError()

"""
Explanation: Okay, now implement your solution using shell commands with a pipeline.
End of explanation
"""


!wget https://s3.amazonaws.com/capitalbikeshare-data/2015-Q4-cabi-trip-history-data.zip

"""
Explanation: Part 2 - csvkit basics
Let's look at some CSV data using csvkit.  Download the 2015 fourth quarter trip dataset from Capital Bikeshare's trip history data:
End of explanation
"""


!unzip 2015-Q4-cabi-trip-history-data.zip

!mv 2015-Q4-Trips-History-Data.csv q4.csv

!head q4.csv

"""
Explanation: Let's unzip it, rename it to something short, and take a look.
End of explanation
"""


!csvcut -n q4.csv

"""
Explanation: csvkit gives us great tools for examining and working with CSV data.  We start by looking at the columns:
End of explanation
"""


!csvcut -c1,5,7 q4.csv | head -10

"""
Explanation: We can also extract just a few columns with csvcut:
End of explanation
"""


!csvcut -c1,5,7 q4.csv | head -10 | csvlook

"""
Explanation: ...and make it look better with csvlook:
End of explanation
"""


!csvcut -c1,5,7 q4.csv | csvgrep -c3 -m '21st & I St NW' | head -10 | csvlook

"""
Explanation: It gets even better.  Try csvgrep:
End of explanation
"""


!csvcut -c1,5,7 q4.csv | csvgrep -c3 -m '21st & I St NW' | csvsort -c2 | head -10 | csvlook

"""
Explanation: But wait, there's more:
End of explanation
"""


!csvcut -c1,5,7 q4.csv | csvgrep -c3 -m '21st & I St NW' | csvcut -c1 | csvstat

"""
Explanation: And you can perform basic statistics very easily:
End of explanation
"""


# YOUR CODE HERE
raise NotImplementedError()

"""
Explanation: Your turn
Which set of trips had the longer average trip duration:  trips starting at "Massachusetts Ave & Dupont Circle NW", or trips ending at "Massachusetts Ave & Dupont Circle NW"?
Use as many new cells as you need to compute the answer, and then write in your answer below in the "YOUR ANSWER HERE" cell.
End of explanation
"""


