!pwd

"""
Explanation: Table of Contents
<p><div class="lev1 toc-item"><a href="#Overview" data-toc-modified-id="Overview-1"><span class="toc-item-num">1&nbsp;&nbsp;</span>Overview</a></div><div class="lev2 toc-item"><a href="#pwd---Print-Working-Directory" data-toc-modified-id="pwd---Print-Working-Directory-11"><span class="toc-item-num">1.1&nbsp;&nbsp;</span>pwd - Print Working Directory</a></div><div class="lev2 toc-item"><a href="#ls---List-files-and-directory-names,-attributes" data-toc-modified-id="ls---List-files-and-directory-names,-attributes-12"><span class="toc-item-num">1.2&nbsp;&nbsp;</span>ls - List files and directory names, attributes</a></div><div class="lev2 toc-item"><a href="#mkdir---Make-a-new-directory" data-toc-modified-id="mkdir---Make-a-new-directory-13"><span class="toc-item-num">1.3&nbsp;&nbsp;</span>mkdir - Make a new directory</a></div><div class="lev2 toc-item"><a href="#cd---Change-to-a-particular-directory" data-toc-modified-id="cd---Change-to-a-particular-directory-14"><span class="toc-item-num">1.4&nbsp;&nbsp;</span>cd - Change to a particular directory</a></div><div class="lev2 toc-item"><a href="#rmdir---Remove-a-directory" data-toc-modified-id="rmdir---Remove-a-directory-15"><span class="toc-item-num">1.5&nbsp;&nbsp;</span>rmdir - Remove a directory</a></div><div class="lev2 toc-item"><a href="#cp---Copy-Files" data-toc-modified-id="cp---Copy-Files-16"><span class="toc-item-num">1.6&nbsp;&nbsp;</span>cp - Copy Files</a></div><div class="lev2 toc-item"><a href="#rm---Remove-files" data-toc-modified-id="rm---Remove-files-17"><span class="toc-item-num">1.7&nbsp;&nbsp;</span>rm - Remove files</a></div><div class="lev2 toc-item"><a href="#mv-:-Move-a-file" data-toc-modified-id="mv-:-Move-a-file-18"><span class="toc-item-num">1.8&nbsp;&nbsp;</span>mv : Move a file</a></div><div class="lev2 toc-item"><a href="#CURL---Getting-Data-from-the-Command-Line" data-toc-modified-id="CURL---Getting-Data-from-the-Command-Line-19"><span class="toc-item-num">1.9&nbsp;&nbsp;</span>CURL - Getting Data from the Command Line</a></div><div class="lev2 toc-item"><a href="#head/tail" data-toc-modified-id="head/tail-110"><span class="toc-item-num">1.10&nbsp;&nbsp;</span>head/tail</a></div><div class="lev2 toc-item"><a href="#grep:" data-toc-modified-id="grep:-111"><span class="toc-item-num">1.11&nbsp;&nbsp;</span>grep:</a></div><div class="lev2 toc-item"><a href="#Redirection-(or-Downloading)" data-toc-modified-id="Redirection-(or-Downloading)-112"><span class="toc-item-num">1.12&nbsp;&nbsp;</span>Redirection (or Downloading)</a></div>

# Overview

This is by no means an exhaustive list, the point is just to give you a feeler for what's possible. If you have used Linux or Mac, or have written code in Ruby, chances are you have used Unix commands already. If you're a Windows user, here are two good resources: 
* https://www.howtogeek.com/249966/how-to-install-and-use-the-linux-bash-shell-on-windows-10/
* https://www.howtogeek.com/howto/41382/how-to-use-linux-commands-in-windows-with-cygwin/

Another great resource in general on the basics of unix commands:
* http://matt.might.net/articles/basic-unix/

## pwd - Print Working Directory
End of explanation
"""


ls

!ls

ls -al

ls -Al

"""
Explanation: ls - List files and directory names, attributes
Some commonly used commands are below:
* -A: list all of the contents of the queried directory, even hidden files.
* -l: detailed format, display additional info for all files and directories.
* -R: recursively list the contents of any subdirectories.
* -t: sort files by the time of the last modification.
* -S: sort files by size.
* -r: reverse any sort order.
* -h: when used in conjunction with -l, gives a more human-readable output.
You can also combine the commands/flags. 
For example:
* -al
* -Al
Read more on this topic here: https://www.mkssoftware.com/docs/man1/ls.1.asp
End of explanation
"""


!mkdir NewFolder

ls

"""
Explanation: mkdir - Make a new directory
End of explanation
"""


cd NewFolder

!pwd

cd ..

!pwd

ls

"""
Explanation: cd - Change to a particular directory
End of explanation
"""


rmdir NewFolder

ls

"""
Explanation: rmdir - Remove a directory
If the folder is not empty, it need the "-r" flag. 
Example:
rmdir -r NewFolder
End of explanation
"""


ls

# Copy in the same directory
!cp 01.Unix_and_Shell_Command_Basics.ipynb Notebook01.ipynb 

ls

rm Notebook01.ipynb

ls

# Copy to another directory
!mkdir TempFolder
!cp 01.Unix_and_Shell_Command_Basics.ipynb TempFolder/File01.ipynb
!ls

cd TempFolder

ls

"""
Explanation: cp - Copy Files
Careful with the filenames! Will be overwritten without warning.
End of explanation
"""


pwd

!rm File01.ipynb

!ls

!pwd

!ls -al

!ls

cd ..

ls

"""
Explanation: rm - Remove files
Note that this is different to rmdir, which exists to remove a directory
End of explanation
"""


pwd

ls

rm -r TempFolder

ls

cp 01.Unix_and_Shell_Command_Basics.ipynb NewFile01.ipynb

ls

mkdir TempFolder02

ls

mv NewFile01.ipynb TempFolder02

ls

cd TempFolder02

ls

cd ..

rm -r TempFolder02

"""
Explanation: mv : Move a file
This is close to the the 'cut' function available for files on Windows.
When you use the 'mv' command, a file is copied to a new location, and removed from it's original location.
End of explanation
"""


!curl -L 'https://dl.dropboxusercontent.com/s/j2yh7nvlli1nsa5/gdp.txt'

!curl -L 'https://dl.dropboxusercontent.com/s/eqyhkf3tpgre0jb/foo.txt'

!curl -s "http://freegeoip.net/json/" | jq .

"""
Explanation: CURL - Getting Data from the Command Line
Let's begin by copying a simple tab-separated file.
The format is as below:

!curl -OptionalFlag 'http://url'
End of explanation
"""


!curl -s "http://api.open-notify.org/iss-now.json"

!curl -s "http://api.open-notify.org/astros.json"

"""
Explanation: In case your system doesn't have jq, you can follow the instructions here.
* https://stedolan.github.io/jq/download/
End of explanation
"""


!curl -X POST --include 'https://community-sentiment.p.mashape.com/text/' \
  -H 'X-Mashape-Key: YFWRiIyfNemshsFin8iTJy0XFUjNp1rXoY7jsnoPlVphvWnKY6' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -H 'Accept: application/json' \
  -d 'txt=My team lost badly! I am sad :('

"""
Explanation: Register for the Mashape API Market here: https://market.mashape.com
End of explanation
"""


pwd

ls

cd Data

ls

!head -n 3 sample.txt

!tail -n 4 sample.txt

!cat sample.txt

# Selecting specific fields
!cut -f2,3 sample.txt

!sort sample.txt

!sort -k 2 sample.txt

!cat nyt.txt

!wc nyt.txt

# Where 21 is the number of lines, 245 is the number of words, and 1515 is the number of characters.

!wc -w nyt.txt

"""
Explanation: Note: This is a free API, so I have exposed my API key in the code. In practice, if you are ever sharing code, please take adequate precautions, and never expose your private key.
head/tail
End of explanation
"""


pwd

ls

!cat nyt.txt

# Count the number of matches
!grep -c 'Kennedy' nyt.txt

!grep -o 'Kennedy' nyt.txt

"""
Explanation: grep:
Grep is a pattern matching utility built into unix and it's flavors. The typical format is:
grep [option] [pattern] [file/s]
End of explanation
"""


!curl -s 'http://freegeoip.net/json/' > location.json

!jq . location.json

!curl -s 'http://freegeoip.net/json/' | jq .

"""
Explanation: More options for grep:
* -c Print only a count of matched lines.
* -l List only filenames
* -i Ignore lowercase and uppercase distinctions
* -o prints only the matching part of the line
* -n Print matching line with its line number
* -v Negate matches; print lines that do not match the regex
* -r Recursively Search subdirectories listed
End of explanation
"""


!curl "https://data.cityofnewyork.us/resource/2pc8-n4xe.json" > venues.json

!cat venues.json

!grep 'Ripley' venues.json

!grep -i 'Theater' venues.json

# Multiple flags, and multiple conditions
!grep -v -e 'Theater' -e 'Theatre' venues.json

"""
Explanation: Redirection (or Downloading)
This is really useful to quickly download a dataset using what is called an API Endpoint. 
Let's download the 'Times Square Entertainment Venues' dataset from New York City's Open Data Portal to demonstrate this. 
https://data.cityofnewyork.us/Business/Times-Square-Entertainment-Venues/jxdc-hnze
End of explanation
"""


