# Import the pandas dataframe library
import pandas as pd

# Import the seaborn library for plotting
import seaborn as sns

# Put all the plots directly into the notebook
%matplotlib inline

"""
Explanation: How to make a clustered heatmap
Now we'll break down how to read the clustered heatmap we made in 1.3_explore_gene_dropout_via_distance_correlation_linkage_clustering
End of explanation
"""


csv = "https://media.githubusercontent.com/media/olgabot/macosko2015/" \
         "master/data/05_make_rentina_subsets_for_teaching/big_clusters_expression.csv"
expression = pd.read_csv(csv, index_col=0)
print(expression.shape)
expression.head()

"""
Explanation: Read Expression data using pandas. Notice that pandas can read URLs (!), not just files on your computer!
End of explanation
"""


csv = "https://media.githubusercontent.com/media/olgabot/macosko2015/" \
         "master/data/05_make_rentina_subsets_for_teaching/big_clusters_cell_metadata.csv"
# YOUR CODE HERE

"""
Explanation: Exercise 1
Now use pd.read_csv to read the csv file of the cell metadata
End of explanation
"""


csv = "https://media.githubusercontent.com/media/olgabot/macosko2015/" \
         "master/data/05_make_rentina_subsets_for_teaching/big_clusters_cell_metadata.csv"
cell_metadata = pd.read_csv(csv, index_col=0)
print(cell_metadata.shape)
cell_metadata.head()

"""
Explanation: 
End of explanation
"""


expression.corr?

"""
Explanation: To correlate columns of dataframes in pandas, you use the function .corr. Let's look at the documentation of .corr

Is the default method Pearson or Spearman correlation? 
Can you correlate between rows, or only between columns?
End of explanation
"""


# YOUR CODE HERE

"""
Explanation: Since .corr only correlates between columns, we need to transpose our dataframe. Here's a little animation of matrix transposition from Wikipedia:

Exercise 2
Transpose the expression matrix so the cells are the columns, which makes it easy to calculate correlations. How do you transpose a dataframe in pandas? (hint: google knows everything)
End of explanation
"""


expression_t = expression.T
print(expression_t.shape)
expression_t.head()

"""
Explanation: 
End of explanation
"""


# YOUR CODE HERE

"""
Explanation: Exercise 3
Use .corr to calculate the Spearman correlation of the transposed expression dataframe. Make sure to print the shape, and show the head of the resulting dataframe.
End of explanation
"""


expression_corr = expression_t.corr(method='spearman')
print(expression_corr.shape)
expression_corr.head()

"""
Explanation: 
End of explanation
"""


expression_t.rank?

"""
Explanation: Pro tip: if your matrix is really big, here's a trick to make spearman correlations faster
Remember that spearman correlation is equal to performing pearson correlation on the ranks? Well, that's exactly what's happening inside the .corr(method='spearman') function! Every time it's calculating spearman, it's converting each row to ranks, which means that it's double-converting to ranks since it has to do it for each pair. Let's cut the work in half by converting to ranks FIRST. Let's take a look at the options for .rank:
End of explanation
"""


df = pd.DataFrame([[5, 6, 7], [5, 6, 7], [5, 6, 7]])
df

"""
Explanation: Notice we can specify axis=1 or axis=0, but what does that really mean? Was this ascending along rows, or ascending along columns?


To figure this out, let's use a small, simple dataframe:
End of explanation
"""


# YOUR CODE HERE

"""
Explanation: Exercise 4
Try axis=0 when using rank on this df
End of explanation
"""


df.rank(axis=0)

"""
Explanation: 
End of explanation
"""


# YOUR CODE HERE

"""
Explanation: Did that make ranks ascending along columns or along rows?
Exercise 5
Now try axis=1 when using rank on this df
End of explanation
"""


df.rank(axis=1)

"""
Explanation: 
End of explanation
"""


# YOUR CODE HERE

"""
Explanation: Did that make ranks ascending along columns or along rows?


Exercise 6
To get the gene (row) ranks for each cell (column), do we want axis=1 or axis=0? Perform .rank on the transposed expression matrix (expression_t), print the shape of the resulting ranks, and show the head() of it.
End of explanation
"""


ranks = expression_t.rank(axis=0)
print(ranks.shape)
ranks.head()

"""
Explanation: 
End of explanation
"""


# YOUR CODE HERE

# YOUR CODE HERE

# YOUR CODE HERE

"""
Explanation: Exercise 6
Now that you're armed with all this information, we'll calculate the ranks. While you're at it, let's compare the time it takes to run ("runtime") of .corr(method="pearson") on the ranks matrix vs .corr(method="spearman") on the expression matrix.

Perform pearson correlation on the ranks
Check that it is equal to the expression spearman correlation.
Use the %timeit magic to check the runtimes of .corr on the ranks and expression matrices. (Feel free to calculate the expression correlation again, below)
Note that when you use timeit, you cannot assign any variables -- using an equals sign doesn't work here.


How much time did it take, in comparison? What's the order of magnitude difference?

Use as many cells as you need.
End of explanation
"""


%timeit expression_t.corr(method='spearman')
%timeit ranks.corr(method='pearson')

ranks_corr = ranks.corr(method='pearson')
print(ranks_corr.shape)
ranks_corr.head()

"""
Explanation: 
End of explanation
"""


(ranks_corr != expression_corr).sum().sum()

"""
Explanation: Use inequality to see if any points are not the same. If this is equal to zero, then we know that they are ALL the same.
End of explanation
"""


(ranks_corr == expression_corr).sum().sum()

"""
Explanation: This is a flip of checking for equality, which is a little trickier because then you have to know exactly how many items are in the matrix. Since we have a 300x300 matrix, that multiplication is a little easier to do in your head and know that you got the right answer.
End of explanation
"""


sns.clustermap?

"""
Explanation: Make a heatmap!!
Now we are ready to make a clustered heatmap! We'll use seaborn's sns.clustermap. Let's read the documentation for sns.clustermap. What is the default distance metric and linkage method?
End of explanation
"""


# YOUR CODE HERE

"""
Explanation: Exercise 7
Now run sns.clustermap on either the ranks or expression correlation matrices, since they are equal :)
End of explanation
"""


sns.clustermap(expression_corr)

"""
Explanation: 
End of explanation
"""


# YOUR CODE HERE

"""
Explanation: How can we add the colors labeling the rows and columns? Check the documentation for sns.clustermap again:
Exercise 8
End of explanation
"""


sns.clustermap?

"""
Explanation: 
End of explanation
"""


palette = sns.color_palette('Accent', n_colors=3)
palette

"""
Explanation: Since I am not a color design expert, I defer to color design experts in choosing my color palettes. One such expert is Cynthia Brewer, who made a ColorBrewer (hah!) list of color maps for both increasing quantity (shades), and for categories (differing colors).
As a reference, I like using this demo of every ColorBrewer scale. Hover over the palette to see its name.
Thankfully, seaborn has the ColorBrewer color maps built-in. Let's see what this output is
Remember -- we never make a variable without looking at it first!!
End of explanation
"""


sns.palplot(palette)

"""
Explanation: Huh that's a bunch of weird numbers. What do they mean? Turns out it's a value from 0 to 1 representing the red, green, and blue (RGB) color channels that computers understand. But I'm not a computer .... what am I supposed to do??
Turns out, seaborn also has a very convenient function called palplot to plot the entire palette. This lets us look at the variable without having to convert from RGB
End of explanation
"""


# YOUR CODE HERE

# YOUR CODE HERE

"""
Explanation: Exercise 9

Get the color palette for the "Set2" colormap and specify that you want 6 colors (read the documentation of sns.color_palette)
Plot the color palette
End of explanation
"""


set2 = sns.color_palette('Set2', n_colors=6)
sns.palplot(set2)

"""
Explanation: 
End of explanation
"""


# YOUR CODE HERE

"""
Explanation: If you are more advanced and want access to more colormaps, I recommend checking out palettable.
Assign colors to clusters
To set a specific color to each cluster, we'll need to see the unique clusters here. For an individual column (called a "Series" in pandas-speak), how can we get only the unique items?
Exercise 10
Get the unique values from the column "cluster_celltype_with_id". Remember, always look at the variable you created!
End of explanation
"""


cluster_ids_unique = cell_metadata['cluster_celltype_with_id'].unique()
cluster_ids_unique

"""
Explanation: 
End of explanation
"""


english = ["hello", "goodbye", "no", "yes", "please", "thank you",]
spanish = ["hola", "adios", "no", "si", "por favor", "gracias"]
zip(english, spanish)

"""
Explanation: Detour: zip and dict
To map colors to each cluster name, we need to talk about some built-in functions in Python, called zip and dict
For this next part, we'll use the built-in function zip which is very useful. It acts like a zipper (like for clothes) to glue together the pairs of items in two lists:
End of explanation
"""


list(zip(english, spanish))

"""
Explanation: To be memory efficient, this doesn't show us what's inside right away. To look inside a zip object, we can use list:
End of explanation
"""


# YOUR CODE HERE

"""
Explanation: Exercise 11
What happened to  "please" and "thank you" from english? Make another list, called spanish2, that contains the Spanish words for "please" and "thank you" (again, google knows everything), then call zip on english and spanish2. Don't forget to use list on them!
End of explanation
"""


english = ["hello", "goodbye", "no", "yes", "please", "thank you",]
spanish = ["hola", "adios", "no", "si", "por favor", "gracias"]
list(zip(english, spanish))

"""
Explanation: 
End of explanation
"""


english_to_spanish = dict(zip(english, spanish))
english_to_spanish

"""
Explanation: Now we'll use a dictionary dict to make a lookup table that uses the pairing made by zip, using the first item as the "key" (what you use to look up) and the second item as the "value" (the result of the lookup)
You can think of it as a translator -- use the word in English to look up the word in Spanish.
End of explanation
"""


english_to_spanish['hello']

"""
Explanation: Now we can use English words to look up the word in Spanish! We use the square brackets and the english word we want to use, to look up the spanish word.
End of explanation
"""


# YOUR CODE HERE

"""
Explanation: Exercise 12
Make an spanish_to_english dictionary and look up the English word for "por favor"
End of explanation
"""


spanish_to_english = dict(zip(spanish, english))
spanish_to_english['por favor']

"""
Explanation: 
End of explanation
"""


# YOUR CODE HERE

"""
Explanation: Okay, detour over! Switching from linguistics back to biology :)
Exercise 13
Use dict and zip to create a variable called id_to_color that assigns labels in cluster_ids_unique to a color in set2
End of explanation
"""


id_to_color = dict(zip(cluster_ids_unique, set2))
id_to_color

"""
Explanation: 
End of explanation
"""


cell_metadata.head()

"""
Explanation: Now we want to use this id_to_color lookup table to make a long list of colors for each cell.
End of explanation
"""


celltypes = cell_metadata.celltype.unique()  # Could also use cell_metadata['celltype'].unique()
celltypes

celltype_to_color = dict(zip(celltypes, sns.color_palette('Accent', n_colors=len(celltypes))))
celltype_to_color

"""
Explanation: As an example, let's use the celltypes column to make a list of each celltype color first. Notice that we can use cell_metadata.celltype or cell_metadata['celltype'] to get the column we want.
We can only use the 'dot' notation because our column name has no unfriendly characters like spaces, dashes, or dots -- characters that mean something special in Python.
End of explanation
"""


per_cell_celltype_color = [celltype_to_color[celltype] for celltype in cell_metadata.celltype]

# Since this list is as long as our number of cells (300!), let's slice it and only look at the first 10
per_cell_celltype_color[:5]

"""
Explanation: Now we'll use the existing column cell_metadata.celltype to make a list of colors for each celltype
End of explanation
"""


# YOUR CODE HERE

"""
Explanation: Exercise 14
Make a variable called per_cell_cluster_color that uses the id_to_color dictionary to look up the color for each value in the cluster_celltype_with_id column of cluster_metadata
End of explanation
"""


per_cell_cluster_color = [id_to_color[i] for i in cell_metadata.cluster_celltype_with_id]

# Since this list is as long as our number of cells (300!), let's slice it and only look at the first 10
per_cell_cluster_color[:10]

"""
Explanation: 
End of explanation
"""


# YOUR CODE HERE

"""
Explanation: Exercise 15
Now use the cluster colors to label the rows and columns in sns.clustermap. How can
End of explanation
"""


sns.clustermap(expression_corr, row_colors=per_cell_cluster_color, col_colors=per_cell_cluster_color)

"""
Explanation: 
End of explanation
"""


combined_colors = [per_cell_cluster_color, per_cell_celltype_color]
len(combined_colors)

sns.clustermap(expression_corr, row_colors=combined_colors, col_colors=combined_colors)

"""
Explanation: We can also combine the celltype and cluster colors we created to create a double-layer colormap!
End of explanation
"""


