import numpy as np

np.array([[0,1,2], [0,0,0], [1,2,-1]])  #+1 /2 **2 .ravel() etc.

"""
Explanation: <center>
<h1>Introduction to Deep Learning with Keras</h1>
<h1>and HEP Data Exploration in Python</h1>
<h3>Michela Paganini - Yale University</h3>
<h4>High Energy Phenomenology, Experiment and Cosmology Seminar Series</h4>
<img src='http://www.edustart.org/wp-content/uploads/2014/03/MIST-Vert-Aggie-Maroon-page-001-300x200.jpg'>
</center>
This is the first of two notebooks for today's tutorial. You will find the second one <a href="deeplearning_intro.ipynb">here</a>.
What is Deep Learning?

<i>Deep learning is a particular kind of machine learning that achieves great power and ﬂexibility by learning to represent the world as a nested hierarchy of concepts, with each concept deﬁned in relation to simpler concepts, and more abstract representations computed in terms of less abstract ones.</i>

<div align="right">
  I. Goodfellow, Y. Bengio, and A. Courville, "Deep Learning", MIT Press, 2016. <br>
  http://www.deeplearningbook.org/
</div>

Representation Learning

<i>Use machine learning to discover not only the mapping from representation to output but also the representation itself. </i>

<div align="right">
  I. Goodfellow, Y. Bengio, and A. Courville, "Deep Learning", MIT Press, 2016. <br>
  http://www.deeplearningbook.org/
</div>

e.g.:
* What’s 1753 + 131?
* What’s MMXVII - MCMXCI?
Our goal: 

Learn a function that maps a set of raw electrical signals from the detector all the way to particle identification

From D. Whiteson:
<img src="./images/dwhiteson.jpg">
Deep learning tackles this by breaking down the task into a series of nested and simpler mappings, which can be described by each different layer of the model.
Deeper nets give us the flexibility to sequentially learn more and more abstract features from iterative non-linear transformations of the input data.
Higher layers of representation amplify aspects of the input that are important for the given task and suppress irrelevant variations.
Learning Recipe
In parametric, supervised learning:
- Objective: in the form of a differentiable loss function, to calculate how far your current prediction is from the target
- Model: parametrized by learnable parameters
- Training strategy: how to update the parameters
- Data: enough to learn from & test your performance

Practical Example
From <a href='http://ippog.web.cern.ch/resources/2012/cms-hep-tutorial'> CMS Software Tutorial</a>, developed by Christian Sander and Alexander Schmidt, and available on <a href='http://opendata.cern.ch/'><img src='./images/opendata.jpg' style="height:70px; display:inline;"></a>.
<img src='./images/ttbar.jpg' style="height:200px;"></a>
The samples come from a $t\bar{t}$ analysis and they are described in a very clear and accessible way in the <a href='http://ippog.web.cern.ch/sites/ippog.web.cern.ch/files/TutorialDocu_3.pdf'>documentation</a>. This was chosen to provide you with a full demo of my typical workflow, going from a ROOT file to a fully trained Keras model.
<i>Disclaimer</i>: my applications won't make 100% physical sense -- please focus on the tools! 
Before diving into the Deep Learning world, I want to spend a few minutes discussing some data handling techniques I use whenever I get started prototyping my applications. 
NumPy
<a href="http://www.numpy.org/"><img src="images/numpy.jpg"></a>
End of explanation
"""


from numpy.lib.recfunctions import stack_arrays
from root_numpy import root2array, root2rec
import glob

"""
Explanation: For a nice numpy intro, check out the CERN tutorial <a href='https://indico.cern.ch/event/487416/contributions/2174910/attachments/1280846/1902714/LYLWN_present_2016_may_slides.pdf'>Loose your Loops with NumPy</a> (and tons of online material).
<i>Idea:</i> What if those columns represented various branches and every line represented an event/physics object?
From ROOT to numpy: root_numpy
Very easy to turn your .root files into Machine Learning compliant inputs using numpy and <a href="https://github.com/scikit-hep/root_numpy/tree/4.7.3">root_numpy</a>.
<a href='http://scikit-hep.org/root_numpy/'><img src='./images/root_numpy.jpg'></a>
End of explanation
"""


ttbar = root2array('files/ttbar.root')

# -- display your newly created object
ttbar 

# -- what data type is it?
type(ttbar) 

# -- how many events are present?
ttbar.shape

# -- what are the names of the branches?
ttbar.dtype.names

"""
Explanation: Using one single function from root_numpy, you can open your .root file and turn it into an ndarray, a Python object equivalent to an n-dimensional matrix. All you need to do is to pass it the name of the file you'd like to open. Other keyword arguments are specified below.
<img src="./images/root2array.jpg">
Let's take a look at the MC signal sample from our CMS open dataset:
End of explanation
"""


import pandas as pd

"""
Explanation: Pandas Dataframes
End of explanation
"""


# -- how to turn an ndarray into a pandas dataframe
df = pd.DataFrame(ttbar)

# -- better way of displaying your data
df.head() # print the first few entries

# -- ... or the last few
df.tail()

# -- check the shape: it should be [nb_events, nb_variables]
df.shape

df.info()

df.keys() #df.columns

"""
Explanation: One way of manipulating your data (slicing, filtering, removing variables, creating new features, taking operations of branches) in a simple, visually appealing way is to use <a href="http://pandas.pydata.org/">pandas</a> dataframes, a beatiful and efficient Python data structure library. 
Recommended for exploratory data analysis, probably not for high performance applications.
End of explanation
"""


def root2pandas(files_path, tree_name, **kwargs):
    '''
    Args:
    -----
        files_path: a string like './data/*.root', for example
        tree_name: a string like 'Collection_Tree' corresponding to the name of the folder inside the root 
                   file that we want to open
        kwargs: arguments taken by root2array, such as branches to consider, start, stop, step, etc
    Returns:
    --------    
        output_panda: a pandas dataframe like allbkg_df in which all the info from the root file will be stored
    
    Note:
    -----
        if you are working with .root files that contain different branches, you might have to mask your data
        in that case, return pd.DataFrame(ss.data)
    '''
    # -- create list of .root files to process
    files = glob.glob(files_path)
    
    # -- process ntuples into rec arrays
    ss = stack_arrays([root2array(fpath, tree_name, **kwargs).view(np.recarray) for fpath in files])

    try:
        return pd.DataFrame(ss)
    except Exception:
        return pd.DataFrame(ss.data)

# -- usage of root2pandas
singletop = root2pandas('./files/single_top.root', 'events')

"""
Explanation: To summarize, if you want to go directly from .root files to pandas dataframes, you can do so in 3 lines of Python code. I like to use this function below in all my application whenever I load in data from a ROOT file. Feel free to copy it and use it!
End of explanation
"""


# -- save a pandas df to hdf5 (better to first convert it back to ndarray, to be fair)
import deepdish.io as io
io.save('ttbar.h5', df)

singletop.to_hdf('try.h5', 'branches')

# -- let's load it back in to make sure it actually worked!
new_df = io.load('ttbar.h5')
new_df.head()

# -- check the shape again -- nice check to run every time you create a df
new_df.shape

"""
Explanation: We just turned a HEP-specific ROOT file into a standard data format that can be used by any ML expert. You can now save your data out to widely accepted data formats such as HDF5 or even CSV, share it with your collaborators from the ML community without them having to learn how to use the ROOT library or other CERN-specific analysis tools.
End of explanation
"""


# slice the dataframe
jet_df = df[[key for key in df.keys() if key.startswith('Jet')]]
jet_df.head()

"""
Explanation: Now, let's create a new dataframe that contains only jet-related branches by slicing our pre-existing ttbar dataframe
End of explanation
"""


def flatten(column):
    '''
    Args:
    -----
        column: a column of a pandas df whose entries are lists (or regular entries -- in which case nothing is done)
                e.g.: my_df['some_variable'] 

    Returns:
    --------    
        flattened out version of the column. 

        For example, it will turn:
        [1791, 2719, 1891]
        [1717, 1, 0, 171, 9181, 537, 12]
        [82, 11]
        ...
        into:
        1791, 2719, 1891, 1717, 1, 0, 171, 9181, 537, 12, 82, 11, ...
    '''
    try:
        return np.array([v for e in column for v in e])
    except (TypeError, ValueError):
        return column

# -- ok, let's try it out!
df_flat = pd.DataFrame({k: flatten(c) for k, c in jet_df.iteritems()})
df_flat.head()

"""
Explanation: This would be useful if you wanted to classify your events only by using the properties of jets in each event. 
What if your application involved classifying jets, instead of events? In this case, you might want to turn your dataset from event-flat to jet-flat, i.e. a dataframe in which every row represents a jet and every column is a property of this jet. This is extremely easy to do using pandas and numpy:
End of explanation
"""


import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline

# iterate through the columns
for key in df_flat.keys():
    # plotting settings
    matplotlib.rcParams.update({'font.size': 16})
    fig = plt.figure(figsize=(5, 5), dpi=100)
    bins = np.linspace(min(df_flat[key]), max(df_flat[key]), 30)
    # plot!
    _ = plt.hist(df_flat[key], bins=bins, histtype='step', label=r'$t\overline{t}$')
    # decorate
    plt.xlabel(key)
    plt.ylabel('Number of Jets')
    plt.legend()
    plt.plot()

"""
Explanation: Plot your data using Matplotlib
Using pandas in conjunction with matplotlib, you can also inspect your variables super quickly. Check out the following cells for a quick example.
End of explanation
"""


df['Jet_P'] = (df['Jet_Px']**2 + df['Jet_Py']**2 + df['Jet_Pz']**2)**(0.5)

# -- again, you can easily slice dataframes by specifying the names of the branches you would like to select 
df[['Jet_P', 'Jet_Px', 'Jet_Py', 'Jet_Pz', 'Jet_E']].head()

# -- you can also build four vectors and store them in a new column in 1 line of code 
from rootpy.vector import LorentzVector
df['Jet_4V'] = [map(lambda args: LorentzVector(*args), zip(px, py, pz, e)) for 
                (_, (px, py, pz, e)) in df[['Jet_Px', 'Jet_Py', 'Jet_Pz', 'Jet_E']].iterrows()]

# -- look at the 4-vectors of the jets in the first 5 events
[_ for _ in df['Jet_4V'].head()]

# -- calculate the mass (or any other property) of all the jets in the first event
[jet.M() for jet in df['Jet_4V'][0]]

"""
Explanation: It's really easy and intuitive to add new columns to a dataframe. You can also define them as functions of other columns. This is great if you need to build your own hand-crafted variables.
End of explanation
"""


