# import the free sample of the dataset
from quantopian.interactive.data.estimize import estimates_free

# or if you want to import the full dataset, use:
# from quantopian.interactive.data.estimize import estimates

# import data operations
from odo import odo
# import other libraries we will use
import pandas as pd
import matplotlib.pyplot as plt

# Let's use blaze to understand the data a bit using Blaze dshape()
estimates_free.dshape

# And how many rows are in this free sample?
# N.B. we're using a Blaze function to do this, not len()
estimates_free.count()

# Let's see what the data looks like. We'll grab the first three rows.
estimates_free.head(3)

"""
Explanation: Estimize: Analyst-by-Analyst Estimates
In this notebook, we'll take a look at Estimizes's Analyst-by-Analyst Estimates dataset, available on the Quantopian. This dataset spans January, 2010 through the current day.
This data contains a record for every estimate made by an individual on the Estimize product. By comparison, the Estimize Revisions product provides rolled-up consensus numbers for each possible earnings announcement.
In this notebook, we'll examine these detailed estimates and pull in that consensus data as well.
Blaze
Before we dig into the data, we want to tell you about how  you generally access Quantopian Store data sets. These datasets are available using the Blaze library. Blaze provides the Quantopian user with a convenient interface to access very large datasets.
Blaze provides an important function for accessing these datasets. Some of these sets are many millions of records. Bringing that data directly into Quantopian Research directly just is not viable. So Blaze allows us to provide a simple querying interface and shift the burden over to the server side.
To learn more about using Blaze and generally accessing Quantopian Store data, clone this tutorial notebook.
Free samples and limits
A few key caveats:
1) We limit the number of results returned from any given expression to 10,000 to protect against runaway memory usage. To be clear, you have access to all the data server side. We are limiting the size of the responses back from Blaze.
2) There is a free version of this dataset as well as a paid one. The free one includes about three years of historical data, though not up to the current day.
With preamble in place, let's get started:
End of explanation
"""


stocks = symbols('TSLA')
one_quarter = estimates_free[(estimates_free.sid == stocks.sid) &
                 (estimates_free.fiscal_year == '2014') &
                 (estimates_free.fiscal_quarter == '1') &
                 (estimates_free.eps < 100)
                ]
one_quarter.head(5)

"""
Explanation: Let's go over the columns:
- analyst_id: the unique identifier assigned by Estimize for the person making the estimate.
- asof_date: Estimize's timestamp of event capture.
- eps: EPS estimate made by the analyst on the asof_date
- fiscal_quarter: fiscal quarter for which this estimate is made, related to fiscal_year
- fiscal_year: fiscal year for which this estimate is made, related to fiscal_quarter
- revenue: revenue estimate made by the analyst on the asof_date
- symbol: ticker symbol provided by Estimize for the company for whom these estimates have been made
- username: Estimize username of the analyst making this estimate
- timestamp: the datetime when Quantopian registered the data. For data loaded up via initial, historic loads, this timestamp is an estimate.
- sid: the equity's unique identifier. Use this instead of the symbol. Derived by Quantopian using the symbol and our market data
We've done much of the data processing for you. Fields like asof_date and sid are standardized across all our Store Datasets, so the datasets are easy to combine. We have standardized the sid across all our equity databases.
We can select columns and rows with ease. Below, let's just look at the estimates made for TSLA for a particular quarter. Also, we're filtering out some spurious data:
End of explanation
"""


one_quarter.count()

"""
Explanation: How many records do we have now?
End of explanation
"""


one_quarter.username.count_values()

"""
Explanation: Let's break it down by user:
End of explanation
"""


one_q_df = odo(one_quarter.sort('asof_date'), pd.DataFrame)
plt.plot(one_q_df.asof_date, one_q_df.eps, marker='.', linestyle='None', color='r')
plt.xlabel("As Of Date (asof_date)")
plt.ylabel("EPS Estimate")
plt.title("Analyst by Analyst EPS Estimates for TSLA")
plt.legend(["Individual Estimate"], loc=2)

"""
Explanation: Let's convert it over to a Pandas DataFrame so we can chart it and examine it closer
End of explanation
"""


from quantopian.interactive.data.estimize import revisions_free
consensus = revisions_free[(revisions_free.sid == stocks.sid) &
                 (revisions_free.fiscal_year == '2014') &
                 (revisions_free.fiscal_quarter == '1') &
                 (revisions_free.source == 'estimize') &
                 (revisions_free.metric == 'eps')
                ]

consensus.head(3)

"""
Explanation: That's neat. But let's add in some data from another dataset -- the Estimize Revisions data. For the same timeframe, the revisions data provides each revision to the overall consensus estimates. So where estimates_free data provides every single estimate made by an individual on the Estimize site, revisions_free provides rolled up summaries of the estimates.
End of explanation
"""


consensus_df = odo(consensus[['asof_date', 'mean']].sort('asof_date'), pd.DataFrame)
consensus_df.rename(columns={'mean':'eps_mean'}, inplace=True)

"""
Explanation: For this quick demonstration, let's just grab the consensus mean from the revisions_free data set and convert it over to Pandas. Note, we need to rename the mean column name because it causes problems otherwise:
End of explanation
"""


plt.plot(consensus_df.asof_date, consensus_df.eps_mean)
plt.plot(one_q_df.asof_date, one_q_df.eps, marker='.', linestyle='None', color='r')
plt.xlabel("As Of Date (asof_date)")
plt.ylabel("EPS Estimate")
plt.title("EPS Estimates for TSLA")
plt.legend(["Mean Estimate", "Individual Estimates"], loc=2)

"""
Explanation: Let's chart that in the same chart again so we get a trend of the mean over time, overlayed on a chart of each individual analyst estimate:
End of explanation
"""


