import qtpylib
qtpylib.__version__

"""
Explanation: <img alt="QTPyLib" align="right" src="http://qtpylib.io/docs/latest/_images/qtpy-logo.png" style="margin:10px 10px 0;width:250px">
Data Workflow
QTPyLib: Quantitative Trading Python Library<br>
https://github.com/ranaroussi/qtpylib
Copyright 2016 Ran Aroussi

<code>Licensed under the GNU Lesser General Public License, v3.0 (the "License"); You may not use this file except in compliance with the License. You may obtain a copy of the License at https://www.gnu.org/licenses/lgpl-3.0.en.html
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
</code>

Overview
This Jupyter Notebook aims to provide developers with a quick overview on the new Workflow module, introduced in version 1.5.47a or QTPyLib.
The Workflow module includes some handly methods for working with external data sources when backtesting, and is planned to include methods for analyzing your backtest results in the near future.
* Please refer to the Workflow API Documentation for complete list of available methods and parametrs.

Let's start by checking out QTPyLib's version:
End of explanation
"""


from qtpylib import workflow as wf

"""
Explanation: Next, we need to import the Workflow module:
End of explanation
"""


# get data from Yahoo! finance
df = wf.get_data_yahoo_intraday("ESZ16.CME")
df.tail()

"""
Explanation: 1. Working with External Data
Sometimes, you want to backtest your strategies using market data you already have from sources other than the Blotter. Before you can use market data from external data sources, you'll need to convert it into a QTPyLib-compatible data format.
Once the data is converted, it can be read by your strategies as CSV files. You can also save the converted data in your Blotter's MySQL database so it can be used just like any other data captured by your Blotter.

1.1. Downloading Data
The Workflow module includes 4 methods for downloading market data from either Yahoo! Finance, Google or Interactive Brokers. The methods are:

get_data_yahoo() - downloads daily bars from Yahoo! finance
get_data_yahoo_intraday() - downloads 1-min intraday bars from Yahoo! finance (2 weeks max)
get_data_google_intraday() - downloads 1-min intraday bars from Google (3 weeks max)
get_data_ib() - downloads 1s or higher bars from Interactive Brokers (requires a running Blotter, connected to TWS/IB Gateway)


In this example, let's first download intraday market data for the S&P E-mini Futures from Yahoo! finance, so we'll have some data to work with...
End of explanation
"""


# read data from existing csv file
# df = pd.read_csv("~/Desktop/existing_data.csv")

"""
Explanation: Of course, if you already have your data as CSV files, simply load them into a pd.DataFrame that we will convert into QTPyLib-compatible data format in the next step.
End of explanation
"""


# prepare data for usage by QTPyLib
ibtuple = ("ES", "FUT", "GLOBEX", "USD", 201612, 0.0, "")
qtpylib_df = wf.prepare_data(ibtuple, data=df, output_path="~/Desktop/")
qtpylib_df.tail()

"""
Explanation: 1.2. Convert data into a QTPyLib-compatible data format
Once you have your existing data loaded as a pd.DataFrame, it's time to convert it into a QTPyLib-compatible data format by using the prepare_data() method and passing our data and the IB-Compatible contract tuple or string as the first argument.
End of explanation
"""


# store data
wf.store_data(qtpylib_df, kind="BAR")

"""
Explanation: The resulting CSV file will be saved in ~/Desktop/ESZ2016_FUT.csv

1.3. Storing Converted Data in MySQL
While this step is optional, you may want to load your converted data into QTPyLib's MySQL for future use by your strategies. This step isn't required as you can backtest directly off of your converted CSV files.
With your Blotter running and connected to TWS/IB Gateway, run this command:
End of explanation
"""


