# First disable the log so the output is neater
import logging, sys
logging.disable(sys.maxsize)

from findatapy.market import Market, MarketDataRequest

# In this case we are saving predefined tick tickers to disk, and then reading back
from findatapy.market.ioengine import IOEngine

md_request = MarketDataRequest(
    start_date='04 Jan 2021',
    finish_date='05 Jan 2021',
    category='fx',
    data_source='dukascopy',
    freq='tick',
    tickers=['EURUSD'],
    fields=['bid', 'ask', 'bidv', 'askv'],
    data_engine=None
)

market = Market()

df = market.fetch_market(md_request=md_request)

"""
Explanation: Using AWS S3 to read/write market data with findatapy
May 2021 - Saeed Amen - https://www.cuemacro.com - saeed@cuemacro.com
What is S3?
S3 is basically storage in the cloud, which is managed by AWS. Dump as much data as want from anywhere on the web and you don't need to worry about scaling your storage, which you'd obviously have to do in your own data centre, and also manage backups. Data is stored in S3 buckets, which are a bit like folders. Google Cloud Storage (GCS) is the equivalent on Google Cloud and Azure Blob is a similar service on Azure.
What is the cost of S3?
There are many other AWS storage services which you can find at https://aws.amazon.com/products/storage/, which are at different price levels and performance too. It is important to use the right services for storage which have the right performance cost balance for your specific use cases.
The cost of S3 depends upon factors like:

how much data you store?
which service you use (are you using S3 Standard Storage for example, or S3 Infrequent Access Storage)?
how many requests you make for the data (PUT/GET etc.)?
which region is it in?
how much data you transfer from S3 out to the internet?

An article on cloudhealthtech.com goes through the various ins and outs of the pricing at https://www.cloudhealthtech.com/blog/s3-cost-aws-cloud-storage-costs-explained. They note that the storage cost for Standard S3 (Nov 2020) is around 0.021 to 0.026 USD per month per GB. So for 1 TB that's around 21-26 USD per month, roughly under 300 USD per year. This excludes any of the various request costs for example, which you need to take into account. The actual cost of a hardware is a lot cheaper (a quick browse of 1 TB drives online, suggested a cost of around 50 USD), but if we manage our own hardware, we need to take into hassle of managing it, including stuff like backup, convenience of access etc. The cost of losing data is likely to be significant if we choose to host our data locally.
Making AWS accessible via Python
Given that S3 is in the cloud, we need to make sure that AWS services need to be accessible from Python, whether we are running our process in the cloud (which seems preferable to reduce latency) or locally. Whilst we are using Python, S3 is also accessible from many other languages.


Hence, before going through this tutorial, you'll need to go through several steps so AWS services are accessible from your machine

You'll need to create an IAM user, with appropriate permissions at https://console.aws.amazon.com/iam when you are logged into the AWS Console
In our case this will to have permissions to use S3
Get the Access key ID and secret access key for the IAM user
If you want to make S3 accessible to users outside of your AWS account, I found this explanation at https://stackoverflow.com/questions/45336781/amazon-s3-access-for-other-aws-accounts
Before changing any access rights to S3, I'd strongly recommend reading https://aws.amazon.com/s3/security/ which explains the various security mechanisms including being able to block any public access at all


Install AWS CLI
run sudo apt install awscli
or you can download the zip file
run aws configure to set the default access key ID, default AWS availability zone etc.
this will create files in ~/.aws/credentials and ~/.aws/config


AWS CLI instructions at https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html#cliv2-linux-install



Once your  credentials are set, we can use boto3, which is an SDK for Python developers to access AWS resources:

boto3 instructions https://boto3.amazonaws.com/v1/documentation/api/latest/index.html
You can install boto3 using pip
You also need to install s3fs using pip to get access to S3 via Python
If you follow the instructions at https://github.com/cuemacro/teaching/blob/master/pythoncourse/installation/installing_anaconda_and_pycharm.ipynb - you'll create a conda environment py38class which includes boto3, and many useful data science libraries, which I use for my Python teaching
You may need to install the latest version of findatapy from GitHub using pip install git+https://github.com/cuemacro/findatapy.git to run the code below.



Creating your bucket on S3
You can create your S3 bucket using AWS CLI (see https://docs.aws.amazon.com/cli/latest/reference/s3api/create-bucket.html). In the below example we create our bucket called my-bucket in the AWS region us-east-1
aws s3api create-bucket --bucket my-bucket --region us-east-1
Alternatively, you can also create it via the web GUI at https://s3.console.aws.amazon.com/s3/
Using S3 with findatapy to store tick market data from Dukascopy
In this notebook I'm going to show how to use S3 to easily store market data using findatapy. We are assuming that we have already setup AWS CLI with our credentials such as our access key ID etc.
As a first step let's download some tick data from Dukascopy for EURUSD spot, which is a free data source using findatapy. Findatapy provides a uniform wrapper to download from many different data sources. We can predefine ticker mappings from our own nicknames for tickers to the vendor tickers. It already comes out of the box, with Dukascopy ticker mappings predefined, but these are all customisable. Note, that we haven't used the data_engine property. If this isn't set, then findatapy will download from our data source directly.
End of explanation
"""


print(df)

"""
Explanation: Let's print the output...
End of explanation
"""


folder = 's3://type-here'

"""
Explanation: Let's type in our S3 bucket address, which you'll need to change below. Note the use of s3:// at the start of the expression.
End of explanation
"""


IOEngine().write_time_series_cache_to_disk(folder, df, engine='parquet', md_request=md_request)

"""
Explanation: We can write our tick data DataFrame in Parquet format. We can give it the MarketDataRequest we used for fetching the data, which basically creates the filename in the format of environment.category.data_source.freq.tickers for high frequency data or in the format of environment.category.data_source.freq for daily data. This will enable us to more easily fetch the data using the same MarketDataRequest interface.
In this case, the filename of the Parquet file is:

s3://bla_bla_bla/backtest.fx.tick.dukascopy.NYC.EURUSD.parquet
ie. the environment of our data is backtest
the category is fx
the data_source is dukascopy
the freq is tick
the cut (or time of close) is NYC
the tickers is EURUSD

The Jupyter notebook market_data_example.ipynb explains in more detail this ticker format and the concept of a MarketDataRequest. We dump it disk using the IOEngine class. Note that the write_time_series_cache_to_disk and read_time_series_from_disk reads/writes from S3 in exactly the same way as we would do locally. We need to make sure that when we're writing to disk, we have a data licence to do so (and this will clearly vary between data vendors), and in particular, that only those who read from the disk are authorised to use that data.
End of explanation
"""


s3_filename = folder + '/backtest.fx.dukascopy.tick.NYC.EURUSD.parquet' 
df = IOEngine().read_time_series_cache_from_disk(s3_filename, engine='parquet')

print(df)

"""
Explanation: We could fetch the data directly using the S3 filename ie.
End of explanation
"""


md_request.data_engine = folder + '/*.parquet'

df = market.fetch_market(md_request)

print(df)

"""
Explanation: But it is more convenient to simply use the MarketDataRequest object we populated earlier. But in order to make it fetch from S3 instead of Dukascopy, we just need to set the data_engine property to give it the path of the S3 bucket and the postfix /*.parquet.
End of explanation
"""


import os
from findatapy.market import Market, MarketDataRequest

# In this case we are saving predefined tick tickers to disk, and then reading back
from findatapy.market.ioengine import IOEngine

# Change this to your own Quandl API key
quandl_api_key = os.environ['QUANDL_API_KEY']

md_request = market.create_md_request_from_str(md_request_str='fx.quandl.daily.NYC', 
    md_request=MarketDataRequest(start_date='01 Jan 2021', finish_date='27 May 2021', quandl_api_key=quandl_api_key))

"""
Explanation: It should be noted there are many other ways to dump and read Parquet files from S3. We can use pandas.read_parquet to directly read Parquet files from S3. Libraries like Dask also support reading Parquet directly from S3 too. I'd also checkout AWS Data Wrangler, which makes it easier to use Pandas with many AWS services including S3, at https://github.com/awslabs/aws-data-wrangler
Using S3 with findatapy to store daily market data from Quandl
In this case we are downloading all G10 FX crosses from Quandl, which are predefined as fx.quandl.daily.NYC where our 
* category is fx
* data_source is quandl
* freq is daily
* cut is NYC
Unlike in the previous where we specified the MarketDataRequest in full, here we just use the above string as shorthand, and we set the quandl_api_key using the MarketDataRequest and also the start_date. If we have Redis running locally (which is an in memory cache), this DataFrame will also be cached in our Redis instance. We'll show how to take advantage of this cache later. If Redis is not installed, it's not a big deal, it just means the cache won't be operational. Hence, doing repeated MarketDataRequest calls will end up taking longer, as findatapy will seek to get the data from the data vendor directly.
End of explanation
"""


print(md_request)

"""
Explanation: We can print out the MarketDataRequest we just constructed. We should be able to see there quandl for the data_source and fx for the category, as well as the start_date (realise it's difficult!)
End of explanation
"""


df = market.fetch_market(md_request)

print(df)

"""
Explanation: We can now fetch the market data from quandl.
End of explanation
"""


IOEngine().write_time_series_cache_to_disk(folder, df, engine='parquet', md_request=md_request)

"""
Explanation: Let's write this to S3.
End of explanation
"""


df = market.fetch_market(md_request_str='fx.quandl.daily.NYC', 
                         md_request=MarketDataRequest(start_date='01 Jan 2021', finish_date='27 May 2021',
                                                      quandl_api_key=quandl_api_key,
                                                     data_engine=folder + '/*.parquet'))

print(df)

"""
Explanation: And we can read it back using a similar call to before, except this time we set the data_engine property of the MarketDataRequest.
End of explanation
"""


df = market.fetch_market(md_request_str='fx.quandl.daily.NYC', 
                         md_request=MarketDataRequest(start_date='01 Jan 2021', finish_date='27 May 2021',
                                                      quandl_api_key=quandl_api_key,
                                                      cache_algo='cache_algo_return'))

print(df)

"""
Explanation: If we set the cache_algo property to cache_algo_return and remove the data_engine parameter (and if we have had Redis running, and make exactly the same data requet call (same assets and dates), findatapy will look in the Redis cache locally to fetch the data. This is significantly quicker at around 20ms versus over 500ms for fetching from S3.
End of explanation
"""


