# data processing
import os
import pandas as pd, numpy as np, dask, json
import ogh
import geopandas as gpd

# data migration library
from utilities import hydroshare

# plotting and shape libraries
%matplotlib inline

import warnings
warnings.filterwarnings("ignore")

"""
Explanation: Notebook to retrieve gridded climate time-series data sets
Case study: the Sauk-Suiattle river watershed, the Elwha river watershed, the Upper Rio Salado watershed
<img src="http://www.sauk-suiattle.com/images/Elliott.jpg" 
style="float:right;width:150px;padding:20px">
Use this Jupyter Notebook to:
1. HydroShare setup and preparation
2. Re-establish the paths to the mapping file
3. Download climate data
4. Summarize the file availability from each watershed mapping file
5. Save results back into HydroShare

<br/><br/><br/>
<img src="https://www.washington.edu/brand/files/2014/09/W-Logo_Purple_Hex.png"
style="float:right;width:150px;padding:20px">
<br/><br/>
This data is compiled to digitally observe the watersheds, powered by HydroShare. <br/>Provided by the Watershed Dynamics Group, Dept. of Civil and Environmental Engineering, University of Washington
1.  HydroShare Setup and Preparation
To run this notebook, we must import several libaries. These are listed in order of 1) Python standard libraries, 2) hs_utils library provides functions for interacting with HydroShare, including resource querying, dowloading and creation, and 3) the observatory_gridded_hydromet library that is downloaded with this notebook.
End of explanation
"""


notebookdir = os.getcwd()

hs=hydroshare.hydroshare()
homedir = hs.getContentPath(os.environ["HS_RES_ID"])
os.chdir(homedir)
print('Data will be loaded from and save to:'+homedir)

# initialize ogh_meta
meta_file = dict(ogh.ogh_meta())
sorted(meta_file.keys())

"""
Explanation: Establish a secure connection with HydroShare by instantiating the hydroshare class that is defined within hs_utils. In addition to connecting with HydroShare, this command also sets and prints environment variables for several parameters that will be useful for saving work back to HydroShare.
End of explanation
"""


# map the mapping files generated for Sauk-Suiattle, Elwha, and Upper Rio Salado from usecase1
mappingfile1 = os.path.join(homedir,'Sauk_mappingfile.csv')
mappingfile2 = os.path.join(homedir,'Elwha_mappingfile.csv')
mappingfile3 = os.path.join(homedir,'RioSalado_mappingfile.csv')

t1 = ogh.mappingfileSummary(listofmappingfiles = [mappingfile1, mappingfile2, mappingfile3], 
                            listofwatershednames = ['Sauk-Suiattle river','Elwha river','Upper Rio Salado'],
                            meta_file=meta_file)

t1

"""
Explanation: 2. Re-establish the paths to the mapping file
End of explanation
"""


%%time

ogh.getDailyMET_livneh2013(homedir, mappingfile1)
ogh.getDailyMET_bcLivneh2013(homedir, mappingfile1)
ogh.getDailyMET_livneh2015(homedir, mappingfile1)
ogh.getDailyVIC_livneh2013(homedir, mappingfile1)
ogh.getDailyVIC_livneh2015(homedir, mappingfile1)
ogh.getDailyWRF_salathe2014(homedir, mappingfile1)
ogh.getDailyWRF_bcsalathe2014(homedir, mappingfile1)

"""
Explanation: 3. Download climate data
Call each get Climate data function
Each function reads in the mapping file table, generates the destination folder, downloads and unzips the files, then catalogs the downloaded files within the mapping file.
Meteorology - MET; Weather Research and Forecasting (WRF); Variable Infiltration Capacity - VIC
1. getDailyMET_livneh2013
2. getDailyMET_bcLivneh2013
3. getDailyMET_livneh2015
4. getDailyVIC_livneh2013
5. getDailyVIC_livneh2015
6. getDailyWRF_salathe2014
7. getDailyWRF_bcsalathe2014

View data extent:
1. Continental United States (CONUS)
Livneh, B. (2017). Gridded climatology locations (1/16th degree): Continental United States extent, HydroShare, http://www.hydroshare.org/resource/14f0a6619c6b45cc90d1f8cabc4129af

2. North America (NAmer)
Livneh, B. (2017). Gridded climatology locations (1/16th degree): North American extent, HydroShare, http://www.hydroshare.org/resource/ef2d82bf960144b4bfb1bae6242bcc7f

3. Pacific Northwest - Columbia River Basin
Bandaragoda, C. (2017). Sauk Suiattle HydroMeteorology (WRF), HydroShare, http://www.hydroshare.org/resource/0db969e4cfb54cb18b4e1a2014a26c82

Please cite:
1. Livneh B., E.A. Rosenberg, C. Lin, B. Nijssen, V. Mishra, K.M. Andreadis, E.P. Maurer, and D.P. Lettenmaier, 2013: A Long-Term Hydrologically Based Dataset of Land Surface Fluxes and States for the Conterminous United States: Update and Extensions, Journal of Climate, 26, 9384–9392.

2. Livneh B., T.J. Bohn, D.S. Pierce, F. Munoz-Ariola, B. Nijssen, R. Vose, D. Cayan, and L.D. Brekke, 2015: A spatially comprehensive, hydrometeorological data set for Mexico, the U.S., and southern Canada 1950-2013, Nature Scientific Data, 5:150042, doi:10.1038/sdata.2015.42.

3. Salathé, EP, AF Hamlet, CF Mass, M Stumbaugh, S-Y Lee, R Steed: 2017. Estimates of 21st Century Flood Risk in the Pacific Northwest Based on Regional Scale Climate Model Simulations.  J. Hydrometeorology. DOI: 10.1175/JHM-D-13-0137.1

Download all available datasets for Sauk-Suiattle Watershed
End of explanation
"""


%%time

ogh.getDailyMET_livneh2013(homedir, mappingfile2)
ogh.getDailyMET_bcLivneh2013(homedir, mappingfile2)
ogh.getDailyMET_livneh2015(homedir, mappingfile2)
ogh.getDailyVIC_livneh2013(homedir, mappingfile2)
ogh.getDailyVIC_livneh2015(homedir, mappingfile2)
ogh.getDailyWRF_salathe2014(homedir, mappingfile2)
ogh.getDailyWRF_bcsalathe2014(homedir, mappingfile2)

"""
Explanation: Download all available datasets for Elwha Watershed
End of explanation
"""


%%time

ogh.getDailyMET_livneh2013(homedir, mappingfile3)
ogh.getDailyMET_bcLivneh2013(homedir, mappingfile3)
ogh.getDailyMET_livneh2015(homedir, mappingfile3)
ogh.getDailyVIC_livneh2013(homedir, mappingfile3)
ogh.getDailyVIC_livneh2015(homedir, mappingfile3)
ogh.getDailyWRF_salathe2014(homedir, mappingfile3)
ogh.getDailyWRF_bcsalathe2014(homedir, mappingfile3)

"""
Explanation: Download all available datasets for Upper Rio Salado Watershed
End of explanation
"""


t1 = ogh.mappingfileSummary(listofmappingfiles = [mappingfile1, mappingfile2, mappingfile3], 
                            listofwatershednames = ['Sauk-Suiattle river','Elwha river','Upper Rio Salado'],
                            meta_file=meta_file)

t1.to_csv(os.path.join(homedir, 'watershed_table.txt'), sep='\t', header=True, index=True)

t1

"""
Explanation: 4. Summarize the file availability from each watershed mapping file
End of explanation
"""


%%time
!tar -zcf livneh2013.tar.gz livneh2013
!tar -zcf livneh2015.tar.gz livneh2015
!tar -zcf salathe2014.tar.gz salathe2014

# the downloaded tar files
climate2013_tar = os.path.join(homedir,'livneh2013.tar.gz')
climate2015_tar = os.path.join(homedir,'livneh2015.tar.gz')
wrf_tar = os.path.join(homedir,'salathe2014.tar.gz')

# for each file downloaded onto the server folder, move to a new HydroShare Generic Resource
title = 'Downloaded data sets from each study site for each of seven gridded data products'
abstract = 'This resource contains the downloaded data files for each study site and the gridded cell centroids that intersected within the study area. The file availability is described within the watershed_table file, which summarizes each of the three mapping file catalogs.'
keywords = ['Sauk', 'Elwha','Rio Salado','climate','hydromet','watershed'] 
rtype = 'genericresource'

# files to migrate
files=[mappingfile1, # sauk
       mappingfile2, # elwha
       mappingfile3, # riosalado
       watershed_table, # watershed summary table
       climate2013_tar, # Livneh et al. 2013 raw MET, bc MET, and VIC
       climate2015_tar, # Livneh et al. 2015 raw MET, and VIC
       wrf_tar] # Salathe et al. 2014 raw WRF and bc WRF

# create the new resource
resource_id = hs.createHydroShareResource(title=title, 
                                          abstract=abstract,
                                          keywords=keywords, 
                                          resource_type=rtype, 
                                          content_files=files, 
                                          public=False)

"""
Explanation: 5. Save results back into HydroShare
<a name="creation"></a>
Using the hs_utils library, the results of the Geoprocessing steps above can be saved back into HydroShare.  First, define all of the required metadata for resource creation, i.e. title, abstract, keywords, content files.  In addition, we must define the type of resource that will be created, in this case genericresource.  
Note: Make sure you save the notebook at this point, so that all notebook changes will be saved into the new HydroShare resource.
Archive the downloaded data files for collaborative use
Create list of files to save to HydroShare. Verify location and names.
End of explanation
"""


