who

"""
Explanation: <div align="right">Python 3.6</div>

Testing The Abstract Base Class - Earlier Version of the Code
This notebook was created because it is easier to test out the core logic separately from the logic that makes web API calls and extracts data from Google Maps.  The intent was to test and debug as much of the logic (and/or similar logic) as might be needed ahead of bringing it all together in the final subclass.
Note that code and test results may differ from the final implementation.  Some issues with the code were corrected, enhanced, and improved during testing of the Google Maps interacting subclass (a different notebook).
Enrich or Change Larger Dataframe Section by Section
The purpose of the <font color=blue><b>DFBuilder</b></font> object is to allow scanning of a larger dataframe, a small number of rows at a time.  It then allows code to be customized to make changes and build up a new dataframe from the results.  The operation is in
a standard loop by design. The original use case was to add a field with data accessed from an API off the web, and time delays were necessary (as well as other logic) to prevent (or at least reduce the risk of) server timeouts during operation.
Scanning through the source a few lines at a time, performing the operation and adding back out to the target DF
creates a "caching effect" where data is saved along the way so in the event of a server time-out all is not lost.  The resulting DF can then be saved out to a file, code modified, and a re-run of the code can pick up the process where it left off instead of having to start over again.
These tests use a subclass that will never be used in the real world and which does not communicate with the web.  The goal of these tests were to shake out problems with the other logic ahead of testing with web interaction.
End of explanation
"""


import pandas as pd
import time

## this entire cell may not be needed for this test but will be needed for the next test notebook of final objects
import os

## for larger data and/or make many requests in one day - get Google API key and use these lines:
# os.environ["GOOGLE_API_KEY"] = "YOUR_GOOGLE_API_Key"
## for better security (PROD environments) - install key to server and use just this line to load it:
# os.environ.get('GOOGLE_API_KEY')

# set up geocode
from geopy.geocoders import Nominatim
geolocator = Nominatim()
from geopy.exc import GeocoderTimedOut


"""
Explanation: Libraries Needed
Import statements included in this notebook are for the main abstract object and a test object.
End of explanation
"""


## Test code on a reasonably small DF
tst_lat_lon_df = pd.read_csv("testset_unique_lat_and_lon_vals.csv", index_col=0)

tst_lat_lon_df.describe()

tst_lat_lon_df.tail()

## Create smaller random sample from above DF for further testing

tst_lat_lon_df_sample = tst_lat_lon_df.sample(frac=0.1).copy(deep=True)
    # frac=0.1 for 10% or use n=100 for get 100 records
    # this variant seemed to create trouble with indexing of the DF in buildOutDF():
    #      tst_lat_lon_df.copy(deep=True).sample(frac=0.1)
    # also: options on reset_index given in next cell were needed as part of the fix
len(tst_lat_lon_df_sample)

tst_lat_lon_df_sample.reset_index(drop=True, inplace=True)  
tst_lat_lon_df_sample.iloc[[24,25,67]]  ## attempt to fix index and show 3 rows that will be manipulated for testing

# sample: sub_df.iloc[0]['A']
# creating some missing values for testing of error handling in the code
# note:  tried tst_lat_lon_df_sample.iloc[67]['lat'] = "" but it seems the pandas dataframe "protects itself"
#        the change failed to occur unless setting the numeric field to None
#        see notes on roundValue() function later in this document for similar pandas related behaviors

tst_lat_lon_df_sample.iloc[67]['lat'] = None
tst_lat_lon_df_sample.iloc[67]['lon'] = None
tst_lat_lon_df_sample.iloc[24]['lat'] = None
tst_lat_lon_df_sample.iloc[25]['lon'] = None
tst_lat_lon_df_sample.iloc[[24,25,67]]

"""
Explanation: Test Data
Input Data Set up Here
End of explanation
"""


who

from abc import ABCMeta, abstractmethod
import pandas as pd

class DFBuilder(object, metaclass=ABCMeta):       # sets up abstract class
    def __init__(self,endRw,time_delay):          # abstract classes can be subclassed
        self.endRow=endRw                         # but cannot be instantiated
        self.delay=time_delay
        self.tmpDF=pd.DataFrame()   # temp DF will be endRow rows in length
        self.outDF=pd.DataFrame()   # final DF build in sets of endRow rows so all is not lost in a failure
        self.lastIndex = None
        # self.start=0
        
    def __str__(self):
        return ("Global Settings for this object: \n" +  
               "endRow: " + str(self.endRow) + "\n" + 
               "delay:  " + str(self.delay) + "\n" + 
               "Length of outDF: " + str(len(self.outDF)) + "\n" +
               "nextIndex: " + str(self.lastIndex))       # if continuing with last added table - index of next rec.
        
    @abstractmethod                               # abstract method definition in Python
    def _modifyTempDF_(): pass                    # This method will operate on TempDF inside the loop
    
    def buildOutDF(self, inputDF):
        '''Scans inputDF using self.endRow rows (default of 5) at a time to do it.  It then calls in logic
from _modifyTempDF()_ to make changes to each subset of rows and appends tiny tempDF onto an outDF.  When the 
subclass is using a web API, self.time_delay tells it how much time to delay each iteration of the loop.  All
parameters are set during initialization of the object.  Should this function fail in the middle, outDF will
have all work up to the failure.  This can be saved out to a DF or csv.  The function can be run again on
a subset of the data (the records not encountered yet before the failure).'''
    
        lenDF = len(inputDF)
        print("Processing inputDF of length: ", lenDF)
        endIndx = 0

        i = 0
        while i < lenDF:
            # print("i: ", i)
            endIndx = i + self.endRow
            if endIndx > lenDF:
                        endIndx = lenDF

            # print("Range to use: ", i, ":", endIndx)
            self.tmpDF = inputDF[i:endIndx].copy(deep=True)
            self._modifyTempDF_()
            time.sleep(self.delay)
            self.outDF = self.outDF.append(self.tmpDF) 
            self.lastIndex = endIndx            
            i = endIndx
            # print("i at end of loop: ", i)        
          
        self.reindex_OutDF()
        
    def reindex_OutDF(self):
        self.outDF.reset_index(drop=True, inplace=True)

"""
Explanation: Code Testing
The abstract class which follows is intended to be the "work horse" of this code.  Intent is that it gets the developer to the point where all they need to think about is what their final subclass will do to enrich the data.  The parent class sets up a loop that can extract from a larger input DF, a small number of rows to be operated on in a temp DF and then be added to an outputDF.  In the event of something interrupting the process (a common event when dealing with web APIs), modified rows created before the incident are waiting in output DF and can be extracted. Then code can be restarted or continued to allow building up the rest of the Dataframe without losing previous work or having to go all the way back to the beginnin.
This test notebook sets up a subclass that will never be used in the real world.  There are more efficient ways to modify a DF with the example selected for this test.  The test's intent is simply to show that most of the core logic works before we test a subclass that is slower and more involved because it actually makes calls to a web API.
End of explanation
"""


class TstModification_DFBuilder(DFBuilder): 
    '''Test of ability to scan a dataframe x rows at a time and add data columns to it.
There are more efficient ways to round cols in a DF; this object is a test of base logic from the abstract class
ahead of creating a more complex subclass that interacts with the web during the loop.  It builds a copy of the 
DF a small number of rows at a time and creates some new fields as it does so.  Input DF must have "lat" and 
"lon" cols. lat=Latitude / lon = Longitude. Defaults set delay to 0 seconds and rows processed at a time to 
5 for this test.'''
    
    def __init__(self, endRw=5,time_delay=0):           
        super().__init__(endRw,time_delay)
        
    def roundValue(self, value, dec_places=4, rtn_null=False):
        '''Takes arguments: value, dec_places. Rounds value to dec_places specified (if not specified, default=4.)
rtn_null defaults to False.  If True, error handling should result in an empty string being returned.   
If false __ErrType__ should be returned to help with debugging code and data by distinguishing why there is  
no rounded answer returned.  Testing shows that while using round() throws errors if input is not a number, 
applying it to a dataframe does not.  Try-except code left in for future research but does not seem to ever 
get triggered as of this writing.'''
        
        try:
            rtnVal = round(value, dec_places)
        except TypeError as terr:
            print(type(terr))
            print(terr)
            rtnVal = "__NAN__"
        except Exception as eerr:
            print(type(eerr))
            print(eerr)
            rtnVal = "__ERR__"
        finally:
            if rtn_null==True:
                if isinstance(rtnVal, str):
                    return ""
                elif rtnVal is None:
                    return ""
                else:
                    return rtnVal
            else: 
                return rtnVal
        
    def _modifyTempDF_(self, dec_places=4, rtn_null=False):
        '''Create rounded lat and lon columns adding them to tempDF. Defaults round to 4 places and return error
strings in the column if the input value is not a number and cannot be rounded.  Note: error handling for 
roundValue() may never come into play due to interaction of apply/lambda/roundValue with DataFrames.  Attempts
to test this on a dataframe resulted in a dataframe with NaNs in it instead of error text.'''
        self.tmpDF["lat_rnd"] = self.tmpDF.apply(lambda x: self.roundValue(x.lat, dec_places, rtn_null), axis=1)
        self.tmpDF["lon_rnd"] = self.tmpDF.apply(lambda x: self.roundValue(x.lon, dec_places, rtn_null), axis=1)

"""
Explanation: Test Sub Class
Before creating, testing, and debugging a subclass that uses a Google maps API to enrich the data, it is desirable to start simpler.  This test object makes use of all of the same logic except for the API calls out to the web to get new data.  For this test, the "delay" will be set to 0 since it is not needed.  This code just shows that we can loop through an original DF, copy rows 5 at a time to a temporary DF, add columns to them using logic that looks at existing rows, and output to our output DF.
Stopping the code in the middle may allow us to test what happens if the code halts, showing what is stored in the dataframe in the object when this happens.
End of explanation
"""


# del loopDFTst

# del tstDFbldr

who

loopDFTst = TstModification_DFBuilder()

print(loopDFTst)

loopDFTst.buildOutDF(tst_lat_lon_df)

print("Length of outDF: ", len(loopDFTst.outDF))
loopDFTst.outDF.tail()

tst_lat_lon_df.iloc[[0,1,2,1157,1158,1159]]  # looking at start and end of source data

loopDFTst.outDF.iloc[[0,1,2,1157,1158,1159]] # comparing to output rows

# reset the looptest object and check it again with two slices of original DF
del loopDFTst

loopDFTst = TstModification_DFBuilder()
print(loopDFTst)

loopDFTst.buildOutDF(tst_lat_lon_df[0:98])  # use number that does not divide evenly by 5 (endRow=5)
print(loopDFTst)

print("Length of outDF: ", len(loopDFTst.outDF))
loopDFTst.outDF.tail()

loopDFTst.buildOutDF(tst_lat_lon_df[98:100])  # test: missed few records scenario (in this case, adding just 2)

print("Length of outDF: ", len(loopDFTst.outDF))
loopDFTst.outDF.tail()

loopDFTst.buildOutDF(tst_lat_lon_df[100:105])  # test: add amount equal to internal self.endRow variable

print("Length of outDF: ", len(loopDFTst.outDF))
loopDFTst.outDF.tail()

loopDFTst.buildOutDF(tst_lat_lon_df[105:205])  # test: add 100 rows (which is divisble by 5)

print("Length of outDF: ", len(loopDFTst.outDF))
loopDFTst.outDF.tail()

loopDFTst.buildOutDF(tst_lat_lon_df[205:306])  # another add records test

print("Length of outDF: ", len(loopDFTst.outDF))
loopDFTst.outDF.tail()

loopDFTst.buildOutDF(tst_lat_lon_df[306:])  # get the rest added

print("Length of outDF: ", len(loopDFTst.outDF))
loopDFTst.outDF.tail()

## sanity checking:
loopDFTst.outDF.iloc[[0,1,2,1157,1158,1159]]

tst_lat_lon_df.iloc[[0,1,2,1157,1158,1159]]

"""
Explanation: Test Loop Math Issue
This section was created to debug an issue with the loop.  Successful runs of these these tests show this problem was corrected.
End of explanation
"""


tstDFbldr = TstModification_DFBuilder()
print(tstDFbldr)  ## show defaults set during object build

tst_lat_lon_df.tail()

tstDFbldr.buildOutDF(tst_lat_lon_df)  ## executes in under 1 second on 1160 rows

tstDFbldr.outDF.describe()            ## use .describe instead of .describe() to see source data ... rounding did work
                                      ## display adds zeros, but our rounded fields are rounded to 4 decimals

tstDFbldr.outDF.tail()  ## tail of DF after first run of the function

"""
Explanation: Simple Test Using Defaults
End of explanation
"""


tstDFbldr.buildOutDF(tst_lat_lon_df[0:5])  # add copy of 5 rows to the end (like adding more data later)
tstDFbldr.outDF.tail(10)                   # function added these new rows and fixed the index
                                           # this test is why the reset_index code was added

tstDFbldr.outDF.head(10)  ## quick check of the head of the DF

# tstDFbldr.outDF  ## uncomment to view whole DF

"""
Explanation: Add 5 Rows - Original Data Source
Using buildOutDF() to add to outDF inside the object.  This test will repeat the first 5 rows on the end of the DF
End of explanation
"""


tstDFbldr2 = TstModification_DFBuilder()  
tstDFbldr2.buildOutDF(tst_lat_lon_df_sample)  ## create another object and run the function on it

tstDFbldr2.outDF.iloc[[24,25,67]]             ## NaN produced from empty Lat/Lon values

# reset object by replacing with fresh blank one
tstDFbldr2 = TstModification_DFBuilder(time_delay=1)  ## set delay to 1 second so we can interrupt during processing

"""
Explanation: Repeat Testing With Random Sample of Data
Test done again on smaller random sample of original data created as deep copy.  If test is run on whole DF, it will complete in 1/10th the time.  Testing, however, revealed sone odd quirks in just spot testing the coding features.
End of explanation
"""


tstDFbldr2.buildOutDF(tst_lat_lon_df_sample)          ## stop this test in middle for next set of tests

print(tstDFbldr2.delay  )    ## show delay used:  1 second
tstDFbldr2.outDF.describe()  ## describe resulting DF ... it has only a fraction of the expected rows
                             ## because we stopped the code early during testing ...

tstDFbldr2.outDF.tail()  ## index values shown here were cleaned up by changing creation of sample DF
                         ## see comments in data preparation at start of NB

## work-around:  this code could be used to reset index with same options as buildOutDF()
# tstDFbldr2.reindex_OutDF()
# tstDFbldr2.outDF.tail()

tstDFbldr2.buildOutDF(tst_lat_lon_df_sample[-5:])   ## first attempt to add last 5 rows again from sample data

tstDFbldr2.outDF.describe()  # count was unchanged from previous in earlier iteration of the code

tstDFbldr2.outDF.tail(10)  # now it seems to work right

tstDFbldr2.buildOutDF(tst_lat_lon_df_sample[-5:])   ## Try it a second time

tstDFbldr2.outDF.describe()                         ## note: count is 5 more than before

tstDFbldr2.outDF.tail(10)    ## comparison of tail helps confirm new records were added
                             ## but clean index reset occurs this time (as it is supposed to)

## Another investigation
'''Idea: If in doubt as to whether the dataframe being passed in for the second run is mutating correctly or not,
try making a deep copy in steps and resetting the index on the copy as shown here. '''

## problems this was investigating now appear to be fixed

tmpDF1 = tst_lat_lon_df_sample[-5:].copy(deep=True)
tmpDF1.reset_index(drop=True, inplace=True)

tmpDF1

tstDFbldr2.buildOutDF(tmpDF1)  ## initial test seems promising

tstDFbldr2.outDF.tail(10)     ##  as shown here, multiple tests seem to add the new rows every time

tstDFbldr2.buildOutDF(tmpDF1)
tstDFbldr2.outDF.tail(10)

print(tstDFbldr2)  ## note: we added tmpDF1 which ended on index 4.  This is why "nextIndex" now reads 5
                   ## nextIndex represents next index if we were to cotinue with the next record in the last
                   ## table we added to outDF using the buildOutDF() function

"""
Explanation: Simulated Interrupt
In the final object, it will be server timeouts from the web that may result in coding failing to complete.  The closest we can come to simulating this without introducing web API content is to set a time delay and interrupt the code run in the middle (manually) from within Jupyter.  That test as well as some other logic checks follows here.
End of explanation
"""


## to illustrate:  we try creating a fresh object to see if we can show that problem in this NB
## build 3 here ...

tstDFbld3 = TstModification_DFBuilder(time_delay=1)
tstDFbld3.buildOutDF(tst_lat_lon_df_sample)          ## stop this test in middle for next set of tests

print(tstDFbld3.delay)      ## show delay used:  1 second
tstDFbld3.outDF.describe()  ## describe resulting DF ... it has only a fraction of the expected rows
                             ## because we stopped the code early during testing ...

tstDFbld3.outDF.tail()

tmpDF2 = tst_lat_lon_df_sample[-5:].copy(deep=True)
tmpDF2.reset_index(drop=True, inplace=True)
tmpDF2

tstDFbld3.buildOutDF(tmpDF2)   # starting with fresh object and fresh deep copy of the sample
tstDFbld3.outDF.tail()         # the problem recurs
                               # first attempt fails

tstDFbld3.buildOutDF(tmpDF2) 
tstDFbld3.outDF.tail()          # second and subsequent attempts succeed

tstDFbld3.buildOutDF(tmpDF2) 
tstDFbld3.outDF.tail()

"""
Explanation: Attempt To Replicate Earlier Problem
This problem was originally created in another Notebook without all the tests before it.  Strangely, removing some of the tests that preceded the one that was expected to work caused it to fail in an initial run of this notebook.  Code has since changed and these tests now show that the content works as expected (problem solved).
End of explanation
"""


# create new object to test the docstrings
testObj1 = TstModification_DFBuilder()

help(testObj1)

print(testObj1.__doc__)  # note: formatting is messed up if you do not use print() on the doc string

print(testObj1.buildOutDF.__doc__) # buildOutDF

"""
Explanation: Documentation Tests
End of explanation
"""


