src/scrape_fun.py
def hist_option_data(stock_of_interest, option_data_path, history_df):
    """
    This function aims to aggregate and adjust all option data for a given ticker.
    Adjustment is made based on historical splits. For example, had a stock undergone
    a forward split of factor 2, all previous strike price, ask/bid price ...etc are
    divided by 2. On the flip side, ask/bid size, volume ...etc are multiplied by 2.
    Function has to be modified depending on structure of options data.

    :param stock_of_interest: ticker symbol (string)
    :param option_data_path: path where all the option data files are stored (string)
    :param history_df: historical end of day prices for ticker (DataFrame)
    :return: options_df: all ticker specific options data (DataFrame)
    """
    options_df = pd.DataFrame()

    # Nested loading of options data
    my_years = [year for year in os.listdir(option_data_path) if not year.startswith(".")]
    for year in tqdm(my_years, desc="year"):
        my_months = [month for month in os.listdir(os.path.join(option_data_path, year)) if not month.startswith(".")]
        for month in tqdm(my_months, desc="month"):
            my_days = [day for day in os.listdir(os.path.join(option_data_path, year, month)) if
                       not day.startswith(".")]
            for day in tqdm(my_days, desc="day"):
                try:
                    daily_option_data = pd.read_csv(os.path.abspath(os.path.join(option_data_path, year, month, day)))
                except FileNotFoundError:
                    raise SystemExit("Option data for " + stock_of_interest + " not found in path:" +
                                     os.path.abspath(os.path.join(option_data_path, year, month, day)))
                # Filtering for the right symbol
                temp_data = daily_option_data[daily_option_data["Symbol"] == stock_of_interest]
                # Dropping columns are reordering others
                temp_data = temp_data.drop(columns=["optionkey", "Symbol", "UnderlyingPrice"])[[
                    "DataDate", "ExpirationDate", "PutCall", "StrikePrice", "AskPrice",
                    "AskSize", "BidPrice", "BidSize", "LastPrice", "Volume", "openinterest"]]
                # Change to datetime
                temp_data["DataDate"] = pd.to_datetime(temp_data["DataDate"]).dt.date
                temp_data["ExpirationDate"] = pd.to_datetime(temp_data["ExpirationDate"]).dt.date
                if len(np.unique(temp_data["DataDate"])) == 1:
                    temp_day = np.unique(temp_data["DataDate"])[0]
                else:
                    raise SystemExit("More than one unique day in each file! Discount Options Data seriously bugged :/")
                # Retrieve adjustment factor and closing price
                temp_df = history_df[history_df["date"] == temp_day][["adjustment factor", "close"]]
                temp_adjustment_factor = float(temp_df["adjustment factor"])
                temp_closing_price = float(temp_df["close"])
                # Adjusting option data as needed
                temp_data[["StrikePrice", "AskPrice", "BidPrice", "LastPrice"]] = \
                    temp_data[[
                        "StrikePrice", "AskPrice", "BidPrice", "LastPrice"]] / temp_adjustment_factor
                temp_data[["AskSize", "BidSize", "Volume", "openinterest"]] = \
                    temp_data[["AskSize", "BidSize", "Volume", "openinterest"]] * temp_adjustment_factor
                temp_data["closing price"] = temp_closing_price
                # Append to DataFrame
                options_df = options_df.append(temp_data).reset_index(drop=True)

    # Renaming columns
    options_df = options_df.rename(columns={"DataDate": "date",
                                            "ExpirationDate": "expiration date",
                                            "PutCall": "type",
                                            "StrikePrice": "strike price",
                                            "AskPrice": "ask price",
                                            "AskSize": "ask size",
                                            "BidPrice": "bid price",
                                            "BidSize": "bid size",
                                            "LastPrice": "last price",
                                            "openinterest": "open interest",
                                            "Volume": "volume"})

    options_df = options_df.sort_values(by=["date", "expiration date", "strike price"]).reset_index(drop=True)
    return options_df

    This function aims to aggregate and adjust all option data for a given ticker.
    Adjustment is made based on historical splits. For example, had a stock undergone
    a forward split of factor 2, all previous strike price, ask/bid price ...etc are
    divided by 2. On the flip side, ask/bid size, volume ...etc are multiplied by 2.
    Function has to be modified depending on structure of options data.

    :param stock_of_interest: ticker symbol (string)
    :param option_data_path: path where all the option data files are stored (string)
    :param history_df: historical end of day prices for ticker (DataFrame)
    :return: options_df: all ticker specific options data (DataFrame)
    
This function aims to aggregate and adjust all option data for a given ticker.
Adjustment is made based on historical splits. For example, had a stock undergone
a forward split of factor 2, all previous strike price, ask/bid price ...etc are
divided by 2. On the flip side, ask/bid size, volume ...etc are multiplied by 2.
Function has to be modified depending on structure of options data.
This function aims to aggregate and adjust all option data for a given ticker . Adjustment is made based on historical splits . For example had a stock undergone a forward split of factor 2 all previous strike price ask / bid price ... etc are divided by 2 . On the flip side ask / bid size volume ... etc are multiplied by 2 . Function has to be modified depending on structure of options data .
def hist_option_data(stock_of_interest, option_data_path, history_df):
    options_df = pd.DataFrame()
    my_years = [year for year in os.listdir(option_data_path) if not year.startswith(".")]
    for year in tqdm(my_years, desc="year"):
        my_months = [month for month in os.listdir(os.path.join(option_data_path, year)) if not month.startswith(".")]
        for month in tqdm(my_months, desc="month"):
            my_days = [day for day in os.listdir(os.path.join(option_data_path, year, month)) if
                       not day.startswith(".")]
            for day in tqdm(my_days, desc="day"):
                try:
                    daily_option_data = pd.read_csv(os.path.abspath(os.path.join(option_data_path, year, month, day)))
                except FileNotFoundError:
                    raise SystemExit("Option data for " + stock_of_interest + " not found in path:" +
                                     os.path.abspath(os.path.join(option_data_path, year, month, day)))
                temp_data = daily_option_data[daily_option_data["Symbol"] == stock_of_interest]
                temp_data = temp_data.drop(columns=["optionkey", "Symbol", "UnderlyingPrice"])[[
                    "DataDate", "ExpirationDate", "PutCall", "StrikePrice", "AskPrice",
                    "AskSize", "BidPrice", "BidSize", "LastPrice", "Volume", "openinterest"]]
                temp_data["DataDate"] = pd.to_datetime(temp_data["DataDate"]).dt.date
                temp_data["ExpirationDate"] = pd.to_datetime(temp_data["ExpirationDate"]).dt.date
                if len(np.unique(temp_data["DataDate"])) == 1:
                    temp_day = np.unique(temp_data["DataDate"])[0]
                else:
                    raise SystemExit("More than one unique day in each file! Discount Options Data seriously bugged :/")
                temp_df = history_df[history_df["date"] == temp_day][["adjustment factor", "close"]]
                temp_adjustment_factor = float(temp_df["adjustment factor"])
                temp_closing_price = float(temp_df["close"])
                temp_data[["StrikePrice", "AskPrice", "BidPrice", "LastPrice"]] = \
                    temp_data[[
                        "StrikePrice", "AskPrice", "BidPrice", "LastPrice"]] / temp_adjustment_factor
                temp_data[["AskSize", "BidSize", "Volume", "openinterest"]] = \
                    temp_data[["AskSize", "BidSize", "Volume", "openinterest"]] * temp_adjustment_factor
                temp_data["closing price"] = temp_closing_price
                options_df = options_df.append(temp_data).reset_index(drop=True)
    options_df = options_df.rename(columns={"DataDate": "date",
                                            "ExpirationDate": "expiration date",
                                            "PutCall": "type",
                                            "StrikePrice": "strike price",
                                            "AskPrice": "ask price",
                                            "AskSize": "ask size",
                                            "BidPrice": "bid price",
                                            "BidSize": "bid size",
                                            "LastPrice": "last price",
                                            "openinterest": "open interest",
                                            "Volume": "volume"})
    options_df = options_df.sort_values(by=["date", "expiration date", "strike price"]).reset_index(drop=True)
    return options_df
def hist_option_data ( stock_of_interest , option_data_path , history_df ) : options_df = pd . DataFrame ( ) my_years = [ year for year in os . listdir ( option_data_path ) if not year . startswith ( "." ) ] for year in tqdm ( my_years , desc = "year" ) : my_months = [ month for month in os . listdir ( os . path . join ( option_data_path , year ) ) if not month . startswith ( "." ) ] for month in tqdm ( my_months , desc = "month" ) : my_days = [ day for day in os . listdir ( os . path . join ( option_data_path , year , month ) ) if not day . startswith ( "." ) ] for day in tqdm ( my_days , desc = "day" ) : try : daily_option_data = pd . read_csv ( os . path . abspath ( os . path . join ( option_data_path , year , month , day ) ) ) except FileNotFoundError : raise SystemExit ( "Option data for " + stock_of_interest + " not found in path:" + os . path . abspath ( os . path . join ( option_data_path , year , month , day ) ) ) temp_data = daily_option_data [ daily_option_data [ "Symbol" ] == stock_of_interest ] temp_data = temp_data . drop ( columns = [ "optionkey" , "Symbol" , "UnderlyingPrice" ] ) [ [ "DataDate" , "ExpirationDate" , "PutCall" , "StrikePrice" , "AskPrice" , "AskSize" , "BidPrice" , "BidSize" , "LastPrice" , "Volume" , "openinterest" ] ] temp_data [ "DataDate" ] = pd . to_datetime ( temp_data [ "DataDate" ] ) . dt . date temp_data [ "ExpirationDate" ] = pd . to_datetime ( temp_data [ "ExpirationDate" ] ) . dt . date if len ( np . unique ( temp_data [ "DataDate" ] ) ) == 1 : temp_day = np . unique ( temp_data [ "DataDate" ] ) [ 0 ] else : raise SystemExit ( "More than one unique day in each file! Discount Options Data seriously bugged :/" ) temp_df = history_df [ history_df [ "date" ] == temp_day ] [ [ "adjustment factor" , "close" ] ] temp_adjustment_factor = float ( temp_df [ "adjustment factor" ] ) temp_closing_price = float ( temp_df [ "close" ] ) temp_data [ [ "StrikePrice" , "AskPrice" , "BidPrice" , "LastPrice" ] ] = temp_data [ [ "StrikePrice" , "AskPrice" , "BidPrice" , "LastPrice" ] ] / temp_adjustment_factor temp_data [ [ "AskSize" , "BidSize" , "Volume" , "openinterest" ] ] = temp_data [ [ "AskSize" , "BidSize" , "Volume" , "openinterest" ] ] * temp_adjustment_factor temp_data [ "closing price" ] = temp_closing_price options_df = options_df . append ( temp_data ) . reset_index ( drop = True ) options_df = options_df . rename ( columns = { "DataDate" : "date" , "ExpirationDate" : "expiration date" , "PutCall" : "type" , "StrikePrice" : "strike price" , "AskPrice" : "ask price" , "AskSize" : "ask size" , "BidPrice" : "bid price" , "BidSize" : "bid size" , "LastPrice" : "last price" , "openinterest" : "open interest" , "Volume" : "volume" } ) options_df = options_df . sort_values ( by = [ "date" , "expiration date" , "strike price" ] ) . reset_index ( drop = True ) return options_df
This function aims to aggregate and adjust all option data for a given ticker.
This function aims to aggregate and adjust all option data for a given ticker .
"""
    This function aims to aggregate and adjust all option data for a given ticker.
    Adjustment is made based on historical splits. For example, had a stock undergone
    a forward split of factor 2, all previous strike price, ask/bid price ...etc are
    divided by 2. On the flip side, ask/bid size, volume ...etc are multiplied by 2.
    Function has to be modified depending on structure of options data.

    :param stock_of_interest: ticker symbol (string)
    :param option_data_path: path where all the option data files are stored (string)
    :param history_df: historical end of day prices for ticker (DataFrame)
    :return: options_df: all ticker specific options data (DataFrame)
    """ # Nested loading of options data
 # Filtering for the right symbol
 # Dropping columns are reordering others
 # Change to datetime
 # Retrieve adjustment factor and closing price
 # Adjusting option data as needed
 # Append to DataFrame
 # Renaming columns

all ticker specific options data (DataFrame)
all ticker specific options data ( DataFrame )
stock_of_interest
ticker symbol (string)
ticker symbol ( string )
option_data_path
path where all the option data files are stored (string)
path where all the option data files are stored ( string )
history_df
historical end of day prices for ticker (DataFrame)
historical end of day prices for ticker ( DataFrame )