back-end/utils/descriptive_analysis.py
def update_time_series_analysis_files(attacks_data_frame, analysis_file_name, path):
    """ This function transforms data from two different sources and combines them to produce a merged result. This
        result is then saved in a file. The first source is attacks data_frame (contains one or more data frames) that
        gets transformed into a dictionary. The second source is the analysis_file_name.JSON that gets tranformed into a
        dictionary too. Now that both sources are represented with the same dictionary structure it is easy to be merged
        by keys. The merged result is then represented into list of lists and saved to analysis_file_name (both
        json and csv), so can be read by highcharts.

    @parameters
        attacks_data_frame      (pandas data frame)    this data frame contains a number of attacks for each day
        analysis_file_name      (str)                  the name of analysis file (with NO EXTENSION)
        path                    (str)                  path of the analysis file
    """

    if attacks_data_frame.empty:
        try:
            raise Warning("No documents in cursor to be written")
        except Exception as e:
            print("descriptive_analysis module > update_time_series_analysis_files: ", e)
        return

    # analysis content results to dictionary
    analysed_attacks_dict = __analysis_data_frame_to_dict__(attacks_data_frame)

    # load content from analysis file (perdayTimeSeriesAnalysis)
    json_file_analysis = path + analysis_file_name + '.json'
    csv_file_analysis = path + analysis_file_name + '.csv'

    try:
        with open(json_file_analysis, 'r') as file:
            file_loaded_attacks = json.loads(file.read())
            # turn data in dictionary representation
        file_loaded_attacks_dict = __analysis_list_to_dict__(file_loaded_attacks)

        # merge contents
        for key, value in analysed_attacks_dict.items():
            file_loaded_attacks_dict[key] = value

        updated_attacks_dict = file_loaded_attacks_dict

        # back to list of lists representation for high charts
        updated_attacks_list = list([])
        for key, value in updated_attacks_dict.items():
            updated_attacks_list.append([key, value])
    except Exception as e:
        print("descriptive_analysis module > update_analysis_files > File not found and will be configured: ", e)
        updated_attacks_list = list([])
        for key, value in analysed_attacks_dict.items():
            updated_attacks_list.append([key, value])

    '''At Feb-2019 a bug occured with visualisation Highchart due to unsorted records. Sorting list is applied
    as a solution.'''
    updated_attacks_list.sort(key=lambda x: x[0])

    print('\nBegin writing results to ' + analysis_file_name + '.json and ' + analysis_file_name + '.csv...', end='')
    # write json file
    try:
        with open(json_file_analysis, 'w') as json_file:
            json.dump(updated_attacks_list, json_file, separators=(',', ': '), indent=4)
    except Exception as e:
        print("descriptive_analysis module > update_analysis_files > Store in JSON file Error: ", e)

    # write csv file
    try:
        with open(csv_file_analysis, 'w') as csv_file:
            for result in updated_attacks_list:
                # highcharts timestamp must be converted in date before saved to csv. External use NCSR Demokritos
                date_notation = __high_charts_timestamp_to_date__(result[0])
                csv_file.write(str(date_notation) + str(',') + str(result[1]) + str("\n"))
    except Exception as e:
        print("descriptive_analysis module > update_analysis_files > Store in CSV file Error: ", e)

    print('\tCompleted')

    # -----------------------------------------------------------------------------------------------------------------#

 This function transforms data from two different sources and combines them to produce a merged result. This
        result is then saved in a file. The first source is attacks data_frame (contains one or more data frames) that
        gets transformed into a dictionary. The second source is the analysis_file_name.JSON that gets tranformed into a
        dictionary too. Now that both sources are represented with the same dictionary structure it is easy to be merged
        by keys. The merged result is then represented into list of lists and saved to analysis_file_name (both
        json and csv), so can be read by highcharts.

    @parameters
        attacks_data_frame      (pandas data frame)    this data frame contains a number of attacks for each day
        analysis_file_name      (str)                  the name of analysis file (with NO EXTENSION)
        path                    (str)                  path of the analysis file
    
This function transforms data from two different sources and combines them to produce a merged result. This
result is then saved in a file. The first source is attacks data_frame (contains one or more data frames) that
gets transformed into a dictionary. The second source is the analysis_file_name.JSON that gets tranformed into a
dictionary too. Now that both sources are represented with the same dictionary structure it is easy to be merged
by keys. The merged result is then represented into list of lists and saved to analysis_file_name (both
json and csv), so can be read by highcharts.

@parameters
attacks_data_frame      (pandas data frame)    this data frame contains a number of attacks for each day
analysis_file_name      (str)                  the name of analysis file (with NO EXTENSION)
path                    (str)                  path of the analysis file
This function transforms data from two different sources and combines them to produce a merged result . This result is then saved in a file . The first source is attacks data_frame ( contains one or more data frames ) that gets transformed into a dictionary . The second source is the analysis_file_name . JSON that gets tranformed into a dictionary too . Now that both sources are represented with the same dictionary structure it is easy to be merged by keys . The merged result is then represented into list of lists and saved to analysis_file_name ( both json and csv ) so can be read by highcharts . @parameters attacks_data_frame ( pandas data frame ) this data frame contains a number of attacks for each day analysis_file_name ( str ) the name of analysis file ( with NO EXTENSION ) path ( str ) path of the analysis file
def update_time_series_analysis_files(attacks_data_frame, analysis_file_name, path):
    if attacks_data_frame.empty:
        try:
            raise Warning("No documents in cursor to be written")
        except Exception as e:
            print("descriptive_analysis module > update_time_series_analysis_files: ", e)
        return
    analysed_attacks_dict = __analysis_data_frame_to_dict__(attacks_data_frame)
    json_file_analysis = path + analysis_file_name + '.json'
    csv_file_analysis = path + analysis_file_name + '.csv'
    try:
        with open(json_file_analysis, 'r') as file:
            file_loaded_attacks = json.loads(file.read())
        file_loaded_attacks_dict = __analysis_list_to_dict__(file_loaded_attacks)
        for key, value in analysed_attacks_dict.items():
            file_loaded_attacks_dict[key] = value
        updated_attacks_dict = file_loaded_attacks_dict
        updated_attacks_list = list([])
        for key, value in updated_attacks_dict.items():
            updated_attacks_list.append([key, value])
    except Exception as e:
        print("descriptive_analysis module > update_analysis_files > File not found and will be configured: ", e)
        updated_attacks_list = list([])
        for key, value in analysed_attacks_dict.items():
            updated_attacks_list.append([key, value])
    updated_attacks_list.sort(key=lambda x: x[0])
    print('\nBegin writing results to ' + analysis_file_name + '.json and ' + analysis_file_name + '.csv...', end='')
    try:
        with open(json_file_analysis, 'w') as json_file:
            json.dump(updated_attacks_list, json_file, separators=(',', ': '), indent=4)
    except Exception as e:
        print("descriptive_analysis module > update_analysis_files > Store in JSON file Error: ", e)
    try:
        with open(csv_file_analysis, 'w') as csv_file:
            for result in updated_attacks_list:
                date_notation = __high_charts_timestamp_to_date__(result[0])
                csv_file.write(str(date_notation) + str(',') + str(result[1]) + str("\n"))
    except Exception as e:
        print("descriptive_analysis module > update_analysis_files > Store in CSV file Error: ", e)
    print('\tCompleted')
def update_time_series_analysis_files ( attacks_data_frame , analysis_file_name , path ) : if attacks_data_frame . empty : try : raise Warning ( "No documents in cursor to be written" ) except Exception as e : print ( "descriptive_analysis module > update_time_series_analysis_files: " , e ) return analysed_attacks_dict = __analysis_data_frame_to_dict__ ( attacks_data_frame ) json_file_analysis = path + analysis_file_name + '.json' csv_file_analysis = path + analysis_file_name + '.csv' try : with open ( json_file_analysis , 'r' ) as file : file_loaded_attacks = json . loads ( file . read ( ) ) file_loaded_attacks_dict = __analysis_list_to_dict__ ( file_loaded_attacks ) for key , value in analysed_attacks_dict . items ( ) : file_loaded_attacks_dict [ key ] = value updated_attacks_dict = file_loaded_attacks_dict updated_attacks_list = list ( [ ] ) for key , value in updated_attacks_dict . items ( ) : updated_attacks_list . append ( [ key , value ] ) except Exception as e : print ( "descriptive_analysis module > update_analysis_files > File not found and will be configured: " , e ) updated_attacks_list = list ( [ ] ) for key , value in analysed_attacks_dict . items ( ) : updated_attacks_list . append ( [ key , value ] ) '''At Feb-2019 a bug occured with visualisation Highchart due to unsorted records. Sorting list is applied
    as a solution.''' updated_attacks_list . sort ( key = lambda x : x [ 0 ] ) print ( '\nBegin writing results to ' + analysis_file_name + '.json and ' + analysis_file_name + '.csv...' , end = '' ) try : with open ( json_file_analysis , 'w' ) as json_file : json . dump ( updated_attacks_list , json_file , separators = ( ',' , ': ' ) , indent = 4 ) except Exception as e : print ( "descriptive_analysis module > update_analysis_files > Store in JSON file Error: " , e ) try : with open ( csv_file_analysis , 'w' ) as csv_file : for result in updated_attacks_list : date_notation = __high_charts_timestamp_to_date__ ( result [ 0 ] ) csv_file . write ( str ( date_notation ) + str ( ',' ) + str ( result [ 1 ] ) + str ( "\n" ) ) except Exception as e : print ( "descriptive_analysis module > update_analysis_files > Store in CSV file Error: " , e ) print ( '\tCompleted' )
This function transforms data from two different sources and combines them to produce a merged result.
This function transforms data from two different sources and combines them to produce a merged result .
""" This function transforms data from two different sources and combines them to produce a merged result. This
        result is then saved in a file. The first source is attacks data_frame (contains one or more data frames) that
        gets transformed into a dictionary. The second source is the analysis_file_name.JSON that gets tranformed into a
        dictionary too. Now that both sources are represented with the same dictionary structure it is easy to be merged
        by keys. The merged result is then represented into list of lists and saved to analysis_file_name (both
        json and csv), so can be read by highcharts.

    @parameters
        attacks_data_frame      (pandas data frame)    this data frame contains a number of attacks for each day
        analysis_file_name      (str)                  the name of analysis file (with NO EXTENSION)
        path                    (str)                  path of the analysis file
    """ # analysis content results to dictionary
 # load content from analysis file (perdayTimeSeriesAnalysis)
 # turn data in dictionary representation
 # merge contents
 # back to list of lists representation for high charts
 '''At Feb-2019 a bug occured with visualisation Highchart due to unsorted records. Sorting list is applied
    as a solution.''' # write json file
 # write csv file
 # highcharts timestamp must be converted in date before saved to csv. External use NCSR Demokritos
 # -----------------------------------------------------------------------------------------------------------------#

attacks_data_frame
analysis_file_name
path