class_docstr: Class that implements the optimization for optimized pre-processing.
  
  Based on:
  http://papers.nips.cc/paper/6988-optimized-pre-processing-for-discrimination-prevention
  
  and
  
  https://github.com/fair-preprocessing/nips2017
  
  The particular formulation implemented here is:
  1. l1 distance between input and transformed distributions
  2. "Excess distortion constraint" - eqn 5 in paper.
  3. Discrimination constraints for all combinations of groups specified
     (there is no distinction between protected and unprotected groups). The
     constraints are given in eqn 2, 3 in the paper. We use a single epsilon
     value for all combinations of y and d values
  
  See section 4.3 in supplementary material of the paper for an example
  
  Attributes:
      features (list): All features
      df (DataFrame): Input data
      dfJoint (DataFrame): Empirical joint distribution
      D_features (list): protected attribute names
      X_features (list): feature names for input data
      Y_features (list): feature names for binary label
      X_values (list): Values that features can take
      Y_values (list): Values that the label can take
      D_values (list): Values that protected attributes can take
      XY_features (list): Combination of X, and Y features
      DXY_features (list): Combination of D, X, and Y features
      XY_values (list): Combination of X, and Y values
      DXY_values (list): Combination of D, X, and Y values
      y_index (Int64Index): Indices for the Y values
      XY_index (MultiIndex): Indices for the combination of X, and Y values
      DXY_index (MultiIndex): Indices for the combination of D, X, and Y
          values
      YD_features_index (MultiIndex): Indices for the combination of Y, and D
          values
  
      clist (list): Distance thresholds for individual distortion
      CMlist (list): List of constraint matrices corresponding to each
          threshold in clist
      dfD (DataFrame): distortion matrix with indices and columns
      dlist (list): Probability bounds given in eq. 5 of the paper for
          each threshold in clist
      epsilon (float): epsilon value used in discrimination constraint
  
      dfD_to_Y_address (Dataframe): matrix for p_yd, with y varying in the
          columns
      dfMask_Pxyd_to_Pyd (DataFrame): Mask to transform P_XYD to P_YD
      dfMask_Pxyd_to_Pxy (DataFrame): Mask to convert from P_XYD to P_XY
      dfPxyd (DataFrame): Representation of only frequencies from dfJoint
      dfMask_Pxyd_to_Py (DataFrame): Mask to convert from P_XYD to P_Y
      dfMask_Pxy_to_Py (DataFrame): Mask to convert from P_XY to P_Y
      dfMask_Pxyd_to_Pd (DataFrame): Mask to convert from P_XYD to P_D
      dfP (DataFrame): Mapping transformation learned from the data
class_docstr_tok: ["Class","that","implements","the","optimization","for","optimized","pre-processing",".","Based","on",":","http",":","\/\/papers.nips.cc\/paper\/6988-optimized-pre-processing-for-discrimination-prevention","and","https",":","\/\/github.com\/fair-preprocessing\/nips2017","The","particular","formulation","implemented","here","is",":","1.","l1","distance","between","input","and","transformed","distributions","2",".","``","Excess","distortion","constraint","''","-","eqn","5","in","paper",".","3",".","Discrimination","constraints","for","all","combinations","of","groups","specified","(","there","is","no","distinction","between","protected","and","unprotected","groups",")",".","The","constraints","are","given","in","eqn","2",",","3","in","the","paper",".","We","use","a","single","epsilon","value","for","all","combinations","of","y","and","d","values","See","section","4.3","in","supplementary","material","of","the","paper","for","an","example","Attributes",":","features","(","list",")",":","All","features","df","(","DataFrame",")",":","Input","data","dfJoint","(","DataFrame",")",":","Empirical","joint","distribution","D_features","(","list",")",":","protected","attribute","names","X_features","(","list",")",":","feature","names","for","input","data","Y_features","(","list",")",":","feature","names","for","binary","label","X_values","(","list",")",":","Values","that","features","can","take","Y_values","(","list",")",":","Values","that","the","label","can","take","D_values","(","list",")",":","Values","that","protected","attributes","can","take","XY_features","(","list",")",":","Combination","of","X",",","and","Y","features","DXY_features","(","list",")",":","Combination","of","D",",","X",",","and","Y","features","XY_values","(","list",")",":","Combination","of","X",",","and","Y","values","DXY_values","(","list",")",":","Combination","of","D",",","X",",","and","Y","values","y_index","(","Int64Index",")",":","Indices","for","the","Y","values","XY_index","(","MultiIndex",")",":","Indices","for","the","combination","of","X",",","and","Y","values","DXY_index","(","MultiIndex",")",":","Indices","for","the","combination","of","D",",","X",",","and","Y","values","YD_features_index","(","MultiIndex",")",":","Indices","for","the","combination","of","Y",",","and","D","values","clist","(","list",")",":","Distance","thresholds","for","individual","distortion","CMlist","(","list",")",":","List","of","constraint","matrices","corresponding","to","each","threshold","in","clist","dfD","(","DataFrame",")",":","distortion","matrix","with","indices","and","columns","dlist","(","list",")",":","Probability","bounds","given","in","eq",".","5","of","the","paper","for","each","threshold","in","clist","epsilon","(","float",")",":","epsilon","value","used","in","discrimination","constraint","dfD_to_Y_address","(","Dataframe",")",":","matrix","for","p_yd",",","with","y","varying","in","the","columns","dfMask_Pxyd_to_Pyd","(","DataFrame",")",":","Mask","to","transform","P_XYD","to","P_YD","dfMask_Pxyd_to_Pxy","(","DataFrame",")",":","Mask","to","convert","from","P_XYD","to","P_XY","dfPxyd","(","DataFrame",")",":","Representation","of","only","frequencies","from","dfJoint","dfMask_Pxyd_to_Py","(","DataFrame",")",":","Mask","to","convert","from","P_XYD","to","P_Y","dfMask_Pxy_to_Py","(","DataFrame",")",":","Mask","to","convert","from","P_XY","to","P_Y","dfMask_Pxyd_to_Pd","(","DataFrame",")",":","Mask","to","convert","from","P_XYD","to","P_D","dfP","(","DataFrame",")",":","Mapping","transformation","learned","from","the","data"]
docstr: Create distortion and constraint matrices
  Args:
      get_distortion (function): Distortion function name
          (See optim_preproc_helper.get_distortion for an example)
      clist (list): Distance thresholds for individual distortion
docstr_tok: ["Create","distortion","and","constraint","matrices","Args",":","get_distortion","(","function",")",":","Distortion","function","name","(","See","optim_preproc_helper.get_distortion","for","an","example",")","clist","(","list",")",":","Distance","thresholds","for","individual","distortion"]
code: def set_distortion(self, get_distortion, clist=[]):
      """Create distortion and constraint matrices
      Args:
          get_distortion (function): Distortion function name
              (See optim_preproc_helper.get_distortion for an example)
          clist (list): Distance thresholds for individual distortion
      """
  
      # set constraint list
      self.clist = clist
  
      # create row dictionay (rows represent old values)
      # this will make it easier to compute distrotion metric
      rows_tuple = self.dfD.index.tolist()
      rows_dict = [
          {
              self.XY_features[i]: t[i]
              for i in range(len(self.XY_features))
          }
          for t in rows_tuple
      ]
  
      # create columns dictionay (columns represent new values)
      cols_tuple = self.dfD.columns.tolist()
      cols_dict = [
          {
              self.XY_features[i]: t[i]
              for i in range(len(self.XY_features))
          }
          for t in cols_tuple
      ]
  
      # Create distortion matrix
      for i in range(self.dfD.shape[0]):
          old_values = rows_dict[i]
          for j in range(self.dfD.shape[1]):
              new_values = cols_dict[j]
              self.dfD.iat[i, j] = get_distortion(
                  old_values, new_values
              )
  
      Dmatrix = self.dfD.values
  
      # Create constraint matrix list for excess distortion
      # since old values index the rows, we go through the D matrix line by
      # line, marking as 1 events where the threshold is violated. This will
      # be multiplied by the probability matrix, resulting in the excess
      # distortion metric
      self.CMlist = [
          np.zeros(Dmatrix.shape) for i in range(len(self.clist))
      ]
      for x in range(len(self.CMlist)):
          c = self.clist[x]
          for i in range(Dmatrix.shape[0]):
              for j in range(Dmatrix.shape[1]):
                  if Dmatrix[i, j] >= c:
                      self.CMlist[x][i, j] = 1.0
  
file_path: AIF360/aif360/algorithms/preprocessing/optim_preproc_helpers/opt_tools.py
imports: import numpy
  import pandas
  import cvxpy
  from cvxpy import Problem, Minimize, Variable
label_desc: Use image node_id 4 for calling the OptTools obj's underlying member method code with example usage: obj.set_distortion(get_distortion, clist) without return types