class_docstr: Class that implements the optimization for optimized pre-processing.
  
  Based on:
  http://papers.nips.cc/paper/6988-optimized-pre-processing-for-discrimination-prevention
  
  and
  
  https://github.com/fair-preprocessing/nips2017
  
  The particular formulation implemented here is:
  1. l1 distance between input and transformed distributions
  2. "Excess distortion constraint" - eqn 5 in paper.
  3. Discrimination constraints for all combinations of groups specified
     (there is no distinction between protected and unprotected groups). The
     constraints are given in eqn 2, 3 in the paper. We use a single epsilon
     value for all combinations of y and d values
  
  See section 4.3 in supplementary material of the paper for an example
  
  Attributes:
      features (list): All features
      df (DataFrame): Input data
      dfJoint (DataFrame): Empirical joint distribution
      D_features (list): protected attribute names
      X_features (list): feature names for input data
      Y_features (list): feature names for binary label
      X_values (list): Values that features can take
      Y_values (list): Values that the label can take
      D_values (list): Values that protected attributes can take
      XY_features (list): Combination of X, and Y features
      DXY_features (list): Combination of D, X, and Y features
      XY_values (list): Combination of X, and Y values
      DXY_values (list): Combination of D, X, and Y values
      y_index (Int64Index): Indices for the Y values
      XY_index (MultiIndex): Indices for the combination of X, and Y values
      DXY_index (MultiIndex): Indices for the combination of D, X, and Y
          values
      YD_features_index (MultiIndex): Indices for the combination of Y, and D
          values
  
      clist (list): Distance thresholds for individual distortion
      CMlist (list): List of constraint matrices corresponding to each
          threshold in clist
      dfD (DataFrame): distortion matrix with indices and columns
      dlist (list): Probability bounds given in eq. 5 of the paper for
          each threshold in clist
      epsilon (float): epsilon value used in discrimination constraint
  
      dfD_to_Y_address (Dataframe): matrix for p_yd, with y varying in the
          columns
      dfMask_Pxyd_to_Pyd (DataFrame): Mask to transform P_XYD to P_YD
      dfMask_Pxyd_to_Pxy (DataFrame): Mask to convert from P_XYD to P_XY
      dfPxyd (DataFrame): Representation of only frequencies from dfJoint
      dfMask_Pxyd_to_Py (DataFrame): Mask to convert from P_XYD to P_Y
      dfMask_Pxy_to_Py (DataFrame): Mask to convert from P_XY to P_Y
      dfMask_Pxyd_to_Pd (DataFrame): Mask to convert from P_XYD to P_D
      dfP (DataFrame): Mapping transformation learned from the data
class_docstr_tok: ["Class","that","implements","the","optimization","for","optimized","pre-processing",".","Based","on",":","http",":","\/\/papers.nips.cc\/paper\/6988-optimized-pre-processing-for-discrimination-prevention","and","https",":","\/\/github.com\/fair-preprocessing\/nips2017","The","particular","formulation","implemented","here","is",":","1.","l1","distance","between","input","and","transformed","distributions","2",".","``","Excess","distortion","constraint","''","-","eqn","5","in","paper",".","3",".","Discrimination","constraints","for","all","combinations","of","groups","specified","(","there","is","no","distinction","between","protected","and","unprotected","groups",")",".","The","constraints","are","given","in","eqn","2",",","3","in","the","paper",".","We","use","a","single","epsilon","value","for","all","combinations","of","y","and","d","values","See","section","4.3","in","supplementary","material","of","the","paper","for","an","example","Attributes",":","features","(","list",")",":","All","features","df","(","DataFrame",")",":","Input","data","dfJoint","(","DataFrame",")",":","Empirical","joint","distribution","D_features","(","list",")",":","protected","attribute","names","X_features","(","list",")",":","feature","names","for","input","data","Y_features","(","list",")",":","feature","names","for","binary","label","X_values","(","list",")",":","Values","that","features","can","take","Y_values","(","list",")",":","Values","that","the","label","can","take","D_values","(","list",")",":","Values","that","protected","attributes","can","take","XY_features","(","list",")",":","Combination","of","X",",","and","Y","features","DXY_features","(","list",")",":","Combination","of","D",",","X",",","and","Y","features","XY_values","(","list",")",":","Combination","of","X",",","and","Y","values","DXY_values","(","list",")",":","Combination","of","D",",","X",",","and","Y","values","y_index","(","Int64Index",")",":","Indices","for","the","Y","values","XY_index","(","MultiIndex",")",":","Indices","for","the","combination","of","X",",","and","Y","values","DXY_index","(","MultiIndex",")",":","Indices","for","the","combination","of","D",",","X",",","and","Y","values","YD_features_index","(","MultiIndex",")",":","Indices","for","the","combination","of","Y",",","and","D","values","clist","(","list",")",":","Distance","thresholds","for","individual","distortion","CMlist","(","list",")",":","List","of","constraint","matrices","corresponding","to","each","threshold","in","clist","dfD","(","DataFrame",")",":","distortion","matrix","with","indices","and","columns","dlist","(","list",")",":","Probability","bounds","given","in","eq",".","5","of","the","paper","for","each","threshold","in","clist","epsilon","(","float",")",":","epsilon","value","used","in","discrimination","constraint","dfD_to_Y_address","(","Dataframe",")",":","matrix","for","p_yd",",","with","y","varying","in","the","columns","dfMask_Pxyd_to_Pyd","(","DataFrame",")",":","Mask","to","transform","P_XYD","to","P_YD","dfMask_Pxyd_to_Pxy","(","DataFrame",")",":","Mask","to","convert","from","P_XYD","to","P_XY","dfPxyd","(","DataFrame",")",":","Representation","of","only","frequencies","from","dfJoint","dfMask_Pxyd_to_Py","(","DataFrame",")",":","Mask","to","convert","from","P_XYD","to","P_Y","dfMask_Pxy_to_Py","(","DataFrame",")",":","Mask","to","convert","from","P_XY","to","P_Y","dfMask_Pxyd_to_Pd","(","DataFrame",")",":","Mask","to","convert","from","P_XYD","to","P_D","dfP","(","DataFrame",")",":","Mapping","transformation","learned","from","the","data"]
docstr: Set many features for the class
  
  Args:
      D (list): names of D features
      X (list): names of X features
      Y (list): names of Y features
docstr_tok: ["Set","many","features","for","the","class","Args",":","D","(","list",")",":","names","of","D","features","X","(","list",")",":","names","of","X","features","Y","(","list",")",":","names","of","Y","features"]
code: def set_features(self, D=[], X=[], Y=[]):
      """Set many features for the class
  
      Args:
          D (list): names of D features
          X (list): names of X features
          Y (list): names of Y features
      """
  
      self.D_features = D
      self.Y_features = Y
      self.X_features = X
  
      # Get values for Pandas multindex
      self.D_values = [
          self.dfJoint[feature].unique().tolist()
          for feature in self.D_features
      ]
      self.Y_values = [
          self.dfJoint[feature].unique().tolist()
          for feature in self.Y_features
      ]
      self.X_values = [
          self.dfJoint[feature].unique().tolist()
          for feature in self.X_features
      ]
  
      # Create multindex for mapping dataframe
      self.DXY_features = (
          self.D_features + self.X_features + self.Y_features
      )
      self.DXY_values = self.D_values + self.X_values + self.Y_values
      self.DXY_index = pd.MultiIndex.from_product(
          self.DXY_values, names=self.DXY_features
      )
  
      # Create multindex for distortion dataframe
      self.XY_features = self.X_features + self.Y_features
      self.XY_values = self.X_values + self.Y_values
      self.XY_index = pd.MultiIndex.from_product(
          self.XY_values, names=self.XY_features
      )
  
      # Initialize mapping dataframe
      self.dfP = pd.DataFrame(
          np.zeros((len(self.DXY_index), len(self.XY_index))),
          index=self.DXY_index,
          columns=self.XY_index,
      )
  
      # Initialize distortion dataframe
      self.dfD = pd.DataFrame(
          np.zeros((len(self.XY_index), len(self.XY_index))),
          index=self.XY_index.copy(),
          columns=self.XY_index.copy(),
      )
  
      ###
      # Generate masks for recovering marginals
      ###
      self.dfPxyd = pd.DataFrame(
          index=self.dfP.index, columns=["Frequency"]
      )
      index_list = [list(x) for x in self.dfPxyd.index.tolist()]
  
      # find corresponding frequency value
      i = 0
      for comb in self.dfJoint[self.DXY_features].values.tolist():
          # get the entry corresponding to the combination
          idx = index_list.index(comb)
          # add marginal to list
          self.dfPxyd.iloc[idx, 0] = self.dfJoint.loc[i, "Frequency"]
          i += 1
  
      # create mask that reduces Pxyd to Pxy
      # so Pxyd.dot(dfMask1) = Pxy
      self.dfMask_Pxyd_to_Pxy = pd.DataFrame(
          np.zeros((len(self.dfP), len(self.dfD))),
          index=self.dfP.index,
          columns=self.dfD.index,
      )
      self.dfMask_Pxyd_to_Pxy = self.get_mask(self.dfMask_Pxyd_to_Pxy)
  
      # compute mask that reduces Pxyd to Pyd
      self.YD_features_index = (
          self.dfJoint.groupby(self.Y_features + self.D_features)[
              "Frequency"
          ]
          .sum()
          .index
      )
      self.dfMask_Pxyd_to_Pyd = pd.DataFrame(
          np.zeros((len(self.dfP), len(self.YD_features_index))),
          index=self.dfP.index,
          columns=self.YD_features_index,
      )
      self.dfMask_Pxyd_to_Pyd = self.get_mask(self.dfMask_Pxyd_to_Pyd)
  
      # get  matrix for p_yd, with y varying in the columns
      self.dfD_to_Y_address = pd.Series(
          range(len(list(self.dfMask_Pxyd_to_Pyd))),
          index=self.dfMask_Pxyd_to_Pyd.columns,
      )
      # print(self.dfD_to_Y_address, self.dfD_to_Y_address.shape)
      self.dfD_to_Y_address = pd.pivot_table(
          self.dfD_to_Y_address.reset_index(),
          columns=self.D_features,
          index=self.Y_features,
          values=0,
      )
  
      # compute mask that reduces Pxyd to Py
      self.y_index = self.dfD_to_Y_address.index
      self.dfMask_Pxyd_to_Py = pd.DataFrame(
          np.zeros((len(self.dfP), len(self.y_index))),
          index=self.dfP.index,
          columns=self.y_index,
      )
      self.dfMask_Pxyd_to_Py = self.get_mask(self.dfMask_Pxyd_to_Py)
  
      # compute mask that reduces Pxy to Py
      self.dfMask_Pxy_to_Py = pd.DataFrame(
          np.zeros((len(list(self.dfP)), len(self.y_index))),
          index=self.dfP.columns,
          columns=self.y_index,
      )
      self.dfMask_Pxy_to_Py = self.get_mask(self.dfMask_Pxy_to_Py)
  
      # compute mask that reduces Pxyd to Pd
      self.dfMask_Pxyd_to_Pd = pd.DataFrame(
          np.zeros((len(self.dfP), self.dfD_to_Y_address.shape[1])),
          index=self.dfP.index,
          columns=self.dfD_to_Y_address.columns,
      )
      self.dfMask_Pxyd_to_Pd = self.get_mask(self.dfMask_Pxyd_to_Pd)
  
file_path: AIF360/aif360/algorithms/preprocessing/optim_preproc_helpers/opt_tools.py
imports: import numpy
  import pandas
  import cvxpy
  from cvxpy import Problem, Minimize, Variable
label_desc: Use image node_id 3 for calling the OptTools obj's underlying member method code with example usage: obj.set_features(D, X, Y) without return types