class_docstr: Only ``kernel_size`` ``in_channels`` and ``out_channels`` are supported. Kernel size candidates should be larger or smaller
  than each other in both candidates. See examples below:
  the following example is not allowed:
      >>> ValueChoice(candidates = [(5, 3), (3, 5)])
          □ ■ ■ ■ □   □ □ □ □ □
          □ ■ ■ ■ □   ■ ■ ■ ■ ■    # candidates are not bigger or smaller on both dimension
          □ ■ ■ ■ □   ■ ■ ■ ■ ■
          □ ■ ■ ■ □   ■ ■ ■ ■ ■
          □ ■ ■ ■ □   □ □ □ □ □
  the following 3 examples are valid:
      >>> ValueChoice(candidates = [5, 3, 1])
          ■ ■ ■ ■ ■   □ □ □ □ □   □ □ □ □ □
          ■ ■ ■ ■ ■   □ ■ ■ ■ □   □ □ □ □ □
          ■ ■ ■ ■ ■   □ ■ ■ ■ □   □ □ ■ □ □
          ■ ■ ■ ■ ■   □ ■ ■ ■ □   □ □ □ □ □
          ■ ■ ■ ■ ■   □ □ □ □ □   □ □ □ □ □
      >>> ValueChoice(candidates = [(5, 7), (3, 5), (1, 3)])
          ■ ■ ■ ■ ■ ■ ■  □ □ □ □ □ □ □   □ □ □ □ □ □ □
          ■ ■ ■ ■ ■ ■ ■  □ ■ ■ ■ ■ ■ □   □ □ □ □ □ □ □
          ■ ■ ■ ■ ■ ■ ■  □ ■ ■ ■ ■ ■ □   □ □ ■ ■ ■ □ □
          ■ ■ ■ ■ ■ ■ ■  □ ■ ■ ■ ■ ■ □   □ □ □ □ □ □ □
          ■ ■ ■ ■ ■ ■ ■  □ □ □ □ □ □ □   □ □ □ □ □ □ □
      >>> # when the difference between any two candidates is not even, the left upper will be picked:
      >>> ValueChoice(candidates = [(5, 5), (4, 4), (3, 3)])
          ■ ■ ■ ■ ■   ■ ■ ■ ■ □   □ □ □ □ □
          ■ ■ ■ ■ ■   ■ ■ ■ ■ □   □ ■ ■ ■ □
          ■ ■ ■ ■ ■   ■ ■ ■ ■ □   □ ■ ■ ■ □
          ■ ■ ■ ■ ■   ■ ■ ■ ■ □   □ ■ ■ ■ □
          ■ ■ ■ ■ ■   □ □ □ □ □   □ □ □ □ □
class_docstr_tok: ["Only","``","kernel_size","``","``","in_channels","``","and","``","out_channels","``","are","supported",".","Kernel","size","candidates","should","be","larger","or","smaller","than","each","other","in","both","candidates",".","See","examples","below",":","the","following","example","is","not","allowed",":",">",">",">","ValueChoice","(","candidates","=","[","(","5",",","3",")",",","(","3",",","5",")","]",")","\u25a1","\u25a0","\u25a0","\u25a0","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a0","\u25a0","\u25a0","\u25a1","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","#","candidates","are","not","bigger","or","smaller","on","both","dimension","\u25a1","\u25a0","\u25a0","\u25a0","\u25a1","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a1","\u25a0","\u25a0","\u25a0","\u25a1","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a1","\u25a0","\u25a0","\u25a0","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","the","following","3","examples","are","valid",":",">",">",">","ValueChoice","(","candidates","=","[","5",",","3",",","1","]",")","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a1","\u25a0","\u25a0","\u25a0","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a1","\u25a0","\u25a0","\u25a0","\u25a1","\u25a1","\u25a1","\u25a0","\u25a1","\u25a1","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a1","\u25a0","\u25a0","\u25a0","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1",">",">",">","ValueChoice","(","candidates","=","[","(","5",",","7",")",",","(","3",",","5",")",",","(","1",",","3",")","]",")","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a1","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a1","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a1","\u25a1","\u25a1","\u25a0","\u25a0","\u25a0","\u25a1","\u25a1","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a1","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1",">",">",">","#","when","the","difference","between","any","two","candidates","is","not","even",",","the","left","upper","will","be","picked",":",">",">",">","ValueChoice","(","candidates","=","[","(","5",",","5",")",",","(","4",",","4",")",",","(","3",",","3",")","]",")","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a1","\u25a1","\u25a0","\u25a0","\u25a0","\u25a1","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a1","\u25a1","\u25a0","\u25a0","\u25a0","\u25a1","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a1","\u25a1","\u25a0","\u25a0","\u25a0","\u25a1","\u25a0","\u25a0","\u25a0","\u25a0","\u25a0","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1","\u25a1"]
code: def forward(self, input):
      # Note that there is no need to handle ``in_channels`` here since it is already handle by the ``out_channels`` in the
      # previous module. If we multiply alpha with refer to ``in_channels`` here again, the alpha will indeed be considered
      # twice, which is not what we expect.
      weight = self.weight
  
      def sum_weight(input_weight, masks, thresholds, indicator):
          """
          This is to get the weighted sum of weight.
  
          Parameters
          ----------
          input_weight : Tensor
              the weight to be weighted summed
          masks : list[Tensor]
              weight masks.
          thresholds : list[float]
              thresholds, should have a length of ``len(masks) - 1``
          indicator : Callable[[Tensor, float], float]
              take a tensor and a threshold as input, and output the weight
  
          Returns
          ----------
          weight : Tensor
              weighted sum of ``input_weight``. this is of the same shape as ``input_sum``
          """
          # Note that ``masks`` and ``thresholds`` have different lengths. There alignment is shown below:
          # self.xxx_candidates = [   c_0  ,   c_1  , ... ,  c_n-2  ,   c_n-1 ] # descending order
          # self.xxx_mask       = [ mask_0 , mask_1 , ... , mask_n-2, mask_n-1]
          # self.t_xxx          = [   t_0  ,   t_2  , ... ,  t_n-2 ]
          # So we zip the first n-1 items, and multiply masks[-1] in the end.
          weight = torch.zeros_like(input_weight)
          for mask, t in zip(masks[:-1], thresholds):
              cur_part = input_weight * mask
              alpha = indicator(cur_part, t)
              weight = (weight + cur_part) * alpha
          # we do not consider skip-op here for out_channel/expansion candidates, which means at least the smallest channel
          # candidate is included
          weight += input_weight * masks[-1]
  
          return weight
  
      if self.kernel_size_candidates is not None:
          weight = sum_weight(
              weight,
              self.kernel_masks,
              self.t_kernel,
              self.Lasso_sigmoid,
          )
  
      if self.out_channel_candidates is not None:
          weight = sum_weight(
              weight,
              self.channel_masks,
              self.t_expansion,
              self.Lasso_sigmoid,
          )
  
      output = self._conv_forward(input, weight, self.bias)
      return output
  
file_path: nni/nni/nas/oneshot/pytorch/supermodule/_singlepathnas.py
imports: import torch
  import torch.nn
  import torch.nn.functional
  from nni.nas.nn.pytorch import ValueChoice
label_desc: Use image node_id 2 for calling the DifferentiableSuperConv2d obj's underlying member method code with example usage: obj.forward(input) and returns: output, weight