class_docstr: MobileNetV3Space implements the largest search space in `TuNAS <https://arxiv.org/abs/2008.06120>`__.
  
  The search dimensions include widths, expand ratios, kernel sizes, SE ratio.
  Some of them can be turned off via arguments to narrow down the search space.
  
  Different from ProxylessNAS search space, this space is implemented with :class:`~nni.nas.nn.pytorch.ValueChoice`.
  
  We use the following snipppet as reference.
  https://github.com/google-research/google-research/blob/20736344591f774f4b1570af64624ed1e18d2867/tunas/mobile_search_space_v3.py#L728
  
  We have ``num_blocks`` which equals to the length of ``self.blocks`` (the main body of the network).
  For simplicity, the following parameter specification assumes ``num_blocks`` equals 8 (body + head).
  If a shallower body is intended, arrays including ``base_widths``, ``squeeze_excite``, ``depth_range``,
  ``stride``, ``activation`` should also be shortened accordingly.
  
  Parameters
  ----------
  num_labels
      Dimensions for classification head.
  base_widths
      Widths of each stage, from stem, to body, to head.
      Length should be 9, i.e., ``num_blocks + 1`` (because there is a stem width in front).
  width_multipliers
      A range of widths multiplier to choose from. The choice is independent for each stage.
      Or it can be a fixed float. This will be applied on ``base_widths``,
      and we would also make sure that widths can be divided by 8.
  expand_ratios
      A list of expand ratios to choose from. Independent for every **block**.
  squeeze_excite
      Indicating whether the current stage can have an optional SE layer.
      Expect array of length 6 for stage 0 to 5. Each element can be one of ``force``, ``optional``, ``none``.
  depth_range
      A range (e.g., ``(1, 4)``),
      or a list of range (e.g., ``[(1, 3), (1, 4), (1, 4), (1, 3), (0, 2)]``).
      If a list, the length should be 5. The depth are specified for stage 1 to 5.
  stride
      Stride for all stages (including stem and head). Length should be same as ``base_widths``.
  activation
      Activation (class) for all stages. Length is same as ``base_widths``.
  se_from_exp
      Calculate SE channel reduction from expanded (mid) channels.
  dropout_rate
      Dropout rate at classification head.
  bn_eps
      Epsilon of batch normalization.
  bn_momentum
      Momentum of batch normalization.
class_docstr_tok: ["MobileNetV3Space","implements","the","largest","search","space","in","`","TuNAS","<","https",":","\/\/arxiv.org\/abs\/2008.06120",">","`","__",".","The","search","dimensions","include","widths",",","expand","ratios",",","kernel","sizes",",","SE","ratio",".","Some","of","them","can","be","turned","off","via","arguments","to","narrow","down","the","search","space",".","Different","from","ProxylessNAS","search","space",",","this","space","is","implemented","with",":","class",":","`","~nni.nas.nn.pytorch.ValueChoice","`",".","We","use","the","following","snipppet","as","reference",".","https",":","\/\/github.com\/google-research\/google-research\/blob\/20736344591f774f4b1570af64624ed1e18d2867\/tunas\/mobile_search_space_v3.py","#","L728","We","have","``","num_blocks","``","which","equals","to","the","length","of","``","self.blocks","``","(","the","main","body","of","the","network",")",".","For","simplicity",",","the","following","parameter","specification","assumes","``","num_blocks","``","equals","8","(","body","+","head",")",".","If","a","shallower","body","is","intended",",","arrays","including","``","base_widths","``",",","``","squeeze_excite","``",",","``","depth_range","``",",","``","stride","``",",","``","activation","``","should","also","be","shortened","accordingly",".","Parameters","--","--","--","--","--","num_labels","Dimensions","for","classification","head",".","base_widths","Widths","of","each","stage",",","from","stem",",","to","body",",","to","head",".","Length","should","be","9",",","i.e.",",","``","num_blocks","+","1","``","(","because","there","is","a","stem","width","in","front",")",".","width_multipliers","A","range","of","widths","multiplier","to","choose","from",".","The","choice","is","independent","for","each","stage",".","Or","it","can","be","a","fixed","float",".","This","will","be","applied","on","``","base_widths","``",",","and","we","would","also","make","sure","that","widths","can","be","divided","by","8.","expand_ratios","A","list","of","expand","ratios","to","choose","from",".","Independent","for","every","*","*","block","*","*",".","squeeze_excite","Indicating","whether","the","current","stage","can","have","an","optional","SE","layer",".","Expect","array","of","length","6","for","stage","0","to","5",".","Each","element","can","be","one","of","``","force","``",",","``","optional","``",",","``","none","``",".","depth_range","A","range","(","e.g.",",","``","(","1",",","4",")","``",")",",","or","a","list","of","range","(","e.g.",",","``","[","(","1",",","3",")",",","(","1",",","4",")",",","(","1",",","4",")",",","(","1",",","3",")",",","(","0",",","2",")","]","``",")",".","If","a","list",",","the","length","should","be","5",".","The","depth","are","specified","for","stage","1","to","5.","stride","Stride","for","all","stages","(","including","stem","and","head",")",".","Length","should","be","same","as","``","base_widths","``",".","activation","Activation","(","class",")","for","all","stages",".","Length","is","same","as","``","base_widths","``",".","se_from_exp","Calculate","SE","channel","reduction","from","expanded","(","mid",")","channels",".","dropout_rate","Dropout","rate","at","classification","head",".","bn_eps","Epsilon","of","batch","normalization",".","bn_momentum","Momentum","of","batch","normalization","."]
code: def load_searched_model(
      cls,
      name: str,
      pretrained: bool = False,
      download: bool = False,
      progress: bool = True,
  ) -> nn.Module:
      init_kwargs = {}  # all default
  
      if name == "mobilenetv3-large-100":
          # NOTE: Use bicsubic interpolation to evaluate this
          # With default interpolation, it yields top-1 75.722
          arch = {
              "stem_ks": 3,
              "s0_i0_ks": 3,
              "s1_depth": 2,
              "s1_i0_exp": 4,
              "s1_i0_ks": 3,
              "s1_i1_exp": 3,
              "s1_i1_ks": 3,
              "s2_depth": 3,
              "s2_i0_exp": 3,
              "s2_i0_ks": 5,
              "s2_i1_exp": 3,
              "s2_i1_ks": 5,
              "s2_i2_exp": 3,
              "s2_i2_ks": 5,
              "s3_depth": 4,
              "s3_i0_exp": 6,
              "s3_i0_ks": 3,
              "s3_i1_exp": 2.5,
              "s3_i1_ks": 3,
              "s3_i2_exp": 2.3,
              "s3_i2_ks": 3,
              "s3_i3_exp": 2.3,
              "s3_i3_ks": 3,
              "s4_depth": 2,
              "s4_i0_exp": 6,
              "s4_i0_ks": 3,
              "s4_i1_exp": 6,
              "s4_i1_ks": 3,
              "s5_depth": 3,
              "s5_i0_exp": 6,
              "s5_i0_ks": 5,
              "s5_i1_exp": 6,
              "s5_i1_ks": 5,
              "s5_i2_exp": 6,
              "s5_i2_ks": 5,
          }
  
          init_kwargs.update(
              base_widths=[16, 16, 24, 40, 80, 112, 160, 960, 1280],
              expand_ratios=[1.0, 2.0, 2.3, 2.5, 3.0, 4.0, 6.0],
              bn_eps=1e-5,
              bn_momentum=0.1,
              width_multipliers=1.0,
              squeeze_excite=[
                  "none",
                  "none",
                  "force",
                  "none",
                  "force",
                  "force",
              ],
          )
  
      elif name.startswith("mobilenetv3-small-"):
          # Evaluate with bicubic interpolation
          multiplier = int(name.split("-")[-1]) / 100
          widths = [16, 16, 24, 40, 48, 96, 576, 1024]
          for i in range(7):
              if i > 0 or multiplier >= 0.75:
                  # fix_stem = True when multiplier < 0.75
                  # https://github.com/rwightman/pytorch-image-models/blob/b7cb8d03/timm/models/mobilenetv3.py#L421
                  widths[i] = make_divisible(widths[i] * multiplier, 8)
          init_kwargs.update(
              base_widths=widths,
              width_multipliers=1.0,
              expand_ratios=[3.0, 3.67, 4.0, 4.5, 6.0],
              bn_eps=1e-05,
              bn_momentum=0.1,
              squeeze_excite=[
                  "force",
                  "none",
                  "force",
                  "force",
                  "force",
              ],
              activation=[
                  "hswish",
                  "relu",
                  "relu",
                  "hswish",
                  "hswish",
                  "hswish",
                  "hswish",
                  "hswish",
              ],
              stride=[2, 2, 2, 2, 1, 2, 1, 1],
              depth_range=(1, 2),
          )
  
          arch = {
              "stem_ks": 3,
              "s0_i0_ks": 3,
              "s1_depth": 2,
              "s1_i0_exp": 4.5,
              "s1_i0_ks": 3,
              "s1_i1_exp": 3.67,
              "s1_i1_ks": 3,
              "s2_depth": 3,
              "s2_i0_exp": 4.0,
              "s2_i0_ks": 5,
              "s2_i1_exp": 6.0,
              "s2_i1_ks": 5,
              "s2_i2_exp": 6.0,
              "s2_i2_ks": 5,
              "s3_depth": 2,
              "s3_i0_exp": 3.0,
              "s3_i0_ks": 5,
              "s3_i1_exp": 3.0,
              "s3_i1_ks": 5,
              "s4_depth": 3,
              "s4_i0_exp": 6.0,
              "s4_i0_ks": 5,
              "s4_i1_exp": 6.0,
              "s4_i1_ks": 5,
              "s4_i2_exp": 6.0,
              "s4_i2_ks": 5,
          }
  
      elif name.startswith("cream"):
          # https://github.com/microsoft/Cream/tree/main/Cream
          # bilinear interpolation
  
          level = name.split("-")[-1]
  
          # region cream arch specification
          if level == "014":
              arch = {
                  "stem_ks": 3,
                  "s0_depth": 1,
                  "s0_i0_ks": 3,
                  "s1_depth": 1,
                  "s1_i0_exp": 4.0,
                  "s1_i0_ks": 3,
                  "s2_depth": 2,
                  "s2_i0_exp": 6.0,
                  "s2_i0_ks": 5,
                  "s2_i1_exp": 6.0,
                  "s2_i1_ks": 5,
                  "s3_depth": 2,
                  "s3_i0_exp": 6.0,
                  "s3_i0_ks": 5,
                  "s3_i1_exp": 6.0,
                  "s3_i1_ks": 5,
                  "s4_depth": 1,
                  "s4_i0_exp": 6.0,
                  "s4_i0_ks": 3,
                  "s5_depth": 1,
                  "s5_i0_exp": 6.0,
                  "s5_i0_ks": 5,
              }
          elif level == "043":
              arch = {
                  "stem_ks": 3,
                  "s0_depth": 1,
                  "s0_i0_ks": 3,
                  "s1_depth": 1,
                  "s1_i0_exp": 4.0,
                  "s1_i0_ks": 3,
                  "s2_depth": 2,
                  "s2_i0_exp": 6.0,
                  "s2_i0_ks": 5,
                  "s2_i1_exp": 6.0,
                  "s2_i1_ks": 3,
                  "s3_depth": 2,
                  "s3_i0_exp": 6.0,
                  "s3_i0_ks": 5,
                  "s3_i1_exp": 6.0,
                  "s3_i1_ks": 3,
                  "s4_depth": 3,
                  "s4_i0_exp": 6.0,
                  "s4_i0_ks": 5,
                  "s4_i1_exp": 6.0,
                  "s4_i1_ks": 5,
                  "s4_i2_exp": 6.0,
                  "s4_i2_ks": 5,
                  "s5_depth": 2,
                  "s5_i0_exp": 6.0,
                  "s5_i0_ks": 5,
                  "s5_i1_exp": 6.0,
                  "s5_i1_ks": 5,
              }
          elif level == "114":
              arch = {
                  "stem_ks": 3,
                  "s0_depth": 1,
                  "s0_i0_ks": 3,
                  "s1_depth": 1,
                  "s1_i0_exp": 4.0,
                  "s1_i0_ks": 3,
                  "s2_depth": 2,
                  "s2_i0_exp": 6.0,
                  "s2_i0_ks": 5,
                  "s2_i1_exp": 6.0,
                  "s2_i1_ks": 5,
                  "s3_depth": 2,
                  "s3_i0_exp": 6.0,
                  "s3_i0_ks": 5,
                  "s3_i1_exp": 6.0,
                  "s3_i1_ks": 5,
                  "s4_depth": 3,
                  "s4_i0_exp": 6.0,
                  "s4_i0_ks": 5,
                  "s4_i1_exp": 6.0,
                  "s4_i1_ks": 5,
                  "s4_i2_exp": 6.0,
                  "s4_i2_ks": 5,
                  "s5_depth": 2,
                  "s5_i0_exp": 6.0,
                  "s5_i0_ks": 5,
                  "s5_i1_exp": 6.0,
                  "s5_i1_ks": 5,
              }
          elif level == "287":
              arch = {
                  "stem_ks": 3,
                  "s0_depth": 1,
                  "s0_i0_ks": 3,
                  "s1_depth": 1,
                  "s1_i0_exp": 4.0,
                  "s1_i0_ks": 3,
                  "s2_depth": 2,
                  "s2_i0_exp": 6.0,
                  "s2_i0_ks": 5,
                  "s2_i1_exp": 6.0,
                  "s2_i1_ks": 5,
                  "s3_depth": 3,
                  "s3_i0_exp": 6.0,
                  "s3_i0_ks": 5,
                  "s3_i1_exp": 6.0,
                  "s3_i1_ks": 3,
                  "s3_i2_exp": 6.0,
                  "s3_i2_ks": 5,
                  "s4_depth": 4,
                  "s4_i0_exp": 6.0,
                  "s4_i0_ks": 5,
                  "s4_i1_exp": 6.0,
                  "s4_i1_ks": 5,
                  "s4_i2_exp": 6.0,
                  "s4_i2_ks": 5,
                  "s4_i3_exp": 6.0,
                  "s4_i3_ks": 5,
                  "s5_depth": 3,
                  "s5_i0_exp": 6.0,
                  "s5_i0_ks": 5,
                  "s5_i1_exp": 6.0,
                  "s5_i1_ks": 5,
                  "s5_i2_exp": 6.0,
                  "s5_i2_ks": 5,
              }
          elif level == "481":
              arch = {
                  "stem_ks": 3,
                  "s0_depth": 1,
                  "s0_i0_ks": 3,
                  "s1_depth": 4,
                  "s1_i0_exp": 6.0,
                  "s1_i0_ks": 5,
                  "s1_i1_exp": 4.0,
                  "s1_i1_ks": 7,
                  "s1_i2_exp": 6.0,
                  "s1_i2_ks": 5,
                  "s1_i3_exp": 6.0,
                  "s1_i3_ks": 3,
                  "s2_depth": 4,
                  "s2_i0_exp": 6.0,
                  "s2_i0_ks": 5,
                  "s2_i1_exp": 4.0,
                  "s2_i1_ks": 5,
                  "s2_i2_exp": 6.0,
                  "s2_i2_ks": 5,
                  "s2_i3_exp": 4.0,
                  "s2_i3_ks": 3,
                  "s3_depth": 5,
                  "s3_i0_exp": 6.0,
                  "s3_i0_ks": 5,
                  "s3_i1_exp": 6.0,
                  "s3_i1_ks": 5,
                  "s3_i2_exp": 6.0,
                  "s3_i2_ks": 5,
                  "s3_i3_exp": 6.0,
                  "s3_i3_ks": 3,
                  "s3_i4_exp": 6.0,
                  "s3_i4_ks": 3,
                  "s4_depth": 4,
                  "s4_i0_exp": 6.0,
                  "s4_i0_ks": 5,
                  "s4_i1_exp": 6.0,
                  "s4_i1_ks": 5,
                  "s4_i2_exp": 6.0,
                  "s4_i2_ks": 5,
                  "s4_i3_exp": 6.0,
                  "s4_i3_ks": 5,
                  "s5_depth": 4,
                  "s5_i0_exp": 6.0,
                  "s5_i0_ks": 5,
                  "s5_i1_exp": 6.0,
                  "s5_i1_ks": 5,
                  "s5_i2_exp": 6.0,
                  "s5_i2_ks": 5,
                  "s5_i3_exp": 6.0,
                  "s5_i3_ks": 5,
              }
          elif level == "604":
              arch = {
                  "stem_ks": 3,
                  "s0_depth": 1,
                  "s0_i0_ks": 3,
                  "s1_depth": 5,
                  "s1_i0_exp": 6.0,
                  "s1_i0_ks": 5,
                  "s1_i1_exp": 6.0,
                  "s1_i1_ks": 5,
                  "s1_i2_exp": 4.0,
                  "s1_i2_ks": 5,
                  "s1_i3_exp": 6.0,
                  "s1_i3_ks": 5,
                  "s1_i4_exp": 6.0,
                  "s1_i4_ks": 5,
                  "s2_depth": 5,
                  "s2_i0_exp": 6.0,
                  "s2_i0_ks": 5,
                  "s2_i1_exp": 4.0,
                  "s2_i1_ks": 5,
                  "s2_i2_exp": 6.0,
                  "s2_i2_ks": 5,
                  "s2_i3_exp": 4.0,
                  "s2_i3_ks": 5,
                  "s2_i4_exp": 6.0,
                  "s2_i4_ks": 5,
                  "s3_depth": 5,
                  "s3_i0_exp": 6.0,
                  "s3_i0_ks": 5,
                  "s3_i1_exp": 4.0,
                  "s3_i1_ks": 5,
                  "s3_i2_exp": 6.0,
                  "s3_i2_ks": 5,
                  "s3_i3_exp": 4.0,
                  "s3_i3_ks": 5,
                  "s3_i4_exp": 6.0,
                  "s3_i4_ks": 5,
                  "s4_depth": 6,
                  "s4_i0_exp": 6.0,
                  "s4_i0_ks": 5,
                  "s4_i1_exp": 6.0,
                  "s4_i1_ks": 5,
                  "s4_i2_exp": 4.0,
                  "s4_i2_ks": 5,
                  "s4_i3_exp": 4.0,
                  "s4_i3_ks": 5,
                  "s4_i4_exp": 6.0,
                  "s4_i4_ks": 5,
                  "s4_i5_exp": 6.0,
                  "s4_i5_ks": 5,
                  "s5_depth": 6,
                  "s5_i0_exp": 6.0,
                  "s5_i0_ks": 5,
                  "s5_i1_exp": 6.0,
                  "s5_i1_ks": 5,
                  "s5_i2_exp": 4.0,
                  "s5_i2_ks": 5,
                  "s5_i3_exp": 6.0,
                  "s5_i3_ks": 5,
                  "s5_i4_exp": 6.0,
                  "s5_i4_ks": 5,
                  "s5_i5_exp": 6.0,
                  "s5_i5_ks": 5,
              }
          else:
              raise ValueError(
                  f"Unsupported cream model level: {level}"
              )
          # endregion
  
          init_kwargs.update(
              base_widths=[16, 16, 24, 40, 80, 96, 192, 320, 1280],
              width_multipliers=1.0,
              expand_ratios=[4.0, 6.0],
              bn_eps=1e-5,
              bn_momentum=0.1,
              squeeze_excite=["force"] * 6,
              activation=["swish"] * 9,
          )
  
      else:
          raise ValueError(
              f"Unsupported architecture with name: {name}"
          )
  
      model_factory = cls.frozen_factory(arch)
      model = model_factory(**init_kwargs)
  
      if pretrained:
          weight_file = load_pretrained_weight(
              name, download=download, progress=progress
          )
          pretrained_weights = torch.load(weight_file)
          model.load_state_dict(pretrained_weights)
  
      return model
  
imports: from functools import partial
  from typing import Tuple, Optional, Callable, Union, List, Type, cast
  from typing_extensions import Literal
  import torch
  from torch import nn
  import nni
  from nni.nas.nn.pytorch import ModelSpace, Repeat, LayerChoice, MutableLinear, MutableConv2d
  from .proxylessnas import ConvBNReLU, InvertedResidual, DepthwiseSeparableConv, MaybeIntChoice, make_divisible, reset_parameters
  from .utils.pretrained import load_pretrained_weight
label_desc: Use image node_id 4 for calling the MobileNetV3Space obj's underlying member method code with example usage: obj.load_searched_model(cls, name, pretrained, download, progress) and returns: model