Understanding custom policies in stable-baselines3
I was trying to understand the policy networks in stable-baselines3 from this doc page.

As explained in this example, to specify custom CNN feature extractor, we extend BaseFeaturesExtractor class and specify it in policy_kwarg.features_extractor_class with first param CnnPolicy:
model = PPO(&quot;CnnPolicy&quot;, &quot;BreakoutNoFrameskip-v4&quot;, policy_kwargs=policy_kwargs)

Q1. Can we follow same approach for custom MLP feature extractor?

As explained in this example, to specify custom MLP feature extractor, we extend ActorCriticPolicy class and override _build_mlp_extractor() and pass it as first param:
class CustomActorCriticPolicy(ActorCriticPolicy): ...
model = PPO(CustomActorCriticPolicy, &quot;CartPole-v1&quot;, verbose=1)

Q2. Can we follow same approach for custom CNN feature extractor?

I feel either we can have CNN extractor or MLP extractor. So it makes no sense to pass MlpPolicy as first param to model and then specify CNN feature extractor in policy_kwarg.features_extractor_class as in this example. This result in following policy (containing both features_extractor and mlp_extractor), which I feel is incorrect:
ActorCriticPolicy(
  (features_extractor): Net(
    (conv1): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1))
    (conv2): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1))
    (fc3): Linear(in_features=384, out_features=512, bias=True)
  )
  (mlp_extractor): MlpExtractor(
    (shared_net): Sequential(
      (0): Linear(in_features=512, out_features=64, bias=True)
      (1): ReLU()
    )
    (policy_net): Sequential(
      (0): Linear(in_features=64, out_features=32, bias=True)
      (1): ReLU()
      (2): Linear(in_features=32, out_features=16, bias=True)
      (3): ReLU()
    )
    (value_net): Sequential(
      (0): Linear(in_features=64, out_features=32, bias=True)
      (1): ReLU()
      (2): Linear(in_features=32, out_features=16, bias=True)
      (3): ReLU()
    )
  )
  (action_net): Linear(in_features=16, out_features=7, bias=True)
  (value_net): Linear(in_features=16, out_features=1, bias=True)
)  

Q3. Am I correct with this understanding? If yes, then is one of the MLP or CNN feature extractor ignored?



What I can say after i went through all the library code. CnnPolicy is differ to MlpPolicy only in implemented default BaseFeatureExtraction class. This make sense only in a case when you are not trying to create your custom BaseFeatureExtraction class.
Let me try to explain, we can see two types of policies:
MlpPolicy = ActorCriticPolicy
CnnPolicy = ActorCriticCnnPolicy

And we can see that class ActorCriticCnnPolicy(ActorCriticPolicy)is just based on ActorCriticPolicy and we can see following in parameters:
features_extractor_class: Type[BaseFeaturesExtractor] = NatureCNN

where NatureCNN is a simple implementation of BaseFeatureExtractor with CNN layers.
Now lets move to your questions!
Here is one thing, we always have following structure of model:
FeatureExtractor -&gt; MlpExtractor -&gt; Policy\Value nets
You can specify feature_exctractor_class and net_arch in policy_kwargs.
Default feature_extractor_class is based on your selection of CnnPolicy or MlpPolicy, but if you specify your own class then there is no difference. So, you can just use MlpPolicy.
If you specify net_arch, for example 'net_arch':[64, dict(pi=[32, 16], vf=[32, 16])] you will have 1 dense layer for mlp_extractor that connects output from feature_extractor.
To summary, my suggestion is to follow the example that you found in your Q3.:

Specify your own feature_extractor with any structure of this network either cnn or dense or whatever you want. The output of feature_extractor should be FC layer.
Specify net_acrh parameter in the following structure: [x1, x2, ..., dict(pi=[px1, px2, ..], vf=[vx1, vx2, ...])] where the first part is specified size of FC layers for mlp_extractor and the second part (inside dict) corresponds to sizes of policy and value networks.

