assignment3_coding/starter_code 2/code/policy_network.py
def build_policy_network_op(self, scope = "policy_network"):
    """
    Build the policy network, construct the tensorflow operation to sample
    actions from the policy network outputs, and compute the log probabilities
    of the actions taken (for computing the loss later). These operations are
    stored in self.sampled_action and self.logprob. Must handle both settings
    of self.discrete.

    Args:
            scope: the scope of the neural network

    TODO:
    Discrete case:
        action_logits: the logits for each action
            HINT: use build_mlp, check self.config for layer_size and
            n_layers
        self.sampled_action: sample from these logits
            HINT: use tf.multinomial + tf.squeeze
        self.logprob: compute the log probabilities of the taken actions
            HINT: 1. tf.nn.sparse_softmax_cross_entropy_with_logits computes
                     the *negative* log probabilities of labels, given logits.
                  2. taken actions are different than sampled actions!

    Continuous case:
        To build a policy in a continuous action space domain, we will have the
        model output the means of each action dimension, and then sample from
        a multivariate normal distribution with these means and trainable standard
        deviation.

        That is, the action a_t ~ N( mu(o_t), sigma)
        where mu(o_t) is the network that outputs the means for each action
        dimension, and sigma is a trainable variable for the standard deviations.
        N here is a multivariate gaussian distribution with the given parameters.

        action_means: the predicted means for each action dimension.
            HINT: use build_mlp, check self.config for layer_size and
            n_layers
        log_std: a trainable variable for the log standard deviations.
            HINT: think about why we use log std as the trainable variable instead of std
            HINT: use tf.get_variable
            HINT: The shape of this should match the shape of action dimension
        self.sampled_action: sample from the gaussian distribution as described above
            HINT: use tf.random_normal
            HINT: use re-parametrization to obtain N(mu, sigma) from N(0, 1)
        self.logprob: the log probabilities of the taken actions
            HINT: use tf.contrib.distributions.MultivariateNormalDiag

    """
    #######################################################
    #########   YOUR CODE HERE - 8-12 lines.   ############

    # logprob: log π_θ(a_t|s_t)
    if self.discrete:
        action_logits = build_mlp(self.observation_placeholder,
                                  self.action_dim,
                                  scope,
                                  self.config.n_layers,
                                  self.config.layer_size,
                                  self.config.activation)
        self.sampled_action = tf.squeeze(tf.multinomial(action_logits, 1))
        self.logprob = - tf.nn.sparse_softmax_cross_entropy_with_logits(labels=self.action_placeholder,
                                                                        logits=action_logits)
    else:
        action_means = build_mlp(self.observation_placeholder,
                                  self.action_dim,
                                  scope,
                                  self.config.n_layers,
                                  self.config.layer_size,
                                  self.config.activation)
        log_std = tf.get_variable("log_std", shape=(1, self.action_dim))
        self.sampled_action = tf.random_normal((1,), mean=action_means, stddev=log_std)
        self.logprob = tfp.distributions.MultivariateNormalDiag(action_means, log_std)

    #######################################################
    #########          END YOUR CODE.          ############

    Build the policy network, construct the tensorflow operation to sample
    actions from the policy network outputs, and compute the log probabilities
    of the actions taken (for computing the loss later). These operations are
    stored in self.sampled_action and self.logprob. Must handle both settings
    of self.discrete.

    Args:
            scope: the scope of the neural network

    TODO:
    Discrete case:
        action_logits: the logits for each action
            HINT: use build_mlp, check self.config for layer_size and
            n_layers
        self.sampled_action: sample from these logits
            HINT: use tf.multinomial + tf.squeeze
        self.logprob: compute the log probabilities of the taken actions
            HINT: 1. tf.nn.sparse_softmax_cross_entropy_with_logits computes
                     the *negative* log probabilities of labels, given logits.
                  2. taken actions are different than sampled actions!

    Continuous case:
        To build a policy in a continuous action space domain, we will have the
        model output the means of each action dimension, and then sample from
        a multivariate normal distribution with these means and trainable standard
        deviation.

        That is, the action a_t ~ N( mu(o_t), sigma)
        where mu(o_t) is the network that outputs the means for each action
        dimension, and sigma is a trainable variable for the standard deviations.
        N here is a multivariate gaussian distribution with the given parameters.

        action_means: the predicted means for each action dimension.
            HINT: use build_mlp, check self.config for layer_size and
            n_layers
        log_std: a trainable variable for the log standard deviations.
            HINT: think about why we use log std as the trainable variable instead of std
            HINT: use tf.get_variable
            HINT: The shape of this should match the shape of action dimension
        self.sampled_action: sample from the gaussian distribution as described above
            HINT: use tf.random_normal
            HINT: use re-parametrization to obtain N(mu, sigma) from N(0, 1)
        self.logprob: the log probabilities of the taken actions
            HINT: use tf.contrib.distributions.MultivariateNormalDiag

    
Build the policy network, construct the tensorflow operation to sample
actions from the policy network outputs, and compute the log probabilities
of the actions taken (for computing the loss later).
Build the policy network construct the tensorflow operation to sample actions from the policy network outputs and compute the log probabilities of the actions taken ( for computing the loss later ) .
def build_policy_network_op(self, scope = "policy_network"):
    if self.discrete:
        action_logits = build_mlp(self.observation_placeholder,
                                  self.action_dim,
                                  scope,
                                  self.config.n_layers,
                                  self.config.layer_size,
                                  self.config.activation)
        self.sampled_action = tf.squeeze(tf.multinomial(action_logits, 1))
        self.logprob = - tf.nn.sparse_softmax_cross_entropy_with_logits(labels=self.action_placeholder,
                                                                        logits=action_logits)
    else:
        action_means = build_mlp(self.observation_placeholder,
                                  self.action_dim,
                                  scope,
                                  self.config.n_layers,
                                  self.config.layer_size,
                                  self.config.activation)
        log_std = tf.get_variable("log_std", shape=(1, self.action_dim))
        self.sampled_action = tf.random_normal((1,), mean=action_means, stddev=log_std)
        self.logprob = tfp.distributions.MultivariateNormalDiag(action_means, log_std)
def build_policy_network_op ( self , scope = "policy_network" ) : if self . discrete : action_logits = build_mlp ( self . observation_placeholder , self . action_dim , scope , self . config . n_layers , self . config . layer_size , self . config . activation ) self . sampled_action = tf . squeeze ( tf . multinomial ( action_logits , 1 ) ) self . logprob = - tf . nn . sparse_softmax_cross_entropy_with_logits ( labels = self . action_placeholder , logits = action_logits ) else : action_means = build_mlp ( self . observation_placeholder , self . action_dim , scope , self . config . n_layers , self . config . layer_size , self . config . activation ) log_std = tf . get_variable ( "log_std" , shape = ( 1 , self . action_dim ) ) self . sampled_action = tf . random_normal ( ( 1 , ) , mean = action_means , stddev = log_std ) self . logprob = tfp . distributions . MultivariateNormalDiag ( action_means , log_std )
Build the policy network, construct the tensorflow operation to sample
actions from the policy network outputs, and compute the log probabilities
of the actions taken (for computing the loss later).
Build the policy network construct the tensorflow operation to sample actions from the policy network outputs and compute the log probabilities of the actions taken ( for computing the loss later ) .
"""
    Build the policy network, construct the tensorflow operation to sample
    actions from the policy network outputs, and compute the log probabilities
    of the actions taken (for computing the loss later). These operations are
    stored in self.sampled_action and self.logprob. Must handle both settings
    of self.discrete.

    Args:
            scope: the scope of the neural network

    TODO:
    Discrete case:
        action_logits: the logits for each action
            HINT: use build_mlp, check self.config for layer_size and
            n_layers
        self.sampled_action: sample from these logits
            HINT: use tf.multinomial + tf.squeeze
        self.logprob: compute the log probabilities of the taken actions
            HINT: 1. tf.nn.sparse_softmax_cross_entropy_with_logits computes
                     the *negative* log probabilities of labels, given logits.
                  2. taken actions are different than sampled actions!

    Continuous case:
        To build a policy in a continuous action space domain, we will have the
        model output the means of each action dimension, and then sample from
        a multivariate normal distribution with these means and trainable standard
        deviation.

        That is, the action a_t ~ N( mu(o_t), sigma)
        where mu(o_t) is the network that outputs the means for each action
        dimension, and sigma is a trainable variable for the standard deviations.
        N here is a multivariate gaussian distribution with the given parameters.

        action_means: the predicted means for each action dimension.
            HINT: use build_mlp, check self.config for layer_size and
            n_layers
        log_std: a trainable variable for the log standard deviations.
            HINT: think about why we use log std as the trainable variable instead of std
            HINT: use tf.get_variable
            HINT: The shape of this should match the shape of action dimension
        self.sampled_action: sample from the gaussian distribution as described above
            HINT: use tf.random_normal
            HINT: use re-parametrization to obtain N(mu, sigma) from N(0, 1)
        self.logprob: the log probabilities of the taken actions
            HINT: use tf.contrib.distributions.MultivariateNormalDiag

    """ ####################################################### #########   YOUR CODE HERE - 8-12 lines.   ############ # logprob: log π_θ(a_t|s_t) ####################################################### #########          END YOUR CODE.          ############
self
scope
the scope of the neural network
the scope of the neural network