lib/rumale/optimizer/nadam.rb
def call(weight, gradient)
        @fst_moment ||= Numo::DFloat.zeros(weight.shape[0])
        @sec_moment ||= Numo::DFloat.zeros(weight.shape[0])

        @iter += 1

        decay1_curr = @params[:decay1] * (1.0 - 0.5 * 0.96**(@iter * 0.004))
        decay1_next = @params[:decay1] * (1.0 - 0.5 * 0.96**((@iter + 1) * 0.004))
        decay1_prod_curr = @decay1_prod * decay1_curr
        decay1_prod_next = @decay1_prod * decay1_curr * decay1_next
        @decay1_prod = decay1_prod_curr

        @fst_moment = @params[:decay1] * @fst_moment + (1.0 - @params[:decay1]) * gradient
        @sec_moment = @params[:decay2] * @sec_moment + (1.0 - @params[:decay2]) * gradient**2
        nm_gradient = gradient / (1.0 - decay1_prod_curr)
        nm_fst_moment = @fst_moment / (1.0 - decay1_prod_next)
        nm_sec_moment = @sec_moment / (1.0 - @params[:decay2]**@iter)

        weight - (@params[:learning_rate] / (nm_sec_moment**0.5 + 1e-8)) * ((1 - decay1_curr) * nm_gradient + decay1_next * nm_fst_moment)
      end
# Calculate the updated weight with Nadam adaptive learning rate.
#
# @param weight [Numo::DFloat] (shape: [n_features]) The weight to be updated.
# @param gradient [Numo::DFloat] (shape: [n_features]) The gradient for updating the weight.
# @return [Numo::DFloat] (shape: [n_feautres]) The updated weight.
Calculate the updated weight with Nadam adaptive learning rate.
Calculate the updated weight with Nadam adaptive learning rate .
def call(weight, gradient)
        @fst_moment ||= Numo::DFloat.zeros(weight.shape[0])
        @sec_moment ||= Numo::DFloat.zeros(weight.shape[0])
        @iter += 1
        decay1_curr = @params[:decay1] * (1.0 - 0.5 * 0.96**(@iter * 0.004))
        decay1_next = @params[:decay1] * (1.0 - 0.5 * 0.96**((@iter + 1) * 0.004))
        decay1_prod_curr = @decay1_prod * decay1_curr
        decay1_prod_next = @decay1_prod * decay1_curr * decay1_next
        @decay1_prod = decay1_prod_curr
        @fst_moment = @params[:decay1] * @fst_moment + (1.0 - @params[:decay1]) * gradient
        @sec_moment = @params[:decay2] * @sec_moment + (1.0 - @params[:decay2]) * gradient**2
        nm_gradient = gradient / (1.0 - decay1_prod_curr)
        nm_fst_moment = @fst_moment / (1.0 - decay1_prod_next)
        nm_sec_moment = @sec_moment / (1.0 - @params[:decay2]**@iter)
        weight - (@params[:learning_rate] / (nm_sec_moment**0.5 + 1e-8)) * ((1 - decay1_curr) * nm_gradient + decay1_next * nm_fst_moment)
      end
def call ( weight , gradient ) @fst_moment ||= Numo :: DFloat . zeros ( weight . shape [ 0 ] ) @sec_moment ||= Numo :: DFloat . zeros ( weight . shape [ 0 ] ) @iter += 1 decay1_curr = @params [ :decay1 ] * ( 1.0 - 0.5 * 0.96 ** ( @iter * 0.004 ) ) decay1_next = @params [ :decay1 ] * ( 1.0 - 0.5 * 0.96 ** ( ( @iter + 1 ) * 0.004 ) ) decay1_prod_curr = @decay1_prod * decay1_curr decay1_prod_next = @decay1_prod * decay1_curr * decay1_next @decay1_prod = decay1_prod_curr @fst_moment = @params [ :decay1 ] * @fst_moment + ( 1.0 - @params [ :decay1 ] ) * gradient @sec_moment = @params [ :decay2 ] * @sec_moment + ( 1.0 - @params [ :decay2 ] ) * gradient ** 2 nm_gradient = gradient / ( 1.0 - decay1_prod_curr ) nm_fst_moment = @fst_moment / ( 1.0 - decay1_prod_next ) nm_sec_moment = @sec_moment / ( 1.0 - @params [ :decay2 ] ** @iter ) weight - ( @params [ :learning_rate ] / ( nm_sec_moment ** 0.5 + 1e-8 ) ) * ( ( 1 - decay1_curr ) * nm_gradient + decay1_next * nm_fst_moment ) end
Calculate the updated weight with Nadam adaptive learning rate.
Calculate the updated weight with Nadam adaptive learning rate .
[n_feautres]) The updated weight.
[ n_feautres ] ) The updated weight .
Numo::DFloat
weight
[n_features]) The weight to be updated.
[ n_features ] ) The weight to be updated .
gradient
[n_features]) The gradient for updating the weight.
[ n_features ] ) The gradient for updating the weight .