@gradient_optimizer<EOL>def adam(lr=<NUM_LIT>, beta=(<NUM_LIT>, <NUM_LIT>), epsilon=<NUM_LIT>):
xk = yield<EOL>mk = np.zeros_like(xk)<EOL>vk = np.zeros_like(xk)<EOL>b1, b2 = beta<EOL>for k in count(start=<NUM_LIT:1>):<EOL><INDENT>grad = yield xk<EOL>mk *= b1<EOL>mk += (<NUM_LIT:1.> - b1) * grad<EOL>vk *= b2<EOL>vk += (<NUM_LIT:1.> - b2) * (grad ** <NUM_LIT:2>)<EOL>momentum_norm = mk / (<NUM_LIT:1> - b1 ** k)<EOL>velocity_norm = np.sqrt(vk / (<NUM_LIT:1> - b2 ** k))<EOL>xk -= lr * momentum_norm / (epsilon + velocity_norm)<EOL><DEDENT>
ADAM

Parameters
----------
lr : float, optional
    Learnin rate (Default: 1e-3)

beta : (float, float)
    (Default: (0.9, 0.999))

epsilon : float
    (Default: 1e-8)