class_docstr: Implements the L-BFGS algorithm. Compatible with multi-batch and full-overlap
  L-BFGS implementations and (stochastic) Powell damping. Partly based on the
  original L-BFGS implementation in PyTorch, Mark Schmidt's minFunc MATLAB code,
  and Michael Overton's weak Wolfe line search MATLAB code.
  
  Implemented by: Hao-Jun Michael Shi and Dheevatsa Mudigere
  Last edited 12/6/18.
  
  Warnings:
    . Does not support per-parameter options and parameter groups.
    . All parameters have to be on a single device.
  
  Inputs:
      lr (float): steplength or learning rate (default: 1)
      history_size (int): update history size (default: 10)
      line_search (str): designates line search to use (default: 'Wolfe')
          Options:
              'None': uses steplength designated in algorithm
              'Armijo': uses Armijo backtracking line search
              'Wolfe': uses Armijo-Wolfe bracketing line search
      dtype: data type (default: torch.float)
      debug (bool): debugging mode
  
  References:
  [1] Berahas, Albert S., Jorge Nocedal, and Martin Takác. "A Multi-Batch L-BFGS
      Method for Machine Learning." Advances in Neural Information Processing
      Systems. 2016.
  [2] Bollapragada, Raghu, et al. "A Progressive Batching L-BFGS Method for Machine
      Learning." International Conference on Machine Learning. 2018.
  [3] Lewis, Adrian S., and Michael L. Overton. "Nonsmooth Optimization via Quasi-Newton
      Methods." Mathematical Programming 141.1-2 (2013): 135-163.
  [4] Liu, Dong C., and Jorge Nocedal. "On the Limited Memory BFGS Method for
      Large Scale Optimization." Mathematical Programming 45.1-3 (1989): 503-528.
  [5] Nocedal, Jorge. "Updating Quasi-Newton Matrices With Limited Storage."
      Mathematics of Computation 35.151 (1980): 773-782.
  [6] Nocedal, Jorge, and Stephen J. Wright. "Numerical Optimization." Springer New York,
      2006.
  [7] Schmidt, Mark. "minFunc: Unconstrained Differentiable Multivariate Optimization
      in Matlab." Software available at http://www.cs.ubc.ca/~schmidtm/Software/minFunc.html
      (2005).
  [8] Schraudolph, Nicol N., Jin Yu, and Simon Günter. "A Stochastic Quasi-Newton
      Method for Online Convex Optimization." Artificial Intelligence and Statistics.
      2007.
  [9] Wang, Xiao, et al. "Stochastic Quasi-Newton Methods for Nonconvex Stochastic
      Optimization." SIAM Journal on Optimization 27.2 (2017): 927-956.
docstr: Performs two-loop recursion on given vector to obtain Hv.
  
  Inputs:
      vec (tensor): 1-D tensor to apply two-loop recursion to
  
  Output:
      r (tensor): matrix-vector product Hv
docstr_tok: ["Performs","two-loop","recursion","on","given","vector","to","obtain","Hv",".","Inputs",":","vec","(","tensor",")",":","1-D","tensor","to","apply","two-loop","recursion","to","Output",":","r","(","tensor",")",":","matrix-vector","product","Hv"]
code: def two_loop_recursion(self, vec):
      """
      Performs two-loop recursion on given vector to obtain Hv.
  
      Inputs:
          vec (tensor): 1-D tensor to apply two-loop recursion to
  
      Output:
          r (tensor): matrix-vector product Hv
  
      """
  
      group = self.param_groups[0]
      history_size = group["history_size"]
  
      state = self.state["global_state"]
      old_dirs = state.get("old_dirs")  # change in gradients
      old_stps = state.get("old_stps")  # change in iterates
      H_diag = state.get("H_diag")
  
      # compute the product of the inverse Hessian approximation and the gradient
      num_old = len(old_dirs)
  
      if "rho" not in state:
          state["rho"] = [None] * history_size
          state["alpha"] = [None] * history_size
      rho = state["rho"]
      alpha = state["alpha"]
  
      for i in range(num_old):
          rho[i] = 1.0 / old_stps[i].dot(old_dirs[i])
  
      q = vec
      for i in range(num_old - 1, -1, -1):
          alpha[i] = old_dirs[i].dot(q) * rho[i]
          q.add_(old_stps[i], alpha=-alpha[i])
  
      # multiply by initial Hessian
      # r/d is the final direction
      r = torch.mul(q, H_diag)
      for i in range(num_old):
          beta = old_stps[i].dot(r) * rho[i]
          r.add_(old_dirs[i], alpha=(alpha[i] - beta))
  
      return r
  
imports: from copy import deepcopy
  from functools import reduce
  import matplotlib.pyplot
  import numpy
  import torch
  from torch.optim import Optimizer
label_desc: Use image node_id 8 for calling the LBFGS obj's underlying member method code with example usage: obj.two_loop_recursion(vec) and returns: r