class_docstr: Implements the L-BFGS algorithm. Compatible with multi-batch and full-overlap
  L-BFGS implementations and (stochastic) Powell damping. Partly based on the
  original L-BFGS implementation in PyTorch, Mark Schmidt's minFunc MATLAB code,
  and Michael Overton's weak Wolfe line search MATLAB code.
  
  Implemented by: Hao-Jun Michael Shi and Dheevatsa Mudigere
  Last edited 12/6/18.
  
  Warnings:
    . Does not support per-parameter options and parameter groups.
    . All parameters have to be on a single device.
  
  Inputs:
      lr (float): steplength or learning rate (default: 1)
      history_size (int): update history size (default: 10)
      line_search (str): designates line search to use (default: 'Wolfe')
          Options:
              'None': uses steplength designated in algorithm
              'Armijo': uses Armijo backtracking line search
              'Wolfe': uses Armijo-Wolfe bracketing line search
      dtype: data type (default: torch.float)
      debug (bool): debugging mode
  
  References:
  [1] Berahas, Albert S., Jorge Nocedal, and Martin Takác. "A Multi-Batch L-BFGS
      Method for Machine Learning." Advances in Neural Information Processing
      Systems. 2016.
  [2] Bollapragada, Raghu, et al. "A Progressive Batching L-BFGS Method for Machine
      Learning." International Conference on Machine Learning. 2018.
  [3] Lewis, Adrian S., and Michael L. Overton. "Nonsmooth Optimization via Quasi-Newton
      Methods." Mathematical Programming 141.1-2 (2013): 135-163.
  [4] Liu, Dong C., and Jorge Nocedal. "On the Limited Memory BFGS Method for
      Large Scale Optimization." Mathematical Programming 45.1-3 (1989): 503-528.
  [5] Nocedal, Jorge. "Updating Quasi-Newton Matrices With Limited Storage."
      Mathematics of Computation 35.151 (1980): 773-782.
  [6] Nocedal, Jorge, and Stephen J. Wright. "Numerical Optimization." Springer New York,
      2006.
  [7] Schmidt, Mark. "minFunc: Unconstrained Differentiable Multivariate Optimization
      in Matlab." Software available at http://www.cs.ubc.ca/~schmidtm/Software/minFunc.html
      (2005).
  [8] Schraudolph, Nicol N., Jin Yu, and Simon Günter. "A Stochastic Quasi-Newton
      Method for Online Convex Optimization." Artificial Intelligence and Statistics.
      2007.
  [9] Wang, Xiao, et al. "Stochastic Quasi-Newton Methods for Nonconvex Stochastic
      Optimization." SIAM Journal on Optimization 27.2 (2017): 927-956.
docstr: Performs curvature update.
  
  Inputs:
      flat_grad (tensor): 1-D tensor of flattened gradient for computing
          gradient difference with previously stored gradient
      eps (float): constant for curvature pair rejection or damping (default: 1e-2)
      damping (bool): flag for using Powell damping (default: False)
docstr_tok: ["Performs","curvature","update",".","Inputs",":","flat_grad","(","tensor",")",":","1-D","tensor","of","flattened","gradient","for","computing","gradient","difference","with","previously","stored","gradient","eps","(","float",")",":","constant","for","curvature","pair","rejection","or","damping","(","default",":","1e-2",")","damping","(","bool",")",":","flag","for","using","Powell","damping","(","default",":","False",")"]
code: def curvature_update(self, flat_grad, eps=1e-2, damping=False):
      """
      Performs curvature update.
  
      Inputs:
          flat_grad (tensor): 1-D tensor of flattened gradient for computing
              gradient difference with previously stored gradient
          eps (float): constant for curvature pair rejection or damping (default: 1e-2)
          damping (bool): flag for using Powell damping (default: False)
      """
  
      assert len(self.param_groups) == 1
  
      # load parameters
      if eps <= 0:
          raise (ValueError("Invalid eps; must be positive."))
  
      group = self.param_groups[0]
      history_size = group["history_size"]
      debug = group["debug"]
  
      # variables cached in state (for tracing)
      state = self.state["global_state"]
      fail = state.get("fail")
  
      # check if line search failed
      if not fail:
          d = state.get("d")
          t = state.get("t")
          old_dirs = state.get("old_dirs")
          old_stps = state.get("old_stps")
          H_diag = state.get("H_diag")
          prev_flat_grad = state.get("prev_flat_grad")
          Bs = state.get("Bs")
  
          # compute y's
          y = flat_grad.sub(prev_flat_grad)
          s = d.mul(t)
          sBs = s.dot(Bs)
          ys = y.dot(s)  # y*s
  
          # update L-BFGS matrix
          if ys > eps * sBs or damping == True:
              # perform Powell damping
              if damping == True and ys < eps * sBs:
                  if debug:
                      print("Applying Powell damping...")
                  theta = ((1 - eps) * sBs) / (sBs - ys)
                  y = theta * y + (1 - theta) * Bs
  
              # updating memory
              if len(old_dirs) == history_size:
                  # shift history by one (limited-memory)
                  old_dirs.pop(0)
                  old_stps.pop(0)
  
              # store new direction/step
              old_dirs.append(s)
              old_stps.append(y)
  
              # update scale of initial Hessian approximation
              H_diag = ys / y.dot(y)  # (y*y)
  
              state["old_dirs"] = old_dirs
              state["old_stps"] = old_stps
              state["H_diag"] = H_diag
  
          else:
              # save skip
              state["curv_skips"] += 1
              if debug:
                  print(
                      "Curvature pair skipped due to failed criterion"
                  )
  
      else:
          # save skip
          state["fail_skips"] += 1
          if debug:
              print("Line search failed; curvature pair update skipped")
  
      return
  
imports: from copy import deepcopy
  from functools import reduce
  import matplotlib.pyplot
  import numpy
  import torch
  from torch.optim import Optimizer
label_desc: Use image node_id 9 for calling the LBFGS obj's underlying member method code with example usage: obj.curvature_update(flat_grad, eps, damping) without return types