code: def minimize(
      value_and_gradients_function: Callable[
          [types.RealTensor], Tuple[types.RealTensor, types.RealTensor]
      ],
      initial_position: types.RealTensor,
      tolerance: types.RealTensor = 1e-8,
      x_tolerance: types.RealTensor = 0,
      f_relative_tolerance: types.RealTensor = 0,
      max_iterations: types.IntTensor = 50,
      parallel_iterations: types.IntTensor = 1,
      stopping_condition: Callable[
          [types.BoolTensor, types.BoolTensor], types.BoolTensor
      ] = None,
      params: ConjugateGradientParams = None,
      name: str = None,
  ) -> OptimizerResult:
      """Minimizes a differentiable function.
  
      Implementation of algorithm described in [HZ2006]. Updated formula for next
      search direction were taken from [HZ2013].
  
      Supports batches with 1-dimensional batch shape.
  
      #### References:
      [HZ2006] Hager, William W., and Hongchao Zhang. "Algorithm 851: CG_DESCENT,
        a conjugate gradient method with guaranteed descent."
        http://users.clas.ufl.edu/hager/papers/CG/cg_compare.pdf
      [HZ2013] W. W. Hager and H. Zhang (2013) The limited memory conjugate gradient
        method.
        https://pdfs.semanticscholar.org/8769/69f3911777e0ff0663f21b67dff30518726b.pdf
  
      ### Usage:
      The following example demonstrates this optimizer attempting to find the
      minimum for a simple two dimensional quadratic objective function.
  
      ```python
        minimum = np.array([1.0, 1.0])  # The center of the quadratic bowl.
        scales = np.array([2.0, 3.0])  # The scales along the two axes.
  
        # The objective function and the gradient.
        def quadratic(x):
          value = tf.reduce_sum(scales * (x - minimum) ** 2)
          return value, tf.gradients(value, x)[0]
  
        start = tf.constant([0.6, 0.8])  # Starting point for the search.
        optim_results = conjugate_gradient.minimize(
            quadratic, initial_position=start, tolerance=1e-8)
  
        with tf.Session() as session:
          results = session.run(optim_results)
          # Check that the search converged
          assert(results.converged)
          # Check that the argmin is close to the actual value.
          np.testing.assert_allclose(results.position, minimum)
      ```
  
      Args:
        value_and_gradients_function:  A Python callable that accepts a point as a
          real `Tensor` and returns a tuple of `Tensor`s of real dtype containing
          the value of the function and its gradient at that point. The function to
          be minimized. The input should be of shape `[..., n]`, where `n` is the
          size of the domain of input points, and all others are batching
          dimensions. The first component of the return value should be a real
          `Tensor` of matching shape `[...]`. The second component (the gradient)
          should also be of shape `[..., n]` like the input value to the function.
        initial_position: Real `Tensor` of shape `[..., n]`. The starting point, or
          points when using batching dimensions, of the search procedure. At these
          points the function value and the gradient norm should be finite.
        tolerance: Scalar `Tensor` of real dtype. Specifies the gradient tolerance
          for the procedure. If the supremum norm of the gradient vector is below
          this number, the algorithm is stopped.
        x_tolerance: Scalar `Tensor` of real dtype. If the absolute change in the
          position between one iteration and the next is smaller than this number,
          the algorithm is stopped.
        f_relative_tolerance: Scalar `Tensor` of real dtype. If the relative change
          in the objective value between one iteration and the next is smaller than
          this value, the algorithm is stopped.
        max_iterations: Scalar positive int32 `Tensor`. The maximum number of
          iterations.
        parallel_iterations: Positive integer. The number of iterations allowed to
          run in parallel.
        stopping_condition: (Optional) A Python function that takes as input two
          Boolean tensors of shape `[...]`, and returns a Boolean scalar tensor. The
          input tensors are `converged` and `failed`, indicating the current status
          of each respective batch member; the return value states whether the
          algorithm should stop. The default is tfp.optimizer.converged_all which
          only stops when all batch members have either converged or failed. An
          alternative is tfp.optimizer.converged_any which stops as soon as one
          batch member has converged, or when all have failed.
        params: ConjugateGradientParams object with adjustable parameters of the
          algorithm. If not supplied, default parameters will be used.
        name: (Optional) Python str. The name prefixed to the ops created by this
          function. If not supplied, the default name 'minimize' is used.
  
      Returns:
        optimizer_results: A namedtuple containing the following items:
          converged: boolean tensor of shape `[...]` indicating for each batch
            member whether the minimum was found within tolerance.
          failed:  boolean tensor of shape `[...]` indicating for each batch
            member whether a line search step failed to find a suitable step size
            satisfying Wolfe conditions. In the absence of any constraints on the
            number of objective evaluations permitted, this value will
            be the complement of `converged`. However, if there is
            a constraint and the search stopped due to available
            evaluations being exhausted, both `failed` and `converged`
            will be simultaneously False.
          num_objective_evaluations: The total number of objective
            evaluations performed.
          position: A tensor of shape `[..., n]` containing the last argument value
            found during the search from each starting point. If the search
            converged, then this value is the argmin of the objective function.
          objective_value: A tensor of shape `[...]` with the value of the
            objective function at the `position`. If the search converged, then
            this is the (local) minimum of the objective function.
          objective_gradient: A tensor of shape `[..., n]` containing the gradient
            of the objective function at the `position`. If the search converged
            the max-norm of this tensor should be below the tolerance.
  
      """
      with tf.compat.v1.name_scope(
          name, "minimize", [initial_position, tolerance]
      ):
          if params is None:
              params = ConjugateGradientParams()
  
          initial_position = tf.convert_to_tensor(
              value=initial_position, name="initial_position"
          )
          dtype = initial_position.dtype
          tolerance = tf.convert_to_tensor(
              value=tolerance, dtype=dtype, name="grad_tolerance"
          )
          f_relative_tolerance = tf.convert_to_tensor(
              value=f_relative_tolerance,
              dtype=dtype,
              name="f_relative_tolerance",
          )
          x_tolerance = tf.convert_to_tensor(
              value=x_tolerance, dtype=dtype, name="x_tolerance"
          )
          max_iterations = tf.convert_to_tensor(
              value=max_iterations, name="max_iterations"
          )
          stopping_condition = stopping_condition or converged_all
          delta = tf.convert_to_tensor(
              params.sufficient_decrease_param,
              dtype=dtype,
              name="delta",
          )
          sigma = tf.convert_to_tensor(
              params.curvature_param, dtype=dtype, name="sigma"
          )
          eps = tf.convert_to_tensor(
              params.threshold_use_approximate_wolfe_condition,
              dtype=dtype,
              name="sigma",
          )
          eta = tf.convert_to_tensor(
              params.direction_update_param, dtype=dtype, name="eta"
          )
          psi_1 = tf.convert_to_tensor(
              params.initial_guess_small_factor,
              dtype=dtype,
              name="psi_1",
          )
          psi_2 = tf.convert_to_tensor(
              params.initial_guess_step_multiplier,
              dtype=dtype,
              name="psi_2",
          )
  
          f0, df0 = value_and_gradients_function(initial_position)
          converged = _norm(df0) < tolerance
  
          initial_state = _OptimizerState(
              converged=converged,
              failed=tf.zeros_like(converged),  # All false.
              num_iterations=tf.convert_to_tensor(value=0),
              num_objective_evaluations=tf.convert_to_tensor(value=1),
              position=initial_position,
              objective_value=f0,
              objective_gradient=df0,
              direction=-df0,
              prev_step=tf.ones_like(f0),
          )
  
          def _cond(state):
              """Continue if iterations remain and stopping condition is not met."""
              return (
                  state.num_iterations < max_iterations
              ) & tf.logical_not(
                  stopping_condition(state.converged, state.failed)
              )
  
          def _body(state):
              """Main optimization loop."""
              # We use notation of [HZ2006] for brevity.
              x_k = state.position
              d_k = state.direction
              f_k = state.objective_value
              g_k = state.objective_gradient
              a_km1 = state.prev_step  # Means a_{k-1}.
  
              # Define scalar function, which is objective restricted to direction.
              def ls_func(alpha):
                  pt = x_k + tf.expand_dims(alpha, axis=-1) * d_k
                  (
                      objective_value,
                      gradient,
                  ) = value_and_gradients_function(pt)
                  return ValueAndGradient(
                      x=alpha,
                      f=objective_value,
                      df=_dot(gradient, d_k),
                      full_gradient=gradient,
                  )
  
              # Generate initial guess for line search.
              # [HZ2006] suggests to generate first initial guess separately, but
              # [JuliaLineSearches] generates it as if previous step length was 1, and
              # we do the same.
              phi_0 = f_k
              dphi_0 = _dot(g_k, d_k)
              ls_val_0 = ValueAndGradient(
                  x=tf.zeros_like(phi_0),
                  f=phi_0,
                  df=dphi_0,
                  full_gradient=g_k,
              )
              step_guess_result = _init_step(
                  ls_val_0,
                  a_km1,
                  ls_func,
                  psi_1,
                  psi_2,
                  params.quad_step,
              )
              init_step = step_guess_result.step
  
              # Check if initial step size already satisfies Wolfe condition, and in
              # that case don't perform line search.
              c = init_step.x
              phi_lim = phi_0 + eps * tf.abs(phi_0)
              phi_c = init_step.f
              dphi_c = init_step.df
              # Original Wolfe conditions, T1 in [HZ2006].
              suff_decrease_1 = delta * dphi_0 >= tf.math.divide_no_nan(
                  phi_c - phi_0, c
              )
              curvature = dphi_c >= sigma * dphi_0
              wolfe1 = suff_decrease_1 & curvature
              # Approximate Wolfe conditions, T2 in [HZ2006].
              suff_decrease_2 = (2 * delta - 1) * dphi_0 >= dphi_c
              curvature = dphi_c >= sigma * dphi_0
              wolfe2 = suff_decrease_2 & curvature & (phi_c <= phi_lim)
              wolfe = wolfe1 | wolfe2
              skip_line_search = (
                  (step_guess_result.may_terminate & wolfe)
                  | state.failed
                  | state.converged
              )
  
              # Call Hager-Zhang line search (L0-L3 in [HZ2006]).
              # Parameter theta from [HZ2006] is not adjustable, it's always 0.5.
              ls_result = linesearch.hager_zhang(
                  ls_func,
                  value_at_zero=ls_val_0,
                  converged=skip_line_search,
                  initial_step_size=init_step.x,
                  value_at_initial_step=init_step,
                  shrinkage_param=params.shrinkage_param,
                  expansion_param=params.expansion_param,
                  sufficient_decrease_param=delta,
                  curvature_param=sigma,
                  threshold_use_approximate_wolfe_condition=eps,
              )
  
              # Moving to the next point, using step length from line search.
              # If line search was skipped, take step length from initial guess.
              # To save objective evaluation, use objective value and gradient returned
              # by line search or initial guess.
              a_k = tf.compat.v1.where(
                  skip_line_search, init_step.x, ls_result.left.x
              )
              x_kp1 = state.position + tf.expand_dims(a_k, -1) * d_k
              f_kp1 = tf.compat.v1.where(
                  skip_line_search, init_step.f, ls_result.left.f
              )
              g_kp1 = tf.compat.v1.where(
                  skip_line_search,
                  init_step.full_gradient,
                  ls_result.left.full_gradient,
              )
  
              # Evaluate next direction.
              # Use formulas (2.7)-(2.11) from [HZ2013] with P_k=I.
              y_k = g_kp1 - g_k
              d_dot_y = _dot(d_k, y_k)
              b_k = tf.math.divide_no_nan(
                  _dot(y_k, g_kp1)
                  - tf.math.divide_no_nan(
                      _norm_sq(y_k) * _dot(g_kp1, d_k), d_dot_y
                  ),
                  d_dot_y,
              )
              eta_k = tf.math.divide_no_nan(
                  eta * _dot(d_k, g_k), _norm_sq(d_k)
              )
              b_k = tf.maximum(b_k, eta_k)
              d_kp1 = -g_kp1 + tf.expand_dims(b_k, -1) * d_k
  
              # Check convergence criteria.
              grad_converged = _norm_inf(g_kp1) <= tolerance
              x_converged = _norm_inf(x_kp1 - x_k) <= x_tolerance
              f_converged = tf.math.abs(
                  f_kp1 - f_k
              ) <= f_relative_tolerance * tf.math.abs(f_k)
              converged = ls_result.converged & (
                  grad_converged | x_converged | f_converged
              )
              failed = ls_result.failed
              # Construct new state for next iteration.
              new_state = _OptimizerState(
                  converged=converged,
                  failed=failed,
                  num_iterations=state.num_iterations + 1,
                  num_objective_evaluations=state.num_objective_evaluations
                  + step_guess_result.func_evals
                  + ls_result.func_evals,
                  position=tf.compat.v1.where(
                      state.converged, x_k, x_kp1
                  ),
                  objective_value=tf.compat.v1.where(
                      state.converged, f_k, f_kp1
                  ),
                  objective_gradient=tf.compat.v1.where(
                      state.converged, g_k, g_kp1
                  ),
                  direction=d_kp1,
                  prev_step=a_k,
              )
              return (new_state,)
  
          final_state = tf.while_loop(
              _cond,
              _body,
              (initial_state,),
              parallel_iterations=parallel_iterations,
          )[0]
          return OptimizerResult(
              converged=final_state.converged,
              failed=final_state.failed,
              num_iterations=final_state.num_iterations,
              num_objective_evaluations=final_state.num_objective_evaluations,
              position=final_state.position,
              objective_value=final_state.objective_value,
              objective_gradient=final_state.objective_gradient,
          )
  
file_path: tf-quant-finance/tf_quant_finance/math/optimizer/conjugate_gradient.py
imports: import collections
  from typing import Callable, Tuple
  import tensorflow.compat.v2
  from tensorflow_probability.python.optimizer import converged_all
  from tensorflow_probability.python.optimizer import linesearch
  from tf_quant_finance import types
  from tf_quant_finance import utils
label_desc: Use image node_id 1 for calling a global function with example usage: minimize(value_and_gradients_function, initial_position, tolerance, x_tolerance, f_relative_tolerance, max_iterations, parallel_iterations, stopping_condition, params, name) and returns: OptimizerResult, unknown, new_state, ValueAndGradient