code: def test_call_with_retries_fails_after_backoff_on_nonpermanent_error(
      self,
  ):
      attempt_times = []
      fake_time = test_util.FakeTime()
  
      def handler(_, context):
          attempt_times.append(fake_time.time())
          context.abort(grpc.StatusCode.UNAVAILABLE, "foo")
  
      server = TestGrpcServer(handler)
      with server.run() as client:
          with self.assertRaises(grpc.RpcError) as raised:
              grpc_util.call_with_retries(
                  client.TestRpc, make_request(42), fake_time
              )
      self.assertEqual(
          grpc.StatusCode.UNAVAILABLE, raised.exception.code()
      )
      self.assertEqual("foo", raised.exception.details())
      self.assertLen(attempt_times, 5)
      self.assertBetween(attempt_times[1] - attempt_times[0], 2, 4)
      self.assertBetween(attempt_times[2] - attempt_times[1], 4, 8)
      self.assertBetween(attempt_times[3] - attempt_times[2], 8, 16)
      self.assertBetween(attempt_times[4] - attempt_times[3], 16, 32)
  
imports: import contextlib
  import hashlib
  import threading
  import time
  from concurrent import futures
  import grpc
  from tensorboard.util import grpc_util
  from tensorboard.util import grpc_util_test_pb2
  from tensorboard.util import grpc_util_test_pb2_grpc
  from tensorboard.util import test_util
  from tensorboard import test
  from tensorboard import version
label_desc: Use image node_id 3 for calling the CallWithRetriesTest obj's underlying member method code with example usage: obj.test_call_with_retries_fails_after_backoff_on_nonpermanent_error() without return types