Calculating Gradients for a graph containing tf.image.extract_patches, using TF 2.9.0
I think I have sought the answer, a comment on _How to handle non-determinism when training on a GPU?_(https://stackoverflow.com/questions/50744565/how-to-handle-non-determinism-when-training-on-a-gpu),
The problem comes from Tensorflow does not currently guarantee determinism for all of its operations. So, you just need to change to Linux or `set TF_DETERMINISTIC_OPS=0` Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62067">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62067">No</a>

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

v2.9.0-rc2-42-g8a20d54a3c1 2.9.0

### Custom code

Yes

### OS platform and distribution

Windows 10-64bit

### Mobile device

_No response_

### Python version

Python 3.7

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

cuda11.2

### GPU model and memory

_No response_

### Current behavior?

My codes extract patches from a batch of  images as an input and produces a batch of  image patches as an output.

The input (batch_size, rows, cols, 1) is split into patches (patches_num, batch_size, patch_row, patch_col) before being fed to next layer of the network.  This function was called by a custom layer (keras.layers.Layer), and it works well during extracts the patches, but when calculating gradients for a graph with `grads = tape.gradient(loss, model.variables)`, an error came:
```
UnimplementedError: Graph execution error:

Detected at node 'gradients/ExtractImagePatches_1_grad/SparseTensorDenseMatMul/SparseTensorDenseMatMul'

Node: 'gradients/ExtractImagePatches_1_grad/SparseTensorDenseMatMul/SparseTensorDenseMatMul'
2 root error(s) found.
  (0) UNIMPLEMENTED:  A deterministic GPU implementation of SparseTensorDenseMatmulOp is not currently available.
	 [[{{node gradients/ExtractImagePatches_1_grad/SparseTensorDenseMatMul/SparseTensorDenseMatMul}}]]
	 [[gradients/fan_weight_8/StatefulPartitionedCall_grad/PartitionedCall/gradients/ExtractImagePatches_grad/ExtractImagePatches/_63]]
  (1) UNIMPLEMENTED:  A deterministic GPU implementation of SparseTensorDenseMatmulOp is not currently available.
	 [[{{node gradients/ExtractImagePatches_1_grad/SparseTensorDenseMatMul/SparseTensorDenseMatMul}}]]
0 successful operations.
0 derived errors ignored. [Op:__inference___backward_call_20302_28692]
```
I found an issue about tf.extract_image_patches() opend on Feb 10, 2017 (https://github.com/tensorflow/tensorflow/issues/7414)
and added `tf.cast(input_batch, dtype=tf.float32, name="castData") ` in the function, but it wont work.

Here is my code of the function.

```
def spilt_patches(input_batch, 
                   blocks_num_cur, 
                   block_shape_cur):
   '''
    input_batch = np.tile(np.random.randint(0,2,(4,3,3)), (1,2,2))
    blocks_num_cur = 4,
    block_shape_cur = [6,6]
   '''
    shape = input_batch.shape
    input_batch = tf.reshape(input_batch,(shape[0],shape[1], shape[2], 1)) # [batch_size, rows, cols, 1]
    input_batch = tf.cast(input_batch, dtype=tf.float32, name="castData")  
    blocks_cur = tf.image.extract_patches(images = input_batch,
                                       sizes=[1, block_shape_cur[0], block_shape_cur[1], 1],
                                       strides=[1, block_shape_cur[0], block_shape_cur[1], 1],
                                       rates=[1, 1, 1, 1],
                                       padding='VALID') 
    blocks_cur = tf.cast(tf.reshape(blocks_cur,[shape[0], blocks_num_cur, block_shape_cur[0], block_shape_cur[1]]), tf.float32) # [batch_size, patch_num, patch_row, patch_col]
    blocks_cur = tf.transpose(blocks_cur,[1, 0, 2, 3]) 
    
    return blocks_cur
```

### Standalone code to reproduce the issue

```shell
https://gist.github.com/RaymondMarzas/7bec1a8099aded089a29d775caef880e
```


### Relevant log output

```shell
Traceback (most recent call last):

  File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\spyder_kernels\py3compat.py", line 356, in compat_exec
    exec(code, globals, locals)

  File "d:\codes\pycoode\ocnn2nd\fanoutmodel_train_ocnn.py", line 112, in <module>
    grads = tape.gradient(loss, model.variables)

  File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\tensorflow\python\eager\backprop.py", line 1106, in gradient
    unconnected_gradients=unconnected_gradients)

  File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\tensorflow\python\eager\imperative_grad.py", line 73, in imperative_grad
    compat.as_str(unconnected_gradients.value))

  File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\tensorflow\python\eager\function.py", line 1206, in _backward_function_wrapper
    processed_args, remapped_captures)

  File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\tensorflow\python\eager\function.py", line 1861, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))

  File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\tensorflow\python\eager\function.py", line 502, in call
    ctx=ctx)

  File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\tensorflow\python\eager\execute.py", line 55, in quick_execute
    inputs, attrs, num_outputs)

UnimplementedError: Graph execution error:

Detected at node 'gradients/ExtractImagePatches_1_grad/SparseTensorDenseMatMul/SparseTensorDenseMatMul' defined at (most recent call last):
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\runpy.py", line 193, in _run_module_as_main
      "__main__", mod_spec)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\runpy.py", line 85, in _run_code
      exec(code, run_globals)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\spyder_kernels\console\__main__.py", line 24, in <module>
      start.main()
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\spyder_kernels\console\start.py", line 340, in main
      kernel.start()
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\ipykernel\kernelapp.py", line 712, in start
      self.io_loop.start()
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\tornado\platform\asyncio.py", line 215, in start
      self.asyncio_loop.run_forever()
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\asyncio\base_events.py", line 541, in run_forever
      self._run_once()
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\asyncio\base_events.py", line 1786, in _run_once
      handle._run()
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\asyncio\events.py", line 88, in _run
      self._context.run(self._callback, *self._args)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\ipykernel\kernelbase.py", line 510, in dispatch_queue
      await self.process_one()
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\ipykernel\kernelbase.py", line 499, in process_one
      await dispatch(*args)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\ipykernel\kernelbase.py", line 406, in dispatch_shell
      await result
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\ipykernel\kernelbase.py", line 730, in execute_request
      reply_content = await reply_content
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\ipykernel\ipkernel.py", line 387, in do_execute
      cell_id=cell_id,
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\ipykernel\zmqshell.py", line 528, in run_cell
      return super().run_cell(*args, **kwargs)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\IPython\core\interactiveshell.py", line 2975, in run_cell
      raw_cell, store_history, silent, shell_futures, cell_id
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\IPython\core\interactiveshell.py", line 3029, in _run_cell
      return runner(coro)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\IPython\core\async_helpers.py", line 78, in _pseudo_sync_runner
      coro.send(None)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\IPython\core\interactiveshell.py", line 3257, in run_cell_async
      interactivity=interactivity, compiler=compiler, result=result)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\IPython\core\interactiveshell.py", line 3472, in run_ast_nodes
      if (await self.run_code(code, result,  async_=asy)):
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\IPython\core\interactiveshell.py", line 3552, in run_code
      exec(code_obj, self.user_global_ns, self.user_ns)
    File "C:\Users\xxx\AppData\Local\Temp\ipykernel_29532\1338903705.py", line 1, in <module>
      runfile('D:/CODES/pycoode/OCNN2nd/fanoutmodel_train_ocnn.py', wdir='D:/CODES/pycoode/OCNN2nd')
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 526, in runfile
      post_mortem, current_namespace, stack_depth=1)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 613, in _exec_file
      capture_last_expression=False)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 469, in exec_code
      exec_fun(compile(ast_code, filename, 'exec'), ns_globals, ns_locals)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\spyder_kernels\py3compat.py", line 356, in compat_exec
      exec(code, globals, locals)
    File "d:\codes\pycoode\ocnn2nd\fanoutmodel_train_ocnn.py", line 97, in <module>
      trn_pred = model(X_trn) # input[BATCH_SIZE,TARGET_ROWS*TARGET_COLS]], output[BATCH_SIZE,1]
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\keras\engine\training.py", line 490, in __call__
      return super().__call__(*args, **kwargs)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\keras\engine\base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\keras\utils\traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
Node: 'gradients/ExtractImagePatches_1_grad/SparseTensorDenseMatMul/SparseTensorDenseMatMul'
Detected at node 'gradients/ExtractImagePatches_1_grad/SparseTensorDenseMatMul/SparseTensorDenseMatMul' defined at (most recent call last):
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\runpy.py", line 193, in _run_module_as_main
      "__main__", mod_spec)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\runpy.py", line 85, in _run_code
      exec(code, run_globals)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\spyder_kernels\console\__main__.py", line 24, in <module>
      start.main()
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\spyder_kernels\console\start.py", line 340, in main
      kernel.start()
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\ipykernel\kernelapp.py", line 712, in start
      self.io_loop.start()
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\tornado\platform\asyncio.py", line 215, in start
      self.asyncio_loop.run_forever()
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\asyncio\base_events.py", line 541, in run_forever
      self._run_once()
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\asyncio\base_events.py", line 1786, in _run_once
      handle._run()
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\asyncio\events.py", line 88, in _run
      self._context.run(self._callback, *self._args)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\ipykernel\kernelbase.py", line 510, in dispatch_queue
      await self.process_one()
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\ipykernel\kernelbase.py", line 499, in process_one
      await dispatch(*args)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\ipykernel\kernelbase.py", line 406, in dispatch_shell
      await result
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\ipykernel\kernelbase.py", line 730, in execute_request
      reply_content = await reply_content
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\ipykernel\ipkernel.py", line 387, in do_execute
      cell_id=cell_id,
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\ipykernel\zmqshell.py", line 528, in run_cell
      return super().run_cell(*args, **kwargs)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\IPython\core\interactiveshell.py", line 2975, in run_cell
      raw_cell, store_history, silent, shell_futures, cell_id
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\IPython\core\interactiveshell.py", line 3029, in _run_cell
      return runner(coro)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\IPython\core\async_helpers.py", line 78, in _pseudo_sync_runner
      coro.send(None)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\IPython\core\interactiveshell.py", line 3257, in run_cell_async
      interactivity=interactivity, compiler=compiler, result=result)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\IPython\core\interactiveshell.py", line 3472, in run_ast_nodes
      if (await self.run_code(code, result,  async_=asy)):
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\IPython\core\interactiveshell.py", line 3552, in run_code
      exec(code_obj, self.user_global_ns, self.user_ns)
    File "C:\Users\xxx\AppData\Local\Temp\ipykernel_29532\1338903705.py", line 1, in <module>
      runfile('D:/CODES/pycoode/OCNN2nd/fanoutmodel_train_ocnn.py', wdir='D:/CODES/pycoode/OCNN2nd')
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 526, in runfile
      post_mortem, current_namespace, stack_depth=1)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 613, in _exec_file
      capture_last_expression=False)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 469, in exec_code
      exec_fun(compile(ast_code, filename, 'exec'), ns_globals, ns_locals)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\spyder_kernels\py3compat.py", line 356, in compat_exec
      exec(code, globals, locals)
    File "d:\codes\pycoode\ocnn2nd\fanoutmodel_train_ocnn.py", line 97, in <module>
      trn_pred = model(X_trn) # input[BATCH_SIZE,TARGET_ROWS*TARGET_COLS]], output[BATCH_SIZE,1]
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\keras\engine\training.py", line 490, in __call__
      return super().__call__(*args, **kwargs)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\keras\engine\base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "C:\ProgramData\Anaconda3\envs\TF2.9.0\lib\site-packages\keras\utils\traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
Node: 'gradients/ExtractImagePatches_1_grad/SparseTensorDenseMatMul/SparseTensorDenseMatMul'
2 root error(s) found.
  (0) UNIMPLEMENTED:  A deterministic GPU implementation of SparseTensorDenseMatmulOp is not currently available.
	 [[{{node gradients/ExtractImagePatches_1_grad/SparseTensorDenseMatMul/SparseTensorDenseMatMul}}]]
	 [[gradients/fan_weight_8/StatefulPartitionedCall_grad/PartitionedCall/gradients/ExtractImagePatches_grad/ExtractImagePatches/_63]]
  (1) UNIMPLEMENTED:  A deterministic GPU implementation of SparseTensorDenseMatmulOp is not currently available.
	 [[{{node gradients/ExtractImagePatches_1_grad/SparseTensorDenseMatMul/SparseTensorDenseMatMul}}]]
0 successful operations.
0 derived errors ignored. [Op:__inference___backward_call_20302_28692]
```
