TensorFlow crashes with a segfault
Hi, @albertz 

Apologize for the delay and I was able to reproduce the issue, I was also getting the same error which you've mentioned above and as per [official documentation](https://www.tensorflow.org/install/pip#macos), There is currently no official GPU support for MacOS. It's better to post this issue [here](https://developer.apple.com/forums/tags/tensorflow-metal) for fast resolution. Thank you!

```
(tf-macos-2.11) gaikwadrahul-macbookpro:returnn gaikwadrahul$ python3 tests/test_TFUtil.py test_get_variable_grad_from_update_ops
Creating converter from 7 to 5
Creating converter from 5 to 7
Creating converter from 7 to 5
Creating converter from 5 to 7
installLibSigSegfault exception: libSegFault not found
tests/test_TFUtil.py:12: DeprecationWarning: Importing from numpy.testing.utils is deprecated since 1.15.0, import from numpy.testing instead.
  from numpy.testing.utils import assert_almost_equal, assert_allclose
TF version: 2.11.0
Metal device set to: Apple M1 Pro

systemMemory: 16.00 GB
maxCacheSize: 5.33 GB

2023-02-25 19:29:40.487656: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:306] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2023-02-25 19:29:40.487681: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:272] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
Executing: test_get_variable_grad_from_update_ops
Optimizer: <tensorflow.python.training.adam.AdamOptimizer object at 0x14e550670>
update ops: [<tf.Operation 'test_get_variable_grad_from_update_ops/Adam/update_test_get_variable_grad_from_update_ops/var/ResourceApplyAdam' type=ResourceApplyAdam>]
update op keys: ['_has_manual_control_dependencies', 'use_locking', 'T', '_class', 'use_nesterov']
update op inputs by name: ['var', 'm', 'v', 'beta1_power', 'beta2_power', 'lr', 'beta1', 'beta2', 'epsilon', 'grad']
2023-02-25 19:29:40.513885: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:357] MLIR V1 optimization pass is not enabled
2023-02-25 19:29:40.517933: W tensorflow/c/c_api.cc:291] Operation '{name:'test_get_variable_grad_from_update_ops/test_get_variable_grad_from_update_ops/var/Adam_1/Assign' id:60 op device:{requested: '', assigned: ''} def:{{{node test_get_variable_grad_from_update_ops/test_get_variable_grad_from_update_ops/var/Adam_1/Assign}} = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](test_get_variable_grad_from_update_ops/test_get_variable_grad_from_update_ops/var/Adam_1, test_get_variable_grad_from_update_ops/test_get_variable_grad_from_update_ops/var/Adam_1/Initializer/zeros)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
grad: Tensor("test_get_variable_grad_from_update_ops/gradients/test_get_variable_grad_from_update_ops/sub_grad/tuple/control_dependency:0", shape=(), dtype=float32)
Optimizer: <tensorflow.python.training.gradient_descent.GradientDescentOptimizer object at 0x14e5506a0>
update ops: [<tf.Operation 'test_get_variable_grad_from_update_ops/GradientDescent/update_test_get_variable_grad_from_update_ops/var/ResourceApplyGradientDescent' type=ResourceApplyGradientDescent>]
update op keys: ['_has_manual_control_dependencies', 'use_locking', 'T', '_class']
update op inputs by name: ['var', 'alpha', 'delta']
2023-02-25 19:29:40.602980: W tensorflow/c/c_api.cc:291] Operation '{name:'test_get_variable_grad_from_update_ops/test_get_variable_grad_from_update_ops/var/Adam_1/Assign' id:60 op device:{requested: '', assigned: ''} def:{{{node test_get_variable_grad_from_update_ops/test_get_variable_grad_from_update_ops/var/Adam_1/Assign}} = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](test_get_variable_grad_from_update_ops/test_get_variable_grad_from_update_ops/var/Adam_1, test_get_variable_grad_from_update_ops/test_get_variable_grad_from_update_ops/var/Adam_1/Initializer/zeros)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
grad: Tensor("test_get_variable_grad_from_update_ops/gradients_1/test_get_variable_grad_from_update_ops/sub_grad/tuple/control_dependency:0", shape=(), dtype=float32)
Optimizer: <tensorflow.python.training.momentum.MomentumOptimizer object at 0x14e550730>
update ops: [<tf.Operation 'test_get_variable_grad_from_update_ops/Momentum/update_test_get_variable_grad_from_update_ops/var/ResourceApplyMomentum' type=ResourceApplyMomentum>]
update op keys: ['_has_manual_control_dependencies', 'use_locking', 'T', '_class', 'use_nesterov']
update op inputs by name: ['var', 'accum', 'lr', 'grad', 'momentum']
2023-02-25 19:29:40.635616: W tensorflow/c/c_api.cc:291] Operation '{name:'test_get_variable_grad_from_update_ops/test_get_variable_grad_from_update_ops/var/Momentum/Assign' id:138 op device:{requested: '', assigned: ''} def:{{{node test_get_variable_grad_from_update_ops/test_get_variable_grad_from_update_ops/var/Momentum/Assign}} = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](test_get_variable_grad_from_update_ops/test_get_variable_grad_from_update_ops/var/Momentum, test_get_variable_grad_from_update_ops/test_get_variable_grad_from_update_ops/var/Momentum/Initializer/zeros)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
grad: Tensor("test_get_variable_grad_from_update_ops/gradients_2/test_get_variable_grad_from_update_ops/sub_grad/tuple/control_dependency:0", shape=(), dtype=float32)
Fatal Python error: Segmentation fault

Thread 0x00000001e7798100 (most recent call first):
  File "/Users/gaikwadrahul/miniconda/envs/tf-macos-2.11/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1454 in _call_tf_sessionrun
  File "/Users/gaikwadrahul/miniconda/envs/tf-macos-2.11/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1361 in _run_fn
  File "/Users/gaikwadrahul/miniconda/envs/tf-macos-2.11/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1378 in _do_call
  File "/Users/gaikwadrahul/miniconda/envs/tf-macos-2.11/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1371 in _do_run
  File "/Users/gaikwadrahul/miniconda/envs/tf-macos-2.11/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1191 in _run
  File "/Users/gaikwadrahul/miniconda/envs/tf-macos-2.11/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 968 in run
  File "tests/test_TFUtil.py", line 3529 in test_get_variable_grad_from_update_ops
  File "tests/test_TFUtil.py", line 4559 in <module>
Segmentation fault: 11
(tf-macos-2.11) gaikwadrahul-macbookpro:returnn gaikwadrahul$ 
``` Ok, I posted it here: https://developer.apple.com/forums/thread/725592 Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59780">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59780">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

No

### Source

source

### Tensorflow Version

tensorflow-macos 2.11.0

### Custom Code

Yes

### OS Platform and Distribution

MacOS 12, tensorflow-macos and tensorflow-metal

### Mobile device

_No response_

### Python version

3.9

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

_No response_

</details>

### GPU model and memory

Apple M1

### Current Behaviour?

It crashes on Apple M1 hardware. It should not crash. It does not crash on other hardware.

I think the code should work. It does work fine on other hardware. But even if there is sth wrong with the code, it still should not crash, but throw some exception instead.


### Standalone code to reproduce the issue

On Apple M1 hardware:

* Checkout https://github.com/rwth-i6/returnn. (Maybe commit 3a67da87c2fd8783c5c2469d72cf1319b5b45837 to be sure.)
* Run: `python3 tests/test_TFUtil.py test_get_variable_grad_from_update_ops`

The relevant code:
* https://github.com/rwth-i6/returnn/blob/3a67da87c2fd8783c5c2469d72cf1319b5b45837/tests/test_TFUtil.py#L3507
* https://github.com/rwth-i6/returnn/blob/3a67da87c2fd8783c5c2469d72cf1319b5b45837/returnn/tf/util/basic.py#L6649

Specifically:

```python
def test_get_variable_grad_from_update_ops():
    with tf_compat.v1.variable_scope("test_get_variable_grad_from_update_ops"):
        var = tf_compat.v1.get_variable("var", (), initializer=tf.zeros_initializer())
        loss = (var - 1.0) ** 2
        for opt in [
            tf_compat.v1.train.AdamOptimizer(),
            tf_compat.v1.train.GradientDescentOptimizer(learning_rate=1.0),
            tf_compat.v1.train.MomentumOptimizer(learning_rate=0.1, momentum=0.9),
            tf_compat.v1.train.RMSPropOptimizer(learning_rate=0.1),
        ]:
            print("Optimizer:", opt)
            minimize_op = opt.minimize(loss=loss, var_list=[var])
            assert isinstance(minimize_op, tf.Operation)
            update_ops = get_var_update_ops(var, fetches=minimize_op)
            print("update ops:", update_ops)
            print("update op keys:", get_op_attrib_keys(update_ops[0]))
            print("update op inputs by name:", get_op_input_names(update_ops[0]))
            session.run(var.initializer)  # reset
            session.run(tf_compat.v1.global_variables_initializer())  # from Adam or so
            assert_equal(session.run(var), 0.0)
            grad = get_variable_grad_from_update_ops(var, update_ops)
            print("grad:", grad)
            _, grad_np = session.run([minimize_op, grad])
            assert_equal(grad_np, -2.0)
```

But there are a few helpers `get_variable_grad_from_update_ops`, `get_var_update_ops` etc.


### Relevant log output

```
Executing: test_get_variable_grad_from_update_ops
Optimizer: <tensorflow.python.training.adam.AdamOptimizer object at 0x16bbe3670>
update ops: [<tf.Operation 'test_get_variable_grad_from_update_ops/Adam/update_test_get_variable_grad_from_update_ops/var/ResourceApplyAdam' type=ResourceApplyAdam>]
update op keys: ['use_nesterov', '_has_manual_control_dependencies', 'use_locking', '_class', 'T']
update op inputs by name: ['var', 'm', 'v', 'beta1_power', 'beta2_power', 'lr', 'beta1', 'beta2', 'epsilon', 'grad']
2023-02-23 10:01:13.694539: W tensorflow/c/c_api.cc:291] Operation '{name:'test_get_variable_grad_from_update_ops/test_get_variable_grad_from_update_ops/var/Adam_1/Assign' id:60 op device:{requested: '', assigned: ''} def:{{{node test_get_variable_grad_from_update_ops/test_get_variable_grad_from_update_ops/var/Adam_1/Assign}} = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](test_get_variable_grad_from_update_ops/test_get_variable_grad_from_update_ops/var/Adam_1, test_get_variable_grad_from_update_ops/test_get_variable_grad_from_update_ops/var/Adam_1/Initializer/zeros)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
grad: Tensor("test_get_variable_grad_from_update_ops/gradients/test_get_variable_grad_from_update_ops/sub_grad/tuple/control_dependency:0", shape=(), dtype=float32)
Optimizer: <tensorflow.python.training.gradient_descent.GradientDescentOptimizer object at 0x16bbe36d0>
update ops: [<tf.Operation 'test_get_variable_grad_from_update_ops/GradientDescent/update_test_get_variable_grad_from_update_ops/var/ResourceApplyGradientDescent' type=ResourceApplyGradientDescent>]
update op keys: ['use_locking', '_class', '_has_manual_control_dependencies', 'T']
update op inputs by name: ['var', 'alpha', 'delta']
2023-02-23 10:01:13.768957: W tensorflow/c/c_api.cc:291] Operation '{name:'test_get_variable_grad_from_update_ops/test_get_variable_grad_from_update_ops/var/Adam_1/Assign' id:60 op device:{requested: '', assigned: ''} def:{{{node test_get_variable_grad_from_update_ops/test_get_variable_grad_from_update_ops/var/Adam_1/Assign}} = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](test_get_variable_grad_from_update_ops/test_get_variable_grad_from_update_ops/var/Adam_1, test_get_variable_grad_from_update_ops/test_get_variable_grad_from_update_ops/var/Adam_1/Initializer/zeros)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
grad: Tensor("test_get_variable_grad_from_update_ops/gradients_1/test_get_variable_grad_from_update_ops/sub_grad/tuple/control_dependency:0", shape=(), dtype=float32)
Optimizer: <tensorflow.python.training.momentum.MomentumOptimizer object at 0x16bbe35b0>
update ops: [<tf.Operation 'test_get_variable_grad_from_update_ops/Momentum/update_test_get_variable_grad_from_update_ops/var/ResourceApplyMomentum' type=ResourceApplyMomentum>]
update op keys: ['use_locking', 'T', '_has_manual_control_dependencies', '_class', 'use_nesterov']
update op inputs by name: ['var', 'accum', 'lr', 'grad', 'momentum']
2023-02-23 10:01:13.805807: W tensorflow/c/c_api.cc:291] Operation '{name:'test_get_variable_grad_from_update_ops/test_get_variable_grad_from_update_ops/var/Momentum/Assign' id:138 op device:{requested: '', assigned: ''} def:{{{node test_get_variable_grad_from_update_ops/test_get_variable_grad_from_update_ops/var/Momentum/Assign}} = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](test_get_variable_grad_from_update_ops/test_get_variable_grad_from_update_ops/var/Momentum, test_get_variable_grad_from_update_ops/test_get_variable_grad_from_update_ops/var/Momentum/Initializer/zeros)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
grad: Tensor("test_get_variable_grad_from_update_ops/gradients_2/test_get_variable_grad_from_update_ops/sub_grad/tuple/control_dependency:0", shape=(), dtype=float32)
Fatal Python error: Segmentation fault

Thread 0x0000000103500580 (most recent call first):
  File "/Users/az/.local/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1454 in _call_tf_sessionrun
  File "/Users/az/.local/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1361 in _run_fn
  File "/Users/az/.local/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1378 in _do_call
  File "/Users/az/.local/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1371 in _do_run
  File "/Users/az/.local/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1191 in _run
  File "/Users/az/.local/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 968 in run
  File "/Users/az/Programmierung/crnn/tests/test_TFUtil.py", line 3529 in test_get_variable_grad_from_update_ops
  File "/Users/az/Programmierung/crnn/tests/test_TFUtil.py", line 4559 in <module>
fish: Job 1, 'python3 tests/test_TFUtil.py te…' terminated by signal SIGSEGV (Address boundary error)
```

Stack trace in LLDB in the crashing thread:
```
* thread #28, queue = 'metal gpu stream', stop reason = EXC_BAD_ACCESS (code=1, address=0xbeaddc3f8010)
  * frame #0: 0x00000001836ea5a0 libobjc.A.dylib`objc_msgSend + 32
    frame #1: 0x000000018df96d38 MPSNDArray`___lldb_unnamed_symbol1550 + 2292
    frame #2: 0x000000018df98bbc MPSNDArray`___lldb_unnamed_symbol1567 + 300
    frame #3: 0x000000018df991e8 MPSNDArray`___lldb_unnamed_symbol1569 + 176
    frame #4: 0x0000000159a7d2b8 libmetal_plugin.dylib`invocation function for block in double dispatchOneKernel<MPSNDArrayIdentity>(MetalStream*, MPSNDArrayIdentity*, NSArray*, MPSNDArray*, char const*, MPSKernelDAGObject*) + 120
    frame #5: 0x00000001836a01b4 libdispatch.dylib`_dispatch_client_callout + 20
    frame #6: 0x00000001836af414 libdispatch.dylib`_dispatch_lane_barrier_sync_invoke_and_complete + 56
    frame #7: 0x0000000159a7d140 libmetal_plugin.dylib`double dispatchOneKernel<MPSNDArrayIdentity>(MetalStream*, MPSNDArrayIdentity*, NSArray*, MPSNDArray*, char const*, MPSKernelDAGObject*) + 120
    frame #8: 0x0000000159a7fffc libmetal_plugin.dylib`metal_plugin::MPSApplyMomentumOp<float>::Compute(metal_plugin::OpKernelContext*) + 2768
    frame #9: 0x0000000159a7f2fc libmetal_plugin.dylib`void metal_plugin::ComputeOpKernel<metal_plugin::MPSApplyMomentumOp<float> >(void*, TF_OpKernelContext*) + 44
    frame #10: 0x000000014cd00028 libtensorflow_framework.2.dylib`tensorflow::PluggableDevice::Compute(tensorflow::OpKernel*, tensorflow::OpKernelContext*) + 148
    frame #11: 0x000000014cc847f0 libtensorflow_framework.2.dylib`tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::Process(tensorflow::SimplePropagatorState::TaggedNode, long long) + 3764
    frame #12: 0x000000028a47eb6c _pywrap_tensorflow_internal.so`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 1496
    frame #13: 0x000000028a47e468 _pywrap_tensorflow_internal.so`tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 80
    frame #14: 0x000000014cb9e878 libtensorflow_framework.2.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 120
    frame #15: 0x000000018386426c libsystem_pthread.dylib`_pthread_start + 148
```

As you see from the output, the crash happens in the last `session.run([minimize_op, grad])`.
