using tf.linalg.det in custom loss function yields matrix inversion error
'''      import tensorflow as tf
         from keras.losses import Loss
          class DeterminantZeroLoss(Loss):
          def __init__(self, name="determinant_zero_loss"):
          super().__init__(name=name)

    def call(self, y_true, y_pred):
        try:
            # Calculate the determinant of the predicted matrix
            det = tf.linalg.det(y_pred)
        except tf.errors.InvalidArgumentError as e:
            # Handle the case where the matrix inversion fails
            # You can print the error message or take any other appropriate action
            print("Matrix inversion failed:", e)
            # Return a large loss value to indicate the failure
            return 1e6

        # Penalize when the determinant is close to zero
        loss = tf.abs(det)

        return loss

''' 
hope it helps  > except tf.errors.InvalidArgumentError as e:

Sadly that didn't catch the error, but I found out the error seems to only occur if I use
`kernel_initializer=tf.keras.initializers.Zeros()`
in one of the layers as well as the determinant=0 loss.

PS:

>         return 1e6
>     # Penalize when the determinant is close to zero
>     loss = tf.abs(det)

You seem to have missunderstood that, when the matrix is not invertible the determinant should be zero, therefore the loss should be low, if the determinant is close to zero that is good and not penalized ;) Hi @Sur3 ,

Could you able to submit a reproducible code snippet to debug this? Thanks! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. Sorry I didn't find time to produce a full code example yet, maybe next week.
### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

binary

### TensorFlow version

2.16.1

### Custom code

No

### OS platform and distribution

_No response_

### Mobile device

_No response_

### Python version

3.11

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

Calculating the determinant should not require a matrix inversion..
The code worked fine in tensorflow 2.15.0 but tensorflow 2.16.1 yields a matrix inversion error when the determinant is included in the loss function:

```
Epoch 1/2000
2024-03-21 22:17:44.390622: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Input is not invertible.
	 [[{{function_node __inference_one_step_on_data_1325}}{{node gradient_tape/compile_loss/Loss_PH_Ne_id_BSE_det0/MatrixInverse}}]]
Traceback (most recent call last):
  File "./tf_train_mat_trans_pop_m_HnP.py", line 1427, in <module>
    history = model.fit(x_train, y_train, epochs=epochs, batch_size=32, verbose=1, validation_data=(x_test,y_test), callbacks=callback_list)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".local/lib/python3.11/site-packages/keras/src/utils/traceback_utils.py", line 122, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File ".local/lib/python3.11/site-packages/tensorflow/python/eager/execute.py", line 53, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tensorflow.python.framework.errors_impl.InvalidArgumentError: Graph execution error:

Detected at node gradient_tape/compile_loss/Loss_PH_Ne_id_BSE_det0/MatrixInverse defined at (most recent call last):
  File "./tf_train_mat_trans_pop_m_HnP.py", line 1427, in <module>

  File ".local/lib/python3.11/site-packages/keras/src/utils/traceback_utils.py", line 117, in error_handler

  File ".local/lib/python3.11/site-packages/keras/src/backend/tensorflow/trainer.py", line 325, in fit

  File ".local/lib/python3.11/site-packages/keras/src/backend/tensorflow/trainer.py", line 118, in one_step_on_iterator

  File ".local/lib/python3.11/site-packages/keras/src/backend/tensorflow/trainer.py", line 106, in one_step_on_data

  File ".local/lib/python3.11/site-packages/keras/src/backend/tensorflow/trainer.py", line 70, in train_step

Input is not invertible.
	 [[{{node gradient_tape/compile_loss/Loss_PH_Ne_id_BSE_det0/MatrixInverse}}]] [Op:__inference_one_step_on_iterator_1360]

```

### Standalone code to reproduce the issue

```shell
class DeterminantZeroLoss(Loss):
    def __init__(self, name="determinant_zero_loss"):
        super().__init__(name=name)

    def call(self, y_true, y_pred):
        # Calculate the determinant of the predicted matrix
        det = tf.linalg.det(y_pred)

        # Penalize when the determinant is close to zero
        loss = tf.abs(det)

        return loss
```

together with

`kernel_initializer=tf.keras.initializers.Zeros()`

in a Dense layer triggers the bug.


### Relevant log output

_No response_