Bus error when importing keras_nlp before tensorflow
Hi @yaddayaddayaddayadda ,

I am able to import `keras_nlp` first followed by `tensorflow`.

Input file 62443.py:

```
import keras_nlp
import tensorflow as tf

print('keras_nlp version:',keras_nlp.__version__)
print('TF version:', tf.__version__)
```


Output:
```
(tf-metal) suryanarayanay-macbookpro:Downloads suryanarayanay$ python 62443.py 
Using TensorFlow backend
keras_nlp version: 0.6.3
TF version: 2.15.0
```

 Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62443">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62443">No</a>

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

tensorflow-macos 2.14

### Custom code

Yes

### OS platform and distribution

Mac OS Ventura 13.5.2

### Mobile device

_No response_

### Python version

3.9.10

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

Importing keras_nlp (version 0.6.3) before importing Tensorflw in a python package results in bus error on mac M1. 
I.e, this gives an error:
import keras_nlp
import tensorflow
While this doesn't:
import tensorflow
import keras_nlp



### Standalone code to reproduce the issue

```shell
import keras_nlp
import tensorflow
```


### Relevant log output

_No response_