Support for int32 variables
Hi @ngc92, I was able to replicate the issue in Colab using TF v2.11 and tf-nightly(). Please find the gists [here(2.11)](https://colab.sandbox.google.com/gist/synandi/63fb7378e6e2adb8a8e9d8d076693248/untitled305.ipynb)  and [here(nightly)](https://colab.sandbox.google.com/gist/synandi/3cfee1ebf013bca0ec9da3c327a7c1ea/59803_nightly.ipynb).
@tilakrayal,  Could you please look into this issue? 
Thank you! There is active work on basic support for int32 as a numeric type on GPU. (This will be opt-in to avoid breaking existing software that relies on reserving int32 for shape computations.) While this is a prerequisite for int32 variables, it is unknown if there are additional specific issues related to the implementation of Variables might also have to be addressed.

> If that is still a concern, maybe automatically placing any int32 Tensors with e.g. less than 1KiB size on the CPU, and larger ones on the GPU, might be the way to go?

Thank you for the suggestion. Having a way for shape computations to remain on host/cpu is critical for managing data transfer to/from GPU.  The current plan is to represent the difference between "int32 as shape" and "int32 as a numeric type" in the internals of TensorFlow used for placement decisions by using different "full type information", `TFT_SHAPE_TENSOR` vs. `TFT_TENSOR` (refer to  [full_type.proto](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/framework/full_type.proto)). Full type information has previously  been used to fix some placement inconsistencies with ragged tensors. The API for users is not yet decided. So while using a threshold based size of int32 tensors could be considered, my guess is that the final design for the part exposed to the user won't include a threshold.

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Feature Request

### Have you reproduced the bug with TF nightly?

No

### Source

binary

### Tensorflow Version

2.11

### Custom Code

Yes

### OS Platform and Distribution

_No response_

### Mobile device

_No response_

### Python version

_No response_

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current Behaviour?
Tensorflow currently does not support `int32` variables being placed on the GPU.
I've found an old [stackoverflow](https://stackoverflow.com/questions/37439299/no-gpu-kernel-for-an-int32-variable-op) post that argued this was because typically, `int32`  Tensors often represent sizes of shapes, which would be needed in host memory. 

If that is still a concern, maybe automatically placing any int32 Tensors with e.g. less than 1KiB size on the CPU, and larger ones on the GPU, might be the way to go?

For large integer tensors (e.g. indices for sparse operations), the current situation requires a trade-off between two undesireable situations:
* Either use `int64`, which wastes both GPU memory, and also memory bandwidth during kernel calls
* Or use `int32`, but then each step has to invoke a host-to-device data transfer.

Declaring the Variable as `float32` and bitcasting to `int32` before each use also does not allow to work around this issue, as the `bitcast` op also lacks a GPU kernel, and thus requires two memory transfers.

This request is somewhat related to #51728, as supporting int32 as index type for `SparseTensor` would run into the aforementioned data transfer problem if the index Tensor was backed by a Variable. 
Even now, [`SparseTensorDenseMatMul`](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/kernels/sparse_tensor_dense_matmul_op_gpu.cu.cc#L151) provides a kernel for `int32`, but again it cannot be used efficiently with indices supplies by a Variable.


### Standalone code to reproduce the issue

```shell
import tensorflow as tf
tf.config.set_soft_device_placement(False)

with tf.device("gpu:0"):
    v = tf.Variable(initial_value=tf.zeros((50, 50), dtype=tf.int32))
```


### Relevant log output

```shell
tensorflow.python.framework.errors_impl.InvalidArgumentError: Could not satisfy device specification '/job:localhost/replica:0/task:0/device:GPU:0'. enable_soft_placement=0. Supported device types [CPU].
```
</details>