[NextPluggableDevice] Disable alignment check for AsyncValueTensor
@jyingl3  can you have a look? thanks Thanks Zhoulong! Do you think we should make kTag = 0x1ULL declared in tensor.h, and let async_value_tensor.h use it? then it will make sure they will check the same value. But I am also fine with keeping this change as-is if the suggestion does not seem appropriate for tensor.h.

@chuanhaozhuge @qqfish FYI. Hi Zhoulong, I'm wondering if you could provide some concrete motivation on disabling the alignment check? We didn't run into any trouble in our internal usage so just wondering. Thanks! > Thanks Zhoulong! Do you think we should make kTag = 0x1ULL declared in tensor.h, and let async_value_tensor.h use it? then it will make sure they will check the same value. But I am also fine with keeping this change as-is if the suggestion does not seem appropriate for tensor.h.
> 
> @chuanhaozhuge @qqfish FYI.

@jyingl3 If the async_value_tensor.h use this kTag, this kTag needs to be defined as a global variable in tensor,h, it may have issue if .h defines a global variable,  all the .cc file which includes the tensor.h will have a copy definition of this kTag, that may cause some duplicate definition if it has the same definition, such as in [xla_tensor.cc](https://github.com/tensorflow/tensorflow/blob/ded1c57d60255621a4f5c885002fb709ce5bb470/tensorflow/compiler/jit/xla_tensor.cc#L110), it also defined this kTag > Hi Zhoulong, I'm wondering if you could provide some concrete motivation on disabling the alignment check? We didn't run into any trouble in our internal usage so just wondering. Thanks!

@chuanhaozhuge Some custom kernels(such as `TensorList` related kernels) doesn't provide C API , for these kernels, instead of implementing these kernels in plugin, proper defines a common kernels for plugin and use device's memcpy/memset/memzero to implemement these kernels. For these kernels, we need to use tensor.flat<>() to access the content of tensor(for example, [SetZero](https://github.com/tensorflow/tensorflow/blob/22ea643435838346eeee6b9dfdcca91a59f3553a/tensorflow/core/kernels/list_kernels.h#L82) in list_kernel.h), that will trigger the alignment check and thus failed on AsyncValueTensor
![image](https://user-images.githubusercontent.com/6346853/236967843-a986426d-5cc2-4aa0-8a61-fc9d5618f49b.png)
 @jzhoulon Thanks for explanation Zhoulong. If I understand right, the use case is to pass AsyncValueTensor to a regular TF Kernel (like this [example](https://github.com/tensorflow/tensorflow/blob/bebc84ec0035e86bc47309b7499ceed5e427064c/tensorflow/core/kernels/matmul_op_impl.h#L386)). I'm wondering do you plan to update the kernel further so that the tensor can be interpreted as AsyncValueTensor? If so maybe [FromTensor](https://github.com/tensorflow/tensorflow/blob/a4b2d805546a9e4ba4c05c48884f419b85f1cc45/tensorflow/core/tfrt/common/async_value_tensor.h#L36) will be good for the use case? > @jzhoulon Thanks for explanation Zhoulong. If I understand right, the use case is to pass AsyncValueTensor to a regular TF Kernel (like this [example](https://github.com/tensorflow/tensorflow/blob/bebc84ec0035e86bc47309b7499ceed5e427064c/tensorflow/core/kernels/matmul_op_impl.h#L386)). I'm wondering do you plan to update the kernel further so that the tensor can be interpreted as AsyncValueTensor? If so maybe [FromTensor](https://github.com/tensorflow/tensorflow/blob/a4b2d805546a9e4ba4c05c48884f419b85f1cc45/tensorflow/core/tfrt/common/async_value_tensor.h#L36) will be good for the use case?

@chuanhaozhuge Thanks for the suggestion, I think it is a good solution for those regular kernels using `FromTensor` to retrive the AsyncValueTensor. I will think how to refactor the code, however, in current stage, I think the change may be big, beacuse lots of code is like:  convert tensor to Eigen Tensor through flat, shaped  in common path first, then dispatch to backend implmentation, such as cpu, gpu, pluggable device. modify the common path code is unclean and since PluggableDevice device type can only get through runtime, so I can't write another kernel with template for it. 

for example,the following code is in common path.
`
        inputs_flat.emplace_back(new typename TTypes<T, 2>::ConstMatrix(
           const_cast<const Tensor&>(zeros).shaped<T, 2>(
                {1, zeros.NumElements()})));
`
 ` auto output_flat = output->shaped<T, 2>({1, output->NumElements()});` 



https://github.com/tensorflow/tensorflow/blob/3ac5290c3c5bab0fca6bb769840102ae5952f735/tensorflow/core/kernels/list_kernels.h#L248
```
        }
        inputs_flat.emplace_back(new typename TTypes<T, 2>::ConstMatrix(
           const_cast<const Tensor&>(zeros).shaped<T, 2>(
                {1, zeros.NumElements()})));
      }
    }
  auto output_flat = output->shaped<T, 2>({1, output->NumElements()});

#if GOOGLE_CUDA || TENSORFLOW_USE_ROCM
    if (std::is_same<Device, Eigen::GpuDevice>::value) {
      ConcatGPU<T>(c, inputs_flat, output, &output_flat);
      return;
    }
#endif  // GOOGLE_CUDA || TENSORFLOW_USE_ROCM
    if (IsPluggableDevice(c)) {
      ConcatPluggableDevice<T>(c, inputs_flat, &output_flat);
    } else {
      ConcatCPU<T>(c->device(), inputs_flat, &output_flat);
    }
  }
```

  Thanks for the explanation Zhoulong. I do still think that code like `auto ptr = se::DeviceMemoryBase(tensor.flat<T>().data(), tensor.TotalBytes());` won't work even with alignment check disabled (perhaps will crash?) on AsyncValueTensor (since the [void* data_](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/framework/tensor.h#L105) in TensorBuffer is AVT). I don't have intention to block this PR though, just in case you have further development to resolve this issue. > 
yes, StreamExecutor based API won't work in here for NextPluggableDevice. As we discussed before, we may have PjRt C API or NextPluggableDevice C API which provides memcpy/memzero/memset functionality. pjrt device context's CopyInSameDevice also need an C API provides memcpy . And as the above code showed, to keep kernel struct, there are some common path which convert tensor to eigen tensor through flat<>, so I think it is better to solve this issue first. thanks. @jyingl3 @chuanhaozhuge 
 Let's take a step back - If the method IsAlign is not meaningful when AsyncValueTensor (NextPluggableDevice) is used, can we disable it? 

How to handle StreamExecutor based API is not finalized yet. Will the decision of how to handle StreamExecutor based API affect whether to disable IsAlign? > Let's take a step back - If the method IsAlign is not meaningful when AsyncValueTensor (NextPluggableDevice) is used, can we disable it?
> 
> How to handle StreamExecutor based API is not finalized yet. Will the decision of how to handle StreamExecutor based API affect whether to disable IsAlign?

@jyingl3 @chuanhaozhuge   for the second question, I think the StreamExecutor based API handle(backend specific) will not affect those common path(convert tensor to Eigen Tensor through flat), so I think it is better to disable IsAlign, or do you have another design choice? thanks

```
  auto output_flat = output->shaped<T, 2>({1, output->NumElements()});

#if GOOGLE_CUDA || TENSORFLOW_USE_ROCM
    if (std::is_same<Device, Eigen::GpuDevice>::value) {
      ConcatGPU<T>(c, inputs_flat, output, &output_flat);
      return;
    }
#endif  // GOOGLE_CUDA || TENSORFLOW_USE_ROCM
    if (IsPluggableDevice(c)) {
      ConcatPluggableDevice<T>(c, inputs_flat, &output_flat);
    } else {
      ConcatCPU<T>(c->device(), inputs_flat, &output_flat);
    }
  }
``` Discussed with @chuanhaozhuge. Let's proceed with this change to disable IsAlign. Thanks Zhoulong!
This PR is to disable alignment check for AsyncValueTensor, the reason why I didn't use ```AsyncValueTensor* AsyncValueTensor::FromOpaquePointer() ```to check whether the pointer is AsyncValueTensor is    async_value_tensor.h include tensor.h, I can;t include the decleartion(async_value_tensor.h) in tensor.h , and async_value_tensor and tensor.h(core/framework) has cycle dependency, so I didn't find a good way to handle this, and just hardcode to check the whether the last bit is 1 or not to indicate whether it is an AsyncValueTensor.

