Fix LSTMBlockCell Check Fail
Forgot to include the first argument for `OP_REQUIRES_OK`. It's added now. I think that should fix the build @mihaimaruseac thanks for the review! I am not sure I understand the difference between what you are asking for and the original behavior where the check fail would trigger a crash. Do you mind expanding on that a little bit? `CHECK` fail results in a crash. `OP_REQUIRES_OK` sets `ctx->Status` to return it to client and finishes execution of current function (it is a macro that ends with `return;`).

Basically the issue is

```cc
void LSTMBlockCellFpropWithCUDA(OpKernelContext* ctx,,...) {
  CHECK(expr);
  ...
}

void LSTMBlockCell::Compute(OpKernelContext* ctx,...) {
  // do some validation
  LSTMBlockCellFpropWithCUDA(ctx, ...);
  // assume validation passed
  RunSomeOtherCode(...);
}
```

Right now, `RunSomeOtherCode` does not run because the `CHECK`-fail terminates the process. What we want is still for `RunSomeOtherCode` to not run! If you just replace it with `OP_REQUIRES`, after expanding the macros the code would be

```cc
void LSTMBlockCellFpropWithCUDA(OpKernelContext* ctx,,...) {
  if(expr) {
     ctx->Status = errors::InvalidArgument(...);
     return; // <----------- this line
  }
  ...
}

void LSTMBlockCell::Compute(OpKernelContext* ctx,...) {
  // do some validation
  LSTMBlockCellFpropWithCUDA(ctx, ...);
  // !!!!! You still need to stop executing the kernel here if ctx->Status is not Ok !!!
  // assume validation passed
  RunSomeOtherCode(...);
}
```

Now how the `return` will just terminate `LSTMBlockCellFpropWithCUDA` but if you don't change the `Compute` call then `RunSomeOtherCode` would run when we know that validation already failed. So this causes more harm than just a process crash Ok, that makes sense. Thanks for your help! I didn't realize how the macro worked. I think this latest commit accomplishes what you're saying, but just let me know if you think anything needs to change Were you able to test your changes and see if the check fail isn't being triggered anymore? 
Please also add unit tests that test your change. 
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/kernel_tests/nn_ops/rnn_cell_test.py or
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/ops/cudnn_rnn_ops_test.cc could be relevant places to add them.  Thanks for the pointers on the unit tests! I'll get those added soon. When I test locally against the error case from the related issue, I get the following error logs, but Python no longer crashes:
``` ** On entry to SGEMM_EX  parameter number 8 had an illegal value
2023-02-10 06:55:35.460448: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at blas_gemm.cc:52 : INTERNAL: cublas error
2023-02-10 06:55:35.460510: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at lstm_ops_gpu.cu.cc:277 : INTERNAL: invalid configuration argument
 ** On entry to SGEMM_EX  parameter number 8 had an illegal value
2023-02-10 06:55:35.465759: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at blas_gemm.cc:52 : INTERNAL: cublas error
2023-02-10 06:55:35.465774: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at lstm_ops_gpu.cu.cc:277 : INTERNAL: invalid configuration argument
 ** On entry to SGEMM_EX  parameter number 8 had an illegal value
2023-02-10 06:55:35.470031: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at blas_gemm.cc:52 : INTERNAL: cublas error
2023-02-10 06:55:35.470046: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at lstm_ops_gpu.cu.cc:277 : INTERNAL: invalid configuration argument
 ** On entry to SGEMM_EX  parameter number 8 had an illegal value
2023-02-10 06:55:35.474474: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at blas_gemm.cc:52 : INTERNAL: cublas error
2023-02-10 06:55:35.474486: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at lstm_ops_gpu.cu.cc:277 : INTERNAL: invalid configuration argument
 ** On entry to SGEMM_EX  parameter number 8 had an illegal value
2023-02-10 06:55:35.478711: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at blas_gemm.cc:52 : INTERNAL: cublas error
2023-02-10 06:55:35.478723: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at lstm_ops_gpu.cu.cc:277 : INTERNAL: invalid configuration argument
 ** On entry to SGEMM_EX  parameter number 8 had an illegal value
2023-02-10 06:55:35.482907: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at blas_gemm.cc:52 : INTERNAL: cublas error
2023-02-10 06:55:35.482920: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at lstm_ops_gpu.cu.cc:277 : INTERNAL: invalid configuration argument
 ** On entry to SGEMM_EX  parameter number 8 had an illegal value
2023-02-10 06:55:35.487107: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at blas_gemm.cc:52 : INTERNAL: cublas error
2023-02-10 06:55:35.487120: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at lstm_ops_gpu.cu.cc:277 : INTERNAL: invalid configuration argument
 ** On entry to SGEMM_EX  parameter number 8 had an illegal value
2023-02-10 06:55:35.492341: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at blas_gemm.cc:52 : INTERNAL: cublas error
2023-02-10 06:55:35.492353: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at lstm_ops_gpu.cu.cc:277 : INTERNAL: invalid configuration argument
 ** On entry to SGEMM_EX  parameter number 8 had an illegal value
2023-02-10 06:55:35.496483: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at blas_gemm.cc:52 : INTERNAL: cublas error
2023-02-10 06:55:35.496495: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at lstm_ops_gpu.cu.cc:277 : INTERNAL: invalid configuration argument
 ** On entry to SGEMM_EX  parameter number 8 had an illegal value
2023-02-10 06:55:35.500759: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at blas_gemm.cc:52 : INTERNAL: cublas error
2023-02-10 06:55:35.500771: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at lstm_ops_gpu.cu.cc:277 : INTERNAL: invalid configuration argument
 ** On entry to SGEMM_EX  parameter number 8 had an illegal value
2023-02-10 06:55:35.504896: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at blas_gemm.cc:52 : INTERNAL: cublas error
2023-02-10 06:55:35.504908: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at lstm_ops_gpu.cu.cc:277 : INTERNAL: invalid configuration argument
 ** On entry to SGEMM_EX  parameter number 8 had an illegal value
2023-02-10 06:55:35.508996: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at blas_gemm.cc:52 : INTERNAL: cublas error
2023-02-10 06:55:35.509008: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at lstm_ops_gpu.cu.cc:277 : INTERNAL: invalid configuration argument
 ** On entry to SGEMM_EX  parameter number 8 had an illegal value
2023-02-10 06:55:35.513247: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at blas_gemm.cc:52 : INTERNAL: cublas error
2023-02-10 06:55:35.513259: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at lstm_ops_gpu.cu.cc:277 : INTERNAL: invalid configuration argument
 ** On entry to SGEMM_EX  parameter number 8 had an illegal value
2023-02-10 06:55:35.517478: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at blas_gemm.cc:52 : INTERNAL: cublas error
2023-02-10 06:55:35.517490: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at lstm_ops_gpu.cu.cc:277 : INTERNAL: invalid configuration argument
 ** On entry to SGEMM_EX  parameter number 8 had an illegal value
2023-02-10 06:55:35.522420: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at blas_gemm.cc:52 : INTERNAL: cublas error
2023-02-10 06:55:35.522433: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at lstm_ops_gpu.cu.cc:277 : INTERNAL: invalid configuration argument
 ** On entry to SGEMM_EX  parameter number 8 had an illegal value
2023-02-10 06:55:35.526810: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at blas_gemm.cc:52 : INTERNAL: cublas error
2023-02-10 06:55:35.526822: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at lstm_ops_gpu.cu.cc:277 : INTERNAL: invalid configuration argument
 ** On entry to SGEMM_EX  parameter number 8 had an illegal value
2023-02-10 06:55:35.530964: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at blas_gemm.cc:52 : INTERNAL: cublas error
2023-02-10 06:55:35.530976: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at lstm_ops_gpu.cu.cc:277 : INTERNAL: invalid configuration argument
 ** On entry to SGEMM_EX  parameter number 8 had an illegal value
2023-02-10 06:55:35.535143: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at blas_gemm.cc:52 : INTERNAL: cublas error
2023-02-10 06:55:35.535155: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at lstm_ops_gpu.cu.cc:277 : INTERNAL: invalid configuration argument
 ** On entry to SGEMM_EX  parameter number 8 had an illegal value
2023-02-10 06:55:35.539309: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at blas_gemm.cc:52 : INTERNAL: cublas error
2023-02-10 06:55:35.539321: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at lstm_ops_gpu.cu.cc:277 : INTERNAL: invalid configuration argument
 ** On entry to SGEMM_EX  parameter number 8 had an illegal value
2023-02-10 06:55:35.543485: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at blas_gemm.cc:52 : INTERNAL: cublas error
2023-02-10 06:55:35.543498: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at lstm_ops_gpu.cu.cc:277 : INTERNAL: invalid configuration argument @swachhandl My ARM CI build is failing for the `testLSTMBlockCellErrorHandling2` unit test that I added to `rnn_cell_test.py`. Do you have any suggestions for things I could try to fix it? Could it be that maybe I'm using the wrong `@test_util` annotation? When I run the unit test locally it passes. > @swachhandl My ARM CI build is failing for the `testLSTMBlockCellErrorHandling2` unit test that I added to `rnn_cell_test.py`. Do you have any suggestions for things I could try to fix it? Could it be that maybe I'm using the wrong `@test_util` annotation? When I run the unit test locally it passes.

It seems your test is failing on ARM/Ubuntu/Mac CPU CI but passes on Ubuntu GPU. Can you check if the CPU version is failing because it's not throwing any error (and hence, failing the assert)? 

If the CPU version isn't throwing _any_ error, I think the ideal solution would be to make the GPU version also return the same result as the CPU counterpart (instead of returning an error). > > @swachhandl My ARM CI build is failing for the `testLSTMBlockCellErrorHandling2` unit test that I added to `rnn_cell_test.py`. Do you have any suggestions for things I could try to fix it? Could it be that maybe I'm using the wrong `@test_util` annotation? When I run the unit test locally it passes.
> 
> It seems your test is failing on ARM/Ubuntu/Mac CPU CI but passes on Ubuntu GPU. Can you check if the CPU version is failing because it's not throwing any error (and hence, failing the assert)?
> 
> If the CPU version isn't throwing _any_ error, I think the ideal solution would be to make the GPU version also return the same result as the CPU counterpart (instead of returning an error).

So this is the CPU variant of the code:
https://github.com/tensorflow/tensorflow/blob/ed3e77f858fdba02e6cc402ddb9a9d722c9d158e/tensorflow/core/kernels/rnn/lstm_ops.cc#L73-L126

And then this is the GPU code:
https://github.com/tensorflow/tensorflow/blob/ed3e77f858fdba02e6cc402ddb9a9d722c9d158e/tensorflow/core/kernels/rnn/lstm_ops_gpu.cu.cc#L98-L194

I guess one or more of the assumptions spelled out in the comment are causing the error. I can try to add some validations and see how that affects things. @swachhandl I confirmed that when run on the CPU, no error is thrown, but I'm having a hard time finding a way to get the GPU code to behave in same way. Would it be valid to mark this test case to run only on GPU? Can you pull master and see if the error for GPU still persists? A recent commit (https://github.com/tensorflow/tensorflow/commit/764eae38919b45556373a320af1079ac54e70521) might have fixed it. Even if it's been fixed, propagating the error/checking the status is still useful. @swachhandl I've been having issues building from source on my machine after merging master into my branch, so I figured I'd just test with the nightly build. The core is no longer being dumped, but an internal error is still being thrown when I run on GPU:
```On entry to SGEMM_EX  parameter number 8 had an illegal value
2023-02-20 02:32:30.937716: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at blas_gemm.cc:52 : INTERNAL: an unsupported value or parameter was passed to the function
```
No error is thrown on CPU, just as before. > @swachhandl I've been having issues building from source on my machine after merging master into my branch, so I figured I'd just test with the nightly build. The core is no longer being dumped, but an internal error is still being thrown when I run on GPU:
> 
> ```
> 2023-02-20 02:32:30.937716: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at blas_gemm.cc:52 : INTERNAL: an unsupported value or parameter was passed to the function
> ```
> 
> No error is thrown on CPU, just as before.

Are you getting the internal error even after your changes related to error propagation? What's the result when you run this on CPU?  > > @swachhandl I've been having issues building from source on my machine after merging master into my branch, so I figured I'd just test with the nightly build. The core is no longer being dumped, but an internal error is still being thrown when I run on GPU:
> > ```
> > 2023-02-20 02:32:30.937716: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at blas_gemm.cc:52 : INTERNAL: an unsupported value or parameter was passed to the function
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > No error is thrown on CPU, just as before.
> 
> Are you getting the internal error even after your changes related to error propagation? What's the result when you run this on CPU?

Ok, I got the build working after merging master into my branch earlier today. With my error propagation code in place it's still behaving the same way. I get the internal error when I run the on the GPU, but no error is produced when I run the same code on the CPU.
` ** On entry to SGEMM_EX  parameter number 8 had an illegal value
2023-03-06 19:51:25.813000: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at blas_gemm.cc:52 : INTERNAL: cublas error` >  I get the internal error when I run the on the GPU, but no error is produced when I run the same code on the CPU.

Did you try investigating where the error is originating from? I'm not sure why it isn't being propagated with your changes in place.   > Did you try investigating where the error is originating from? I'm not sure why it isn't being propagated with your changes in place.

I was on the wrong track comparing those two functions in `lstm_ops.cc` and `lstm_ops_gpu.cu.cc`. The internal error is originating here:
https://github.com/tensorflow/tensorflow/blob/4139a2ada21479f923a850bf2708dfe077a22254/tensorflow/core/kernels/rnn/blas_gemm.cc#L52-L56 @swachhandl I think I'm getting closer. The line of code where the internal error is being thrown is at the end of this function: 
https://github.com/tensorflow/tensorflow/blob/c99a73e9ee06d254ec5cafe4183d98342421827b/tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc#L356-L389
`DoBlasInternalImpl` is being called here with `cublasSgemmEx` acting as the `cublas_func`:
https://github.com/tensorflow/tensorflow/blob/c99a73e9ee06d254ec5cafe4183d98342421827b/tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc#L653-L658
From the error message: `** On entry to SGEMM_EX  parameter number 8 had an illegal value`, it looks like `a.opaque()` is returning an invalid value as a parameter for the function `cublasSgemmEx` It looks like `a.opaque()` is returning a null pointer. I tried adding the following lines of code before the call to `DoBlasInternalImpl`, and my test error was triggered.

```
const void *test = a.opaque();

if (!test) {
    return tsl::errors::Internal("TEST ERROR TRIGGERED");
}
```

@swachhandl do you have any ideas on how we might handle this better? The line in the `lstm_ops_gpu.cu.cc` file where the internal error is getting thrown seems to be this one:
https://github.com/tensorflow/tensorflow/blob/b9a128ea6397f3d49211bff68563846254734d5b/tensorflow/core/kernels/rnn/lstm_ops_gpu.cu.cc#L256-L258
It used to crash a few lines beyond that:
https://github.com/tensorflow/tensorflow/blob/b9a128ea6397f3d49211bff68563846254734d5b/tensorflow/core/kernels/rnn/lstm_ops_gpu.cu.cc#L269-L283
So that might explain why we're no longer seeing the core dump. I wonder if it has to do with this commit where `port::InternalError` was switched to `tsl::errors::Internal`: https://github.com/tensorflow/tensorflow/commit/a801af081952535b541852ad6ad96e0d536f2965#diff-9a98b7aa9b93a7357e8bbcdfc82739ef47c69251c684c6c4e58b3b1ff9cd4ac3 Apologies for the late response. I'm not quite sure that it could be happening due to switching out `port::InternalError` with `tsl::errors::Internal`. Are you able to check why `a.opaque()` is a nullptr? And how does it (or whether it even does) map to the input arguments to the op? If we can, we might be able to put a check in the op.  I haven't looked into this deeply, but `nullptr`s usually come up with 0-sized inputs, and that has often been a cause of these CPU vs GPU different error behaviors, where the CPU returns a (valid) empty result, but the GPU crashes because CUDA doesn't often properly handle empty inputs. > Apologies for the late response. I'm not quite sure that it could be happening due to switching out `port::InternalError` with `tsl::errors::Internal`. Are you able to check why `a.opaque()` is a nullptr? And how does it (or whether it even does) map to the input arguments to the op? If we can, we might be able to put a check in the op.

No problem! I'm pretty confident `a.opaque()` is our problem parameter. It is the 11th parameter for the call to `DoBlasInternalImpl`, which has four defined parameters, and then the rest of the `args` are passed to the `cublas_func`, which has one other parameter... 11 - 4 = 8 - 1

https://github.com/tensorflow/tensorflow/blob/c99a73e9ee06d254ec5cafe4183d98342421827b/tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc#L653-L658

https://github.com/tensorflow/tensorflow/blob/c99a73e9ee06d254ec5cafe4183d98342421827b/tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc#L356-L358

https://github.com/tensorflow/tensorflow/blob/c99a73e9ee06d254ec5cafe4183d98342421827b/tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc#L384

When I add the `CHECK(a.opaque() != nullptr)` the `InternalError` no longer gets thrown, and the behavior matches what I see on CPU. > When I add the `CHECK(a.opaque() != nullptr)` the `InternalError` no longer gets thrown, and the behavior matches what I see on CPU.

How can it match CPU?  I thought the CPU behavior _didn't_ crash. > > When I add the `CHECK(a.opaque() != nullptr)` the `InternalError` no longer gets thrown, and the behavior matches what I see on CPU.
> 
> How can it match CPU? I thought the CPU behavior _didn't_ crash.

Hmm, I'm not sure what happened over the weekend when I tested this, but when I run it now on GPU with the check, it does crash. Must have been an issue with how I was testing. That's my mistake. I'll remove that check.

And that's correct - it does not crash on CPU. It doesn't throw any error at all on CPU. I'm still kind of stumped as to how to get the behaviors to match. > And that's correct - it does not crash on CPU. It doesn't throw any error at all on CPU. I'm still kind of stumped as to how to get the behaviors to match.

What's the output on CPU? > What's the output on CPU?

Here's what I get when I print the result on CPU:

```
LSTMBlockCell(i=<tf.Tensor: shape=(2, 0), dtype=float16, numpy=array([], shape=(2, 0), dtype=float16)>, cs=<tf.Tensor: shape=(2, 0), dtype=float16, numpy=array([], shape=(2, 0), dtype=float16)>, f=<tf.Tensor: shape=(2, 0), dtype=float16, numpy=array([], shape=(2, 0), dtype=float16)>, o=<tf.Tensor: shape=(2, 0), dtype=float16, numpy=array([], shape=(2, 0), dtype=float16)>, ci=<tf.Tensor: shape=(2, 0), dtype=float16, numpy=array([], shape=(2, 0), dtype=float16)>, co=<tf.Tensor: shape=(2, 0), dtype=float16, numpy=array([], shape=(2, 0), dtype=float16)>, h=<tf.Tensor: shape=(2, 0), dtype=float16, numpy=array([], shape=(2, 0), dtype=float16)>)
``` > ```
> LSTMBlockCell(i=<tf.Tensor: shape=(2, 0), dtype=float16, numpy=array([], shape=(2, 0), dtype=float16)>, cs=<tf.Tensor: shape=(2, 0), dtype=float16, numpy=array([], shape=(2, 0), dtype=float16)>, f=<tf.Tensor: shape=(2, 0), dtype=float16, numpy=array([], shape=(2, 0), dtype=float16)>, o=<tf.Tensor: shape=(2, 0), dtype=float16, numpy=array([], shape=(2, 0), dtype=float16)>, ci=<tf.Tensor: shape=(2, 0), dtype=float16, numpy=array([], shape=(2, 0), dtype=float16)>, co=<tf.Tensor: shape=(2, 0), dtype=float16, numpy=array([], shape=(2, 0), dtype=float16)>, h=<tf.Tensor: shape=(2, 0), dtype=float16, numpy=array([], shape=(2, 0), dtype=float16)>)
> ```

As I suspected, most of those are actually _empty_ tensors.  In TF, an empty tensor (e.g. one dimension has size zero) ends up having a `nullptr` for a data pointer.  This is mainly because on most systems, when you try to allocate something of size 0 via `malloc`, you end up getting a `nullptr` back.

The input sizes are likely degenerate (i.e. there's an empty input).  The CPU kernel correctly handles this, returning empty outputs of the correct size.  The GPU kernel does not - it blindly forwards the arguments along until the CUDA GEMM call, which then crashes because one of the inputs (whatever becomes `a`) is a `nullptr`.  To fix this, you need to track down which input leads to the empty tensor, and do special handling _before_ calling into CUDA. ROCm CI build failure appears to be due to the removal of this constructor from `status.h` in commit: https://github.com/tensorflow/tensorflow/commit/667081b1c3d84c8fee4b5965d5e151158c74d4f8#diff-2497c155e2f97993a2cea18969d80f2d414e2f1b346cf006d8a8b655a8316b1d

Looks like other pull request builds are failing with the same errors. Just because I'm new to the project, would anybody mind filling me in on how the import/copybara works? At this point is it pending because the AMD ROCm and the MacOS CPU builds failed? I'm seeing other pull requests for which the import completed even though various builds failed The full process is at

![image](https://user-images.githubusercontent.com/323199/229561784-0a2f5509-b731-493f-ad88-bad487688c8d.png)

Right now, this is pending Gerrit review. CC @gbaned 
Fixes #58270 