PR #10182: [ROCM] fixing hipblas-lt large startup overhead
PR #10182: [ROCM] fixing hipblas-lt large startup overhead

Imported from GitHub PR https://github.com/openxla/xla/pull/10182


This PR https://github.com/openxla/xla/pull/6595 introduced **matmul_algorithm_cache_** which calls MatmulPlan::GetAlgorithms while holding a mutex. This is not bad on CUDA side but on ROCM, hipblas-lt has a large startup overhead, and when running this code with multiple GPUs, this startup overhead increases linearly with the number of GPUs.
However, **MatmulPlan::GetAlgorithms** can be run in parallel across different StreamExecutorsL anyway library functions are called under [per-instance mutex](https://github.com/ROCm/xla/blob/43c67a4d135b5e1e023c3e67b9062b9aa88d6ab8/xla/stream_executor/cuda/cuda_blas_lt.cc#L244).

I also did the same for **matmul_plans_cache_**, besides I added a multi-GPU matmul test which runs with cublas and cublas-lt libraries.

@xla-rotation: could you please have a look ?

Copybara import of the project:

--
43c67a4d135b5e1e023c3e67b9062b9aa88d6ab8 by Pavel Emeliyanenko <pavel.emeliyanenko@amd.com>:

fixing hipblas-lt large startup overhead

Merging this change closes #10182

FUTURE_COPYBARA_INTEGRATE_REVIEW=https://github.com/openxla/xla/pull/10182 from ROCm:ci_hipblas_lt_startup_overhead 43c67a4d135b5e1e023c3e67b9062b9aa88d6ab8
