[xla] Extend collective-permute decomposer to also make decision for
[xla] Extend collective-permute decomposer to also make decision for
Send-Recv pipeling and record the decision with frontend attributes.

We first use a simple heuristics to decide on the decomposition of which
CollectivePermute operations will be pipelined. We will only pipeline
CollectivePermute that sends loop input data, and pick the first
pipelineable CollectivePermute for pipelining. Then, if there is another
pipelineable CollectivePermute that forms a cycle with the to-be-pipelined
CollectivePermute, we will pipeline both CollectivePermute. Otherwise, we will
only pipeline one CollectivePermute.

Then, when we decompose CollectivePermute operations, we add a frontend
attribute to the Send/Recv operation to represent the pipelining decision.

Add tests.

FUTURE_COPYBARA_INTEGRATE_REVIEW=https://github.com/tensorflow/tensorflow/pull/62782 from DanielYang59:warn-shuffle 134fc245a3f6c9362b04ba8dff3b84e703401549
