Fix allreduce bug

[Binyang2014]

Fix allreduce correctness issue

[chhwang]

Why couldn't we see correctness failure earlier from mscclpp-test?

[saeedmaleki]

I also don't understand why it wasn't an error before?

[Binyang2014]

Previously, we did not encounter this error because each rank initialized its elements with the rank number. For instance, all elements for rank 3 were initialized to 3. In the all-reduce algorithm, we divide the tensor into 8 shards. The expected behavior is to place shard 1 on GPU 1, shard 2 on GPU 2, and so on. However, the current implementation places shard 0 on both GPU 1 and GPU 2. As all elements within each shard are identical, this results in consistent output, masking the error. To detect such issues in the future, we need a more robust data initialization approach, such as using the formula: (rank_number + 1) * shard_number.

[chhwang]

Passed integration tests: https://github.com/microsoft/mscclpp/actions/runs/6562648457