Data loss during cluster fail over.

[sgurudarshan]

What version of redis operator are you using?

redis-operator version: ghcr.io/ot-container-kit/redis-operator/redis-operator:v0.18.1

Does this issue reproduce with the latest release?
yes

What operating system and processor architecture are you using (kubectl version)?

kubectl version

Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:46:05Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4

What did you do?

Created sentinel and replication cluster with below configuration.
redis-replication clusterSize = 2
redis-sentinel clusterSize = 3

Node1 contains redis-replication-0, sentinel-0 and sentinel-2 instances.
Node2 contains redis-replication-1 and sentinel-1 instances

Disabled scheduling on Node1. Manually deleted master instance (redis-replication-0). Sentinel elected new master (redis-replication-1) and able to write data.
Enabled scheduling on Node1, when old master instance(redis-replication-0) comes up, data written into master (redis-replication-1) is not present.
Basically data written to master (redis-replication-1) is replaced with old master(redis-replication-0) data.

What did you expect to see?

Data should persist in newly elected master.

What did you see instead?

Data loss in newly elected master.