Possible stale cache when using RunInTransaction with specific order of memache failure.

[jongillham]

This issue revolves around RunInTransaction which locks all memcache keys that use Put and Delete within the transaction prior to committing to the datastore.

The following scenario would produce stale results when calling Get.

• RunInTransaction locks memcache successfully.
• Memcache fails/locks are evicted.
• Memcache recovers.
• Another client calls Get on the entities that were supposed to be locked and repopulates memcache.
• RunInTransaction commits to the datastore and memcache is left with stale entity values.

I currently don't see any way of completely eliminating this consistency issue. Note that it does not affect Get, Put or Delete when used outside of RunInTransaction.

Note that this issue does not affect consistency guarantees when just using nds.Get* and nds.Put* functions.

[derekperkins]

Do you get an error or other notification that the memcache lock failed?

[jongillham]

There isn't an error. It's just that there is a very tiny possibility that nds.Get will return stale results from its cache after using nds.RunInTransaction if the sequence of events occur exactly as described above.

It's makes the difference of the nds package having complete cache consistency compared to strong cache consistency.

[derekperkins]

If there isn't any way to know whether or not memcache failed in the way you describe, you could force a cache reload at the end of every RunInTransaction request.

[jongillham]

That would help a bit, but if the cache reload failed we would still end up with a stale cache. I am beginning to think the package as it is now, although not perfect, is the best possible situation.

[derekperkins]

I don't have any issues with the possibility of that cache error. It seems very unlikely, and as the datastore is eventually consistent anyways, people should already be ready for the possibility of stale data.

[jongillham]

The Get, Put and Delete datastore calls are strongly consistent. Only queries without an ancestor are eventually consistent. I rely on those guarantees for the type of code I write - which is why it is taking me so long to get this package right. 😅

[derekperkins]

What about adding FlushCache(key) / FlushCacheAll() functions so that you can eliminate keys in the event that you absolutely can't have stale results?

[derekperkins]

What about a GetIgnoreCache?

[jongillham]

With NDS the datastore never has the opportunity to be populated with stale results. It will always be 100% intact. So if you really need to return 100% non stale results you could just use datastore.Get perfectly fine which would be the equivalent of the GetIgnoreCache you suggested.