Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CNDB-12460: Fix PQVector reencoding when refining in CompactionGraph #1506

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

jkni
Copy link

@jkni jkni commented Jan 15, 2025

What is the issue

When refining PQVectors in CompactionGraph, indexing of vectors may fail because vectors in the graph aren't yet encoded, or vectors may incorrectly be encoded as all zeroes.

What does this PR fix and why was it fixed

When re-encoding vectors after refinement, use the previous count of compressed vectors rather than the max ordinal in the graph. Because adding to the graph is asynchronous, there may be encoded vectors that aren't in the graph. The old approach would not re-encode any vectors with ordinals above the max ordinal. Depending on timing, this could cause vectors to fail to index (if they try to compare to an ordinal that isn't in the encoded vectors) or be encoded as all zeroes (due to the code that fills holes in compressed vectors).

Checklist before you submit for review

  • Make sure there is a PR in the CNDB project updating the Converged Cassandra version
  • Use NoSpamLogger for log lines that may appear frequently in the logs
  • Verify test results on Butler
  • Test coverage for new/modified code is > 80%
  • Proper code formatting
  • Proper title for each commit staring with the project-issue number, like CNDB-1234
  • Each commit has a meaningful description
  • Each commit is not very long and contains related changes
  • Renames, moves and reformatting are in distinct commits

…already encoded vectors rather than the max ordinal in the graph, as addition to the graph is asynchronous and some encoded vectors may not yet be indexed.
@jbellis
Copy link

jbellis commented Jan 15, 2025

Restating the problem:

Although compressedVectors.count() == builder.getGraph().getIdUpperBound() when all insertions have completed, these are not equivalent during construction since addGraphNode (which modifies the graph upper bound) is called separately and asynchronously, so refine() needs to use the former instead of the latter.

Fix LGTM.

@cassci-bot
Copy link

❌ Build ds-cassandra-pr-gate/PR-1506 rejected by Butler


1 new test failure(s) in 1 builds
See build details here


Found 1 new test failures

Test Explanation Branch history Upstream history
...ToolEnableDisableBinaryTest.testMaybeChangeDocs regression 🔴 🔵🔵🔵🔵🔵🔵🔵

Found 3 known test failures

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants