Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CNDB-12154: Port latest commits from main to main-5.0 #1467

Open
wants to merge 139 commits into
base: main-5.0
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
139 commits
Select commit Hold shift + click to select a range
88085c7
Include checksums in CorruptBlockException when relevant (#1291)
michaeljmarshall Sep 19, 2024
29d2430
Implement FSError#getMessage to ensure file name is logged (#1292)
michaeljmarshall Sep 20, 2024
35e4038
Use query view's locked indexes for Plan#estimateAnnNodesVisited (#1238)
michaeljmarshall Sep 23, 2024
d09b969
CNDB-10923: Fix expressions accidentally joined by OR instead of AND
michaeljmarshall Sep 20, 2024
8c04f9c
Update src/java/org/apache/cassandra/db/filter/RowFilter.java
pkolaczk Sep 23, 2024
b6f2046
Update src/java/org/apache/cassandra/db/filter/RowFilter.java
pkolaczk Sep 23, 2024
46d4d15
Cleanup unused throws Throwable and other minor warnings
pkolaczk Sep 23, 2024
4eddc05
Remove unused enum variant
pkolaczk Sep 23, 2024
2d296a6
Revert "Use query view's locked indexes for Plan#estimateAnnNodesVisi…
michaeljmarshall Sep 24, 2024
ca61eec
CNDB-10862: Moves the deletion of index components on index drop to a…
pcmanus Sep 25, 2024
4761098
CNDB-10455: Automatically tune compaction for vector tables (#1265)
ethan-brown2022 Sep 25, 2024
a4c5c42
Add ShardManagerReplicaAware class to align UCS shards with replica s…
michaeljmarshall Sep 25, 2024
64a1aa4
CNDB-10988: inspect no space left exception on compaction (#1297)
jasonstack Sep 26, 2024
d4b7a54
CNDB-11048 ChunkCache eviction metrics improvements (#1303)
eolivelli Sep 27, 2024
2586d51
CNDB-10966: Add ability to read the TTLs and write times of the eleme…
adelapena Sep 24, 2024
e45beb7
CNDB-10939: Port CASSANDRA-19461 (index empty values)
adelapena Sep 16, 2024
36f06d1
CNDB-10870: Upgrade JVector dependency to 3.0.1
jkni Sep 30, 2024
b0feab1
CNDB-10945: Change calculation of sstable span for small sstables (#1…
blambov Oct 1, 2024
d2759f9
CNDB-11021: Improve performance of ORDER BY using KD-tree index
pkolaczk Sep 30, 2024
03b1881
CNDB-9850: Row-level trie memtable with DeletionInfo tombstone handling
blambov Jun 19, 2024
f6c8a6a
CNDB-9850 followup: Additional rebase changes to review
djatnieks Dec 2, 2024
86c00ae
CNDB-9850 followup: fix failing tests
djatnieks Dec 3, 2024
c1a3fb6
CNDB-9850 followup: Fix RowIndexTest
djatnieks Dec 4, 2024
7707d09
CNDB-9850 followup: Align ByteComparable.Version values with main branch
djatnieks Dec 3, 2024
7831acb
CNDB-10295 Followup - Do not use lambdas in InMemoryTrie (#1326)
eolivelli Oct 3, 2024
d246168
CNDB-11134: Fix data race in adjustForCrossDatacenterClashes to preve…
michaeljmarshall Oct 4, 2024
0de0951
Fix access to JDK internals for jamm post CNDB-9850
ekaterinadimitrova2 Oct 4, 2024
262b135
CNDB-10990: include archive size when constructing Bucket#constructAg…
jasonstack Oct 5, 2024
6ff57bf
CNDB-11028: Port CASSANDRA-18190, this fixes the tests for now, but m…
ekaterinadimitrova2 Sep 25, 2024
ad42706
CNDB-10783: Reuse CRC when appending index components
pkolaczk Sep 11, 2024
a373d3f
CC5 update IndexFileUtils to consistently use the version based getCh…
djatnieks Dec 3, 2024
47681b3
CNDB-10991: skip aggregate that includes any sstables that are alread…
jasonstack Oct 7, 2024
97b8d7f
CNDB-11090: add CompactionRealm#makeUCSEnvironment API (#1322)
jasonstack Oct 8, 2024
9ef942c
DSP-24330 QueryProcessor evictPrepared deadlock (#1257)
bereng Oct 8, 2024
28ac0b8
CNDB-8187 Add metrics about time spent writing SAI (compaction and fl…
eolivelli Oct 8, 2024
98cf299
CNDB-10886: Bump JVector to 3.0.2 (#1343)
jkni Oct 10, 2024
626e873
CNDB-8187 Move SAI writing metrics to micros and use counters (#1345)
eolivelli Oct 11, 2024
7541f42
CNDB-11283: Unify CQLTester#waitForIndex and SAITester#waitForIndexQu…
adelapena Oct 15, 2024
1899291
Add warning when rewriting TOC that does not exist (#1354)
pcmanus Oct 16, 2024
3569d65
CNDB-11336: Fix tests post JDK-8210522 (rewrite reflection of modifie…
ekaterinadimitrova2 Jun 10, 2024
527ccbd
CNDB-11092: Fix IndexInputLeakDetector (#1310)
adelapena Oct 16, 2024
2a6a311
CNDB-10629: Estimate predicate selectivity using histograms
pkolaczk Aug 27, 2024
9bd6ad1
Revert Version.LATEST to DB (#1364)
pkolaczk Oct 17, 2024
4c1b32d
CNDB-10988: fix CompactionOutOfSpaceTest by wrapping IOException with…
jasonstack Oct 17, 2024
3a15aba
CNDB-11338: Fix ClassCastException from jdk GaloisCounterMode when u…
ekaterinadimitrova2 Oct 18, 2024
5b2fb66
CNDB-9877: Add pull request template (#1368)
jacek-lewandowski Oct 21, 2024
2149b5b
CNDB-11345: mock CFS#makeUCSEnvironment in ControllerTest (#1360)
michaeljmarshall Oct 21, 2024
3eb89ac
CNDB-11388: Set timeout when waiting on futures in ChunkCache.rebuffe…
jkni Oct 22, 2024
6515b3a
CNDB-10716: do not use lambdas as they don't work with Jamm 3 on JDK2…
eolivelli Oct 23, 2024
7345020
CNDB-11398: Fix concurrency issues with new TrieMemtable (#1371)
blambov Oct 23, 2024
7c12281
CNDB-11133: Port CASSANDRA-17224, CASSANDRA-18267, CASSANDRA-17294 an…
ekaterinadimitrova2 Oct 23, 2024
475ba3a
CNDB-11217: Implemenet DelegatingShardManager (#1339)
michaeljmarshall Oct 23, 2024
15c23c9
CNDB-11210: Fix false negatives for non-eq (#1359)
k-rus Oct 24, 2024
d6508c7
CNDB-11438 main: Fix CompactionStrategyStatisticsTest (#1378)
djatnieks Oct 24, 2024
7e54436
CNDB-11210 simplify test of Non-eq on truncated types (#1381)
k-rus Oct 25, 2024
301b89d
CNDB-11219: Make TokenAllocatorBase take a supplier for first token a…
michaeljmarshall Oct 28, 2024
c8ccb4f
CNDB-11492: Bump up Version.LATEST to DC
pkolaczk Oct 28, 2024
9d1b66b
CNDB-1461: add REPLACE reload reason (#975)
jasonstack Oct 29, 2024
1516421
CNDB-11519: Apply pk restriction earlier in hybrid sai execution (#1391)
michaeljmarshall Oct 29, 2024
5b7e78a
CNDB-11498: Make main branch compile on JDK22 (#1385)
ekaterinadimitrova2 Oct 30, 2024
152c9d7
CNDB-11434: Refactor tests to mitigate https://openjdk.org/jeps/416 (…
ekaterinadimitrova2 Oct 30, 2024
221f776
CNDB-11518: Split o.a.c.index.sai.cql.VectorTypeTest to prevent timeo…
adelapena Oct 31, 2024
3041d26
CNDB-11459 and 11413: More logging for SAI vector index events (#1404)
michaeljmarshall Nov 5, 2024
878d032
CNDB-11579: Rewrite MessagePayloadTest to account for JEP 416
ekaterinadimitrova2 Nov 4, 2024
6dc1d0e
CNDB-11579: Use try-with-resources in MessagePayloadTest
ekaterinadimitrova2 Nov 5, 2024
88dafb0
CNDB-11499: Fix incorrect thread names in CompactionControllerTest
blambov Nov 5, 2024
eb85f95
CNDB-11563 move and rename SAI iterators and postings as in ASF (#1400)
k-rus Nov 7, 2024
2638063
CNDB-11441: Port update SyncUtil for JDK11 and JDK17
ekaterinadimitrova2 Jun 28, 2023
3dc2722
CNDB-11663: Handle posting list ordinal assignment race (#1410)
michaeljmarshall Nov 7, 2024
96a3d0b
CNDB-11646: Reclaculate ShardManager#minimumPerPartitionSpan on each …
michaeljmarshall Nov 7, 2024
c64eb5e
Rename few iterators to KeyRangeIterator
k-rus Nov 8, 2024
1b7fea9
CNDB-11630: Minimal port of CASSANDRA-18049 and additional fixes arou…
ekaterinadimitrova2 Nov 12, 2024
47589d2
CNDB-11118 return null serializer if response verb is null (#1328)
jtgrabowski Nov 13, 2024
95361eb
CNDB-11678: Use DenseIntMap for CassandraOnHeapGraph (#1418)
michaeljmarshall Nov 14, 2024
3d18d6c
Allow custom SAI components discovery (#1338)
pcmanus Nov 14, 2024
f4a26bd
CNDB-10731: Add support for index analyzers to RowFilter (#1270)
adelapena Nov 14, 2024
30a1094
CNDB-10085: Add guardrail for the number of column filters per query …
adelapena Nov 19, 2024
e70e5de
CNDB-8491: add existing sstables into SAI even if intiial build is sk…
jasonstack Nov 19, 2024
2a115b6
CNDB-11680: Add source sstable/memtable id to vector traces (#1411)
michaeljmarshall Nov 20, 2024
945002f
HCD-18 add option to ignore DC name validation (#1428)
shunsaker Nov 21, 2024
34c38be
CNDB-11801: custom stage exeuctors
jakubzytka Nov 19, 2024
d240fcf
CNDB-11801: test for the functionality of using a custom stage executor
jakubzytka Nov 19, 2024
fa79f58
CNDB-9046: cross-node messaging metrics fixes
jakubzytka Nov 4, 2024
6c0d818
Fix minor bug in intersection's propagate access (#1430)
k-rus Nov 25, 2024
bc2d573
CNDB-11782: additional messaging metrics
jakubzytka Nov 18, 2024
5e880de
CNDB-11581: Stage metrics improvements
jakubzytka Nov 18, 2024
680024b
CNDB-11714: Optimize SAI ORDER BY row materialization (#1419)
michaeljmarshall Nov 26, 2024
ba82683
CNDB-11832: add LifecycleNewTracker#trackNewWritten when sstable and …
jasonstack Nov 28, 2024
a4932a1
CNDB-10851 Lower paxos max backoff sleep from 100 to 50 ms (#1444)
jtgrabowski Nov 28, 2024
2d1b768
CNDB-11768: DynamicEndpointSnitch tweaks: quantization and quantile i…
jakubzytka Nov 29, 2024
1595cc2
CNDB-11760: Prevent full deserialization in CQL's CONTAINS operator (…
adelapena Nov 29, 2024
97d1301
CNDB-11655: Limit the number of clauses before optimizing the Plan
pkolaczk Nov 15, 2024
77861d4
CNDB-11742: Add a client warning when using an n-gram analyzer withou…
adelapena Dec 2, 2024
83a97ea
CNDB-10732: Reproduction tests and fixes for CNDB-10732 and CNDB-1053…
ekaterinadimitrova2 Dec 2, 2024
5411d56
Add method to check if component diff creates unused components (#1438)
pcmanus Dec 3, 2024
3ccd01d
CNDB-11988: Bump jvector to 3.0.4 (#1450)
michaeljmarshall Dec 4, 2024
90aa046
CNDB-10759 Make native transport stage async in regards to read and w…
jakubzytka Dec 6, 2024
faec14b
CNDB-11950: cache sstable density result to speed up test (#1454)
michaeljmarshall Dec 6, 2024
926fee7
CNDB-11762: Remove StorageAttachedIndexQueryPlan#postProcessor (#1422)
adelapena Dec 9, 2024
9e812ac
CNDB-12051: addtional tracing messages about query execution (#1459)
jakubzytka Dec 10, 2024
5d2df65
CNDB-11495: add REMOTE_STORAGE_HANDLER_FACTORY (#1443)
jasonstack Dec 11, 2024
4418146
CNDB-10718: Make RowFilter non-iterable (#1448)
adelapena Dec 12, 2024
88edb5b
CNDB-11932 plan union of half-ranges for inequality (#1451)
k-rus Dec 12, 2024
4e8143c
SAI: Reduce DirectReader object creation (#1011)
michaeljmarshall Dec 12, 2024
7febcba
STAR-1872: Parallelize UCS compactions per output shard (#1342)
blambov Dec 13, 2024
5a9a04f
CNDB-12130: read source file sequentially when sending file through Z…
jasonstack Dec 16, 2024
bf4ab4c
CNDB-12128 Expose to CNDB some methods around ExpirationTask (#1465)
eolivelli Dec 16, 2024
13f1f59
CNDB-11070: Limited backport of CASSANDRA-19534 (#1460)
jkni Dec 16, 2024
ad7ee6b
CNDB-12120: 3 test failures in MultiNodeBillingTest (#1462)
adelapena Dec 16, 2024
bbc7f67
DSP-24606: Replace nulls with negative defaults in system_schema (#1445)
szymon-miezal Dec 16, 2024
7c0097e
CNDB-11242: Allow to specify a custom cluster version provider (#1458)
jacek-lewandowski Dec 16, 2024
762f657
CNDB-12139 minor refactor row count estimation code (#1463)
k-rus Dec 17, 2024
7fdd10d
CC5 Fix for CNDB-11070 to use 'native_transport_timeout' as 5.0 name …
djatnieks Dec 19, 2024
0272b19
CNDB-12215 Replace [On|Off]HeapTrieMemtableIndexTest classes withTrie…
djatnieks Dec 19, 2024
9a7229d
Add row count test for estimated row counts for SAI plan on single re…
k-rus Dec 19, 2024
9482d5e
CNDB-12222 Fix flaky tests in SSTableCorruptionDetectionTest by addin…
djatnieks Dec 23, 2024
bddd5bb
CNDB-12223 Fix CorruptedSSTablesCompactionTest by modifying SharedCom…
djatnieks Dec 23, 2024
17f983b
CNBD-12239: Replace System.nanoTime with approxTime in SAI where appr…
michaeljmarshall Dec 20, 2024
c31eeea
CNDB-12257: do not call System.getProperty on an hotpath (ChunkCache …
eolivelli Dec 23, 2024
c27bf0a
CNDB-12277: Upgrade to jvector 3.0.6 (#1478)
jbellis Dec 25, 2024
462b7b3
CNDB-12272: Fix SimpleClientPerfTest (#1477)
djatnieks Dec 31, 2024
61b0205
Improve typed reads in RandomAccessReader
blambov Dec 30, 2024
118b35d
CNDB-12297: Fix flaky test MutatorTest.testMutatateAtomically by poll…
jkni Jan 2, 2025
eb9ef2b
CNDB-11437: Disable key cache in LazyBloomFilterTest
jkni Jan 2, 2025
4f494ee
CNDB-12296: Skip metric creation for "fake" index contexts generated …
jkni Jan 2, 2025
cc068ed
CNDB-9850 followup: BtiFormat set byteComparableVersion based on trie…
djatnieks Jan 7, 2025
aba71bb
CNDB-9850 followup: Revert part of CNDB-11010 that added PartitionUpd…
djatnieks Jan 7, 2025
974c4d6
CNDB-9850 followup: Implement PR comments
djatnieks Jan 7, 2025
3b558ba
CNDB-9850 followup: more alignment with main branch
djatnieks Jan 7, 2025
8edc8ed
CNDB-12346: Fix reporting enqueue time for stage tasks (#1490)
jakubzytka Jan 9, 2025
2207746
CNDB-12289: Cache USE_DSE_COMPATIBLE_HISTOGRAM_BOUNDARIES to save cpu…
eolivelli Jan 9, 2025
942cf76
CNDB-8501 Propagate read and write request sensors via native protoco…
aymkhalil Jan 9, 2025
86a11a4
CNDB-12393: Refactor TransportTest to account for JEP-416 (#1497)
ekaterinadimitrova2 Jan 10, 2025
59740b7
CNDB-11508: Add config to enable JDK22 test and run (#1503)
ekaterinadimitrova2 Jan 15, 2025
3563121
CNDB-12423: Fix flaky test failures in LegacySSTableTest by blocking …
jkni Jan 14, 2025
ea4ac1e
moving PQ refine into CompactionGraph (#1471)
jbellis Jan 14, 2025
66faff8
CNDB-12342: Avoid unnecessary looping in QueryView#build (#1491)
pkolaczk Jan 14, 2025
5700159
CNDB-12445: parameterize ByteComparable.Version for clusteringFromByt…
michaeljmarshall Jan 14, 2025
de16993
CNDB-12075: Allow UDFs within GROUP BY clause (#1494)
adelapena Jan 15, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,59 @@

package org.apache.cassandra.db.memtable;

import java.util.concurrent.atomic.AtomicReference;

import org.apache.cassandra.db.commitlog.CommitLogPosition;
import org.apache.cassandra.db.partitions.PartitionUpdate;
import org.apache.cassandra.db.partitions.TriePartitionUpdate;
import org.apache.cassandra.metrics.TableMetrics;
import org.apache.cassandra.schema.TableMetadataRef;

/**
* This class exists solely to avoid initialization of the default memtable class.
* Some tests want to setup table parameters before initializing DatabaseDescriptor -- this allows them to do so.
*/
public class DefaultMemtableFactory
public class DefaultMemtableFactory implements Memtable.Factory
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems I forgot in CNDB-9850 that the OSS branch has a better solution to the problem this is meant to solve: extract the factory outside the memtable class, so that it no longer has a dependency on the memtable class being constructed. On the OSS branch that's the SkipListMemtableFactory, here we can do the same moving TrieMemtable.Factory to top level as TrieMemtableFactory, and then referencing its INSTANCE from the default params no longer involves initializing TrieMemtable.

This avoids having to reproduce every method of Memtable.Factory here which we can easily forget to do.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I'll do that

{
// We can't use TrieMemtable.FACTORY as that requires DatabaseDescriptor to have been initialized.
public static final Memtable.Factory INSTANCE = (commitLogLowerBound, metadataRef, owner) -> new TrieMemtable(commitLogLowerBound, metadataRef, owner, null);
@Override
public Memtable create(AtomicReference<CommitLogPosition> commitLogLowerBound, TableMetadataRef metadaRef, Memtable.Owner owner)
{
return TrieMemtable.FACTORY.create(commitLogLowerBound, metadaRef, owner);
}

@Override
public boolean writesShouldSkipCommitLog()
{
return TrieMemtable.FACTORY.writesShouldSkipCommitLog();
}

@Override
public boolean writesAreDurable()
{
return TrieMemtable.FACTORY.writesAreDurable();
}

@Override
public boolean streamToMemtable()
{
return TrieMemtable.FACTORY.streamToMemtable();
}

@Override
public boolean streamFromMemtable()
{
return TrieMemtable.FACTORY.streamFromMemtable();
}

@Override
public TableMetrics.ReleasableMetric createMemtableMetrics(TableMetadataRef metadataRef)
{
return TrieMemtable.FACTORY.createMemtableMetrics(metadataRef);
}

@Override
public PartitionUpdate.Factory partitionUpdateFactory()
{
return TriePartitionUpdate.FACTORY;
}
}
9 changes: 9 additions & 0 deletions src/java/org/apache/cassandra/db/memtable/TrieMemtable.java
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,8 @@ public class TrieMemtable extends AbstractShardedMemtable
{
private static final Logger logger = LoggerFactory.getLogger(TrieMemtable.class);

public static final Factory FACTORY = new TrieMemtable.Factory(null);

/** Buffer type to use for memtable tries (on- vs off-heap) */
public static final BufferType BUFFER_TYPE = DatabaseDescriptor.getMemtableAllocationType().toBufferType();

Expand Down Expand Up @@ -781,6 +783,13 @@ public UnfilteredRowIterator next()
}
}

public static Factory factory(Map<String, String> optionsCopy)
{
String shardsString = optionsCopy.remove(SHARDS_OPTION);
Integer shardCount = shardsString != null ? Integer.parseInt(shardsString) : null;
return new Factory(shardCount);
}

static class Factory implements Memtable.Factory
{
final Integer shardCount;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -454,7 +454,7 @@ private KeyRangeIterator rangeMatch(Expression expression, AbstractBounds<Partit
// Before version DB, we encoded composite types using a non order-preserving function. In order to
// perform a range query on a map, we use the bounds to get all entries for a given map key and then
// only keep the map entries that satisfy the expression.
byte[] key = ByteSourceInverse.readBytes(entry.getKey().asComparableBytes(ByteComparable.Version.OSS50));
byte[] key = ByteSourceInverse.readBytes(entry.getKey().asComparableBytes(TypeUtil.BYTE_COMPARABLE_VERSION));
if (expression.isSatisfiedBy(ByteBuffer.wrap(key)))
mergingIteratorBuilder.add(entry.getValue());
});
Expand Down