Adaptive Planning in FTE

[gaurav8297]

Adaptive planning is part of FTE, wherein the engine can modify the plan at runtime based on exchange-level statistics. For example, reordering of join or mitigation of skewness. It will significantly impact cost and performance if the plan chosen by the static optimiser isn’t the best due to the underestimation or lack of statistics.

Here are some initial roadmap items:

• Adaptive planning framework in FTE #20276, [Extracted Commits] Adaptive planning FTE #20513
• Implement runtime adaptive partitioning for FTE #18349 [Moved to Adaptive planning framework as part of above pr)
• [FTE] Adaptive reorder partitioned join #23046
• Adaptive broadcast to partitioned join in FTE #23179
• Improve AdaptivePlanner to handle speculative execution #23180
• Add metrics to track how many times AdaptivePlanner rules are fired #23181
• Improve AdaptiveBroadcastToPartitionedJoin by reusing existing exchanges #23372

Future todos:

• Handle skewness in join or aggregation.
• Estimate cardinality using probabilistic data structures. This can be applied in various areas, such as join optimization
  and eliminating redundant partial aggregation etc.