Code

Hubspot cell balancer & normalizer#126

Code

Open

szabowexler wants to merge 134 commits intohubspot-2.5HubSpot/hbase:hubspot-2.5from hubspot-cell-balancerHubSpot/hbase:hubspot-cell-balancerCopy head branch name to clipboard

+1,608-85

Commits

Commits on Oct 16, 2024

• HubSpotCellCostFunction

  

  Ray Mattingly

  committed
  9c62045

  Copy full SHA for 9c62045

Commits on Oct 17, 2024

• Adjust to handle little endian cell encoding

  

  Elias Szabo

  committed
  e4f5a14

  Copy full SHA for e4f5a14

• Mark as private

  

  Elias Szabo

  committed
  aad121f

  Copy full SHA for aad121f

• Revert to big endian, simplify heuristics

  

  Elias Szabo

  committed
  9b5002b

  Copy full SHA for 9b5002b

Commits on Oct 29, 2024

• Fix NPE, add logging, run spotless

  

  Ray Mattingly

  committed
  9a954dd

  Copy full SHA for 9a954dd

• Clean up

  

  Elias Szabo

  committed
  6271c26

  Copy full SHA for 6271c26

Commits on Oct 30, 2024

• Add init debug

  

  Elias Szabo

  committed
  995b8cb

  Copy full SHA for 995b8cb

• Clarify expectations via preconditions

  

  Elias Szabo

  committed
  d94d862

  Copy full SHA for d94d862

• Update debug and add guard for non default tables

  

  Elias Szabo

  committed
  8202674

  Copy full SHA for 8202674

• Emit setup at info level to ensure we see it

  

  Elias Szabo

  committed
  f83dc2e

  Copy full SHA for f83dc2e

• Add info state dump on every cost calc call

  

  Elias Szabo

  committed
  8c6c48c

  Copy full SHA for 8c6c48c

Commits on Oct 31, 2024

• Add some debug so we can see why regionlocation would be null

  

  Elias Szabo

  committed
  ab52ea6

  Copy full SHA for ab52ea6

• emit if we disable locationfinder

  

  Elias Szabo

  committed
  0ac73bb

  Copy full SHA for 0ac73bb

• Ensure the region finder is set if the cell cost function exists

  

  Elias Szabo

  committed
  275ba6b

  Copy full SHA for 275ba6b

• Emit the multiplier

  

  Elias Szabo

  committed
  1f58743

  Copy full SHA for 1f58743

• Missed one spot

  

  Elias Szabo

  committed
  57205da

  Copy full SHA for 57205da

• Fix debug

  

  Elias Szabo

  committed
  a9e1547

  Copy full SHA for a9e1547

• skip any that snuck in, emit better logs, and fail more obviously here

  

  Elias Szabo

  committed
  b59c17c

  Copy full SHA for b59c17c

• include count w/o servers

  

  Elias Szabo

  committed
  840496d

  Copy full SHA for 840496d

• Make it legible

  

  Elias Szabo

  committed
  d7081eb

  Copy full SHA for d7081eb

• list details of the unknown region

  

  Elias Szabo

  committed
  856e440

  Copy full SHA for 856e440

• Emit which table

  

  Elias Szabo

  committed
  7df5fc2

  Copy full SHA for 7df5fc2

• Skip if empty region server mapping, assume it's empty for now

  

  Elias Szabo

  committed
  a1d849f

  Copy full SHA for a1d849f

• Emit the cells in the region here

  

  Elias Szabo

  committed
  45c8182

  Copy full SHA for 45c8182

• Tell us about which cells this region holds

  

  Elias Szabo

  committed
  1a67f81

  Copy full SHA for 1a67f81

• Add emission for region size

  

  Elias Szabo

  committed
  5328064

  Copy full SHA for 5328064

• Make clear if we skip any non-empty regions

  

  Elias Szabo

  committed
  4490e1b

  Copy full SHA for 4490e1b

• Include this

  

  Elias Szabo

  committed
  580e31c

  Copy full SHA for 580e31c

• If the first two bytes of start/stop are the same, the region holds exactly one cell

  

  Elias Szabo

  committed
  ba831d9

  Copy full SHA for ba831d9

• Correct how we calculate the cells

  

  Elias Szabo

  committed
  cdd6e77

  Copy full SHA for cdd6e77

Commits on Nov 5, 2024

• add a version identifier here

  

  Elias Szabo

  committed
  426aeca

  Copy full SHA for 426aeca

• This isn't really an edge case so much as the main case

  

  Elias Szabo

  committed
  4a0a7fa

  Copy full SHA for 4a0a7fa

• Update logging & calcs

  

  Elias Szabo

  committed
  867f492

  Copy full SHA for 867f492

• Use shaded version

  

  Elias Szabo

  committed
  6120bc8

  Copy full SHA for 6120bc8

• emit the table name and namespace

  

  Elias Szabo

  committed
  b3a664b

  Copy full SHA for b3a664b

• Switch to multidimensional array to reduce allocations

  

  Elias Szabo

  committed
  a3b9b88

  Copy full SHA for a3b9b88

Commits on Nov 6, 2024

• Optimize the balancer eval function

  

  Elias Szabo

  committed
  45ed606

  Copy full SHA for 45ed606

• Deps

  

  Elias Szabo

  committed
  941d2d2

  Copy full SHA for 941d2d2

• Add custom step generator for stochastic load to prioritize shuffling regions from high cell count servers to low cell count servers

  

  Elias Szabo

  committed
  0123942

  Copy full SHA for 0123942

• Use our own candidate generator for the stochastic balancer

  

  Elias Szabo

  committed
  43fb3da

  Copy full SHA for 43fb3da

• Yep that sure is a 5

  

  Elias Szabo

  committed
  3d45b4d

  Copy full SHA for 3d45b4d

• It has to be the order of the ordinal of course

  

  Elias Szabo

  committed
  cf1064d

  Copy full SHA for cf1064d

Commits on Nov 7, 2024

• Correct reservoir sampling seed, and use boolean[] instead of set

  

  Elias Szabo

  committed
  04fecfe

  Copy full SHA for 04fecfe

• Cost is invoked 2-3 times per use, memoize it

  

  Elias Szabo

  committed
  35adc46

  Copy full SHA for 35adc46

• Filter out non-default regions

  

  Elias Szabo

  committed
  3a744ab

  Copy full SHA for 3a744ab

• Prevent being out of bounds

  

  Elias Szabo

  committed
  fe67705

  Copy full SHA for fe67705

• Correct off-by-1

  

  Elias Szabo

  committed
  1d9b7ef

  Copy full SHA for 1d9b7ef

• Add guards here

  

  Elias Szabo

  committed
  0ac31f5

  Copy full SHA for 0ac31f5

• Include the tables

  

  Elias Szabo

  committed
  06c2ad5

  Copy full SHA for 06c2ad5

• Do not emit null actions

  

  Elias Szabo

  committed
  d25a9ac

  Copy full SHA for d25a9ac

• only use these balancer tools on objects-3

  

  Elias Szabo

  committed
  235fe34

  Copy full SHA for 235fe34

• This is a bug - only add the cost of the function if it's needed

  

  Elias Szabo

  committed
  3517598

  Copy full SHA for 3517598

• Add a lot of trace logging

  

  Elias Szabo

  committed
  c4c6296

  Copy full SHA for c4c6296

• More logging

  

  Elias Szabo

  committed
  1b1bc44

  Copy full SHA for 1b1bc44

• Trace enough to figure out why cell count is 0

  

  Elias Szabo

  committed
  6ac45ba

  Copy full SHA for 6ac45ba

• Fix the subtle array access bug

  

  Elias Szabo

  committed
  583aca9

  Copy full SHA for 583aca9

Commits on Nov 8, 2024

• Undo memoization on cluster state change, and allow to trace teh balancer exploration

  

  Elias Szabo

  committed
  26b4f21

  Copy full SHA for 26b4f21

• Also emit the full cost breakdown per step

  

  Elias Szabo

  committed
  bbed188

  Copy full SHA for bbed188

• Rework the cost function to be the number of cells (over all servers) above the best possible isolation

  

  Elias Szabo

  committed
  dbe3263

  Copy full SHA for dbe3263

• Update debug to focus on which region/cells are getting picked

  

  Elias Szabo

  committed
  41a0b41

  Copy full SHA for 41a0b41

• Tweak down to trace

  

  Elias Szabo

  committed
  1873181

  Copy full SHA for 1873181

• Rework how the cost function calculates and updates cost

  

  Elias Szabo

  committed
  adf28a6

  Copy full SHA for adf28a6

• Fix edge case for short rowkeys

  

  Elias Szabo

  committed
  06e6b83

  Copy full SHA for 06e6b83

• Merge pull request #120 from HubSpot/isolate-generator-cost-mismatch

  Show description for e270d6a
  szabowexler

  authored
  e270d6a

  Copy full SHA for e270d6a

• Add debug and fix the state error here

  

  Elias Szabo

  committed
  4f61691

  Copy full SHA for 4f61691

• No noop

  

  Elias Szabo

  committed
  c6a84ec

  Copy full SHA for c6a84ec

• Print which generator we've selected

  

  Elias Szabo

  committed
  f40e83b

  Copy full SHA for f40e83b

Commits on Nov 18, 2024

• Tweak logs to allow for local run

  

  Elias Szabo

  committed
  7011733

  Copy full SHA for 7011733

• use shaded version

  

  Elias Szabo

  committed
  bd83602

  Copy full SHA for bd83602

• Use gson

  

  Elias Szabo

  committed
  5d69edc

  Copy full SHA for 5d69edc

• Try exposing only specific fields

  

  Elias Szabo

  committed
  d8cef32

  Copy full SHA for d8cef32

• Only emit objects-3, and include the full region info

  

  Elias Szabo

  committed
  e1f1da1

  Copy full SHA for e1f1da1

Commits on Nov 19, 2024

• Mark as exposed

  

  Elias Szabo

  committed
  eb52446

  Copy full SHA for eb52446

• Refine when we print, and what

  

  Elias Szabo

  committed
  eca1dfe

  Copy full SHA for eca1dfe

• Update serde for int2int map so we can run the balancer locally

  

  Elias Szabo

  committed
  d68f91a

  Copy full SHA for d68f91a

Commits on Nov 20, 2024

• Stash partial balancer rework

  

  Elias Szabo

  committed
  aad1e1b

  Copy full SHA for aad1e1b

• Stash2 -- gets to a balance of 1-6 cells/RS

  

  Elias Szabo

  committed
  5a1dc79

  Copy full SHA for 5a1dc79

Commits on Nov 21, 2024

• First cleanup

  

  Elias Szabo

  committed
  9199cd8

  Copy full SHA for 9199cd8

• Disable automatic logging for local runs

  

  Elias Szabo

  committed
  a986195

  Copy full SHA for a986195

• Merge pull request #122 from HubSpot/rework-cell-balancer

  Show description for 0677ff3
  szabowexler

  authored
  0677ff3

  Copy full SHA for 0677ff3

• Fix test

  

  Elias Szabo

  committed
  258caf9

  Copy full SHA for 258caf9

Commits on Nov 26, 2024

• Stash work

  

  Elias Szabo

  committed
  e95ef44

  Copy full SHA for e95ef44

• cost is actually how far we are from having as many cells as possible

  

  Elias Szabo

  committed
  221a8c4

  Copy full SHA for 221a8c4

• add custom normalizer

  

  Hernan Gelaf-Romer

  committed
  20cbb95

  Copy full SHA for 20cbb95

• Revert "add custom normalizer"

  Show description for 9eedc86
  

  Hernan Gelaf-Romer

  committed
  9eedc86

  Copy full SHA for 9eedc86

• add hubspot normalizer

  

  Hernan Gelaf-Romer

  committed
  3526567

  Copy full SHA for 3526567

• Prioritize spreading cells out

  

  Elias Szabo

  committed
  83dad7f

  Copy full SHA for 83dad7f

• Not for inclusion

  

  Elias Szabo

  committed
  378ad33

  Copy full SHA for 378ad33

Commits on Nov 27, 2024

• Merge pull request #124 from HubSpot/cell-spread-out

  Show description for faf41c9
  szabowexler

  authored
  faf41c9

  Copy full SHA for faf41c9

Commits on Dec 2, 2024

• Extract static methods, simplify

  

  Elias Szabo

  committed
  c0895ba

  Copy full SHA for c0895ba

• Clean up the candidate generator

  

  Elias Szabo

  committed
  5ba25ae

  Copy full SHA for 5ba25ae

• Elevate to higher package so normalizer can share common cell ops

  

  Elias Szabo

  committed
  3bd6efb

  Copy full SHA for 3bd6efb

• Clean up + normalize cost

  

  Elias Szabo

  committed
  61b89c5

  Copy full SHA for 61b89c5

• Update the normalizer to avoid merging across cell lines

  

  Elias Szabo

  committed
  66b2adf

  Copy full SHA for 66b2adf

• Mark addition

  

  Elias Szabo

  committed
  ddb3017

  Copy full SHA for ddb3017

• Finish clean up

  

  Elias Szabo

  committed
  bf9fe28

  Copy full SHA for bf9fe28

• Fix import

  

  Elias Szabo

  committed
  6044fc7

  Copy full SHA for 6044fc7

• Print error if cell id is out of bounds

  

  Elias Szabo

  committed
  edc89ff

  Copy full SHA for edc89ff

• Move this up

  

  Elias Szabo

  committed
  6a7511b

  Copy full SHA for 6a7511b

• Improve debug output

  

  Elias Szabo

  committed
  ff1ef1f

  Copy full SHA for ff1ef1f

Commits on Dec 3, 2024

• Cap max cells per RS to be 10% of all cells

  

  Elias Szabo

  committed
  e0695c0

  Copy full SHA for e0695c0

• Do not install unless multiplier is positive

  

  Elias Szabo

  committed
  e6a9d7c

  Copy full SHA for e6a9d7c

Commits on Dec 4, 2024

• Target a specific capped cell count

  

  Elias Szabo

  committed
  7b69247

  Copy full SHA for 7b69247

• Merge pull request #127 from HubSpot/target-specific-cell-count

  Show description for 9a41c05
  szabowexler

  authored
  9a41c05

  Copy full SHA for 9a41c05

• Simplify when we fill underloaded

  

  Elias Szabo

  committed
  6b2df75

  Copy full SHA for 6b2df75

• include target

  

  Elias Szabo

  committed
  4cae482

  Copy full SHA for 4cae482

• Emit which generator

  

  Elias Szabo

  committed
  28b0271

  Copy full SHA for 28b0271

• randomize the under-/overloaded server picked

  

  Elias Szabo

  committed
  f8085d5

  Copy full SHA for f8085d5

• Mark if we keep or reject

  

  Elias Szabo

  committed
  e8361df

  Copy full SHA for e8361df

• Print region counts

  

  Elias Szabo

  committed
  ca45b36

  Copy full SHA for ca45b36

• Prioritize balance by region and THEN evening out cell isolation

  

  Elias Szabo

  committed
  b8ac383

  Copy full SHA for b8ac383

• Merge pull request #128 from HubSpot/add-unbalance-dominating-factor

  Show description for ce825c9
  szabowexler

  authored
  ce825c9

  Copy full SHA for ce825c9

Commits on Dec 5, 2024

• Add guard in case of error computing online cost, and do a deep reset if it goes negative

  

  Elias Szabo

  committed
  5a42bb3

  Copy full SHA for 5a42bb3

• Clean MutableRegionInfo

  

  Elias Szabo

  committed
  b685173

  Copy full SHA for b685173

• Clean up ServerName

  

  Elias Szabo

  committed
  cf40b93

  Copy full SHA for cf40b93

• Clean up TableName

  

  Elias Szabo

  committed
  b7f108a

  Copy full SHA for b7f108a

• Clean up Address

  

  Elias Szabo

  committed
  2a22aea

  Copy full SHA for 2a22aea

• Clean up BalancerClusterState

  

  Elias Szabo

  committed
  d87742d

  Copy full SHA for d87742d

• Clean up RegionLocationFinder

  

  Elias Szabo

  committed
  0f67a71

  Copy full SHA for 0f67a71

• Clean up StochasticLoadBalancer

  

  Elias Szabo

  committed
  b425e9a

  Copy full SHA for b425e9a

• More cleanup StochasticLoadBalancer

  

  Elias Szabo

  committed
  d8e9f9a

  Copy full SHA for d8e9f9a

• Clean up RegionNormalizerFactory

  

  Elias Szabo

  committed
  2337b4c

  Copy full SHA for 2337b4c

• Merge pull request #129 from HubSpot/clean-up-for-merge

  Show description for d5b2b9a
  szabowexler

  authored
  d5b2b9a

  Copy full SHA for d5b2b9a

• clean imports

  

  Elias Szabo

  committed
  6caeb48

  Copy full SHA for 6caeb48

• style

  

  Elias Szabo

  committed
  e865850

  Copy full SHA for e865850

• Style

  

  Elias Szabo

  committed
  f8d48d9

  Copy full SHA for f8d48d9

Commits on Dec 6, 2024

• Small clusters may not have enough regions/cell to support lower isolation

  

  Elias Szabo

  committed
  70bc20f

  Copy full SHA for 70bc20f

• Merge pull request #130 from HubSpot/handle-small-clusters

  Show description for 36d4fd2
  szabowexler

  authored
  36d4fd2

  Copy full SHA for 36d4fd2

• Fix which Ints

  

  Elias Szabo

  committed
  f0cf9ff

  Copy full SHA for f0cf9ff

Commits on Dec 16, 2024

• Emit the cluster state at the end of balance

  

  Elias Szabo

  committed
  71ecc2b

  Copy full SHA for 71ecc2b

• Revert "Remove all the debugging changes, generally make ready for real review & merge"

  szabowexler

  authored
  da0834e

  Copy full SHA for da0834e

• Merge pull request #133 from HubSpot/revert-129-clean-up-for-merge

  Show description for 59b41c2
  szabowexler

  authored
  59b41c2

  Copy full SHA for 59b41c2

Commits on Dec 18, 2024

• Measure this distance by region count from balanced

  

  Elias Szabo

  committed
  25e0cf9

  Copy full SHA for 25e0cf9

Commits on Jan 7, 2025

• Set target to 20% of cells

  

  Elias Szabo

  committed
  c2fde1d

  Copy full SHA for c2fde1d