Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Arby gets stuck unable to replace order #121

Open
kilrau opened this issue Nov 6, 2020 · 11 comments
Open

Arby gets stuck unable to replace order #121

kilrau opened this issue Nov 6, 2020 · 11 comments
Labels
bug Something isn't working P2 mid priority

Comments

@kilrau
Copy link
Contributor

kilrau commented Nov 6, 2020

Below output of my environment which is configured to issue ETH/BTC orders. As one can see, sell order is missing even though all balances and limits are fine. Arby logs show that it failed to replace an order because ALREADY_EXISTS. Restart of arby did not fix this.

simnet > listorders

Trading pair: BTC/USDT
┌────────────────────────────────────────────────┬────────────────────────────────────────────────┐
│ Buy                                            │ Sell                                           │
├───────────────┬─────────────┬──────────────────┼───────────────┬─────────────┬──────────────────┤
│ Quantity      │ Price       │ Alias            │ Quantity      │ Price       │ Alias            │
├───────────────┼─────────────┼──────────────────┼───────────────┼─────────────┼──────────────────┤
│ 0.01878902    │ 15350.3868  │ NestGrid         │ 0.29509678    │ 15935.4498  │ NestGrid         │
└───────────────┴─────────────┴──────────────────┴───────────────┴─────────────┴──────────────────┘

Trading pair: ETH/BTC
┌────────────────────────────────────────────────┬────────────────────────────────────────────────┐
│ Buy                                            │ Sell                                           │
├───────────────┬─────────────┬──────────────────┼───────────────┬─────────────┬──────────────────┤
│ Quantity      │ Price       │ Alias            │ Quantity      │ Price       │ Alias            │
├───────────────┼─────────────┼──────────────────┼───────────────┼─────────────┼──────────────────┤
│ 14.1075       │ 0.0279006   │ SheriffSolution  │               │             │                  │
└───────────────┴─────────────┴──────────────────┴───────────────┴─────────────┴──────────────────┘

Trading pair: LTC/BTC
┌────────────────────────────────────────────────┬────────────────────────────────────────────────┐
│ Buy                                            │ Sell                                           │
├───────────────┬─────────────┬──────────────────┼───────────────┬─────────────┬──────────────────┤
│ Quantity      │ Price       │ Alias            │ Quantity      │ Price       │ Alias            │
├───────────────┼─────────────┼──────────────────┼───────────────┼─────────────┼──────────────────┤
│ 10            │ 0.004       │ OzoneYellow      │ 9.4           │ 0.0042      │ OzoneYellow      │
└───────────────┴─────────────┴──────────────────┴───────────────┴─────────────┴──────────────────┘

Trading pair: USDT/DAI
┌────────────────────────────────────────────────┬────────────────────────────────────────────────┐
│ Buy                                            │ Sell                                           │
├───────────────┬─────────────┬──────────────────┼───────────────┬─────────────┬──────────────────┤
│ Quantity      │ Price       │ Alias            │ Quantity      │ Price       │ Alias            │
└───────────────┴─────────────┴──────────────────┴───────────────┴─────────────┴──────────────────┘
simnet > getbalance

Balance:
┌──────────┬───────────────┬───────────────────────────────┬────────────────────────────┐
│ Currency │ Total Balance │ Wallet Balance (Not Tradable) │ Channel Balance (Tradable) │
├──────────┼───────────────┼───────────────────────────────┼────────────────────────────┤
│ BTC      │ 1000.72965053 │ 997.9116993                   │ 2.81795123                 │
├──────────┼───────────────┼───────────────────────────────┼────────────────────────────┤
│ DAI      │ 4000          │ 2500                          │ 1500                       │
├──────────┼───────────────┼───────────────────────────────┼────────────────────────────┤
│ ETH      │ 38.55040222   │ 29.99901497                   │ 8.55138725                 │
├──────────┼───────────────┼───────────────────────────────┼────────────────────────────┤
│ LTC      │ 1013.88977649 │ 1004.9997805                  │ 8.88999599                 │
├──────────┼───────────────┼───────────────────────────────┼────────────────────────────┤
│ USDT     │ 2000          │ 1000                          │ 1000                       │
└──────────┴───────────────┴───────────────────────────────┴────────────────────────────┘
simnet > tradinglimits

Trading Limits:
┌──────────┬────────────┬────────────┬───────────────────┬──────────────────┐
│ Currency │ Max Buy    │ Max Sell   │ Reserved Outbound │ Reserved Inbound │
├──────────┼────────────┼────────────┼───────────────────┼──────────────────┤
│ BTC      │ 2.13195826 │ 2.37434352 │ 0.39360771        │ 0                │
├──────────┼────────────┼────────────┼───────────────────┼──────────────────┤
│ DAI      │ 525        │ 1500       │ 0                 │ 0                │
├──────────┼────────────┼────────────┼───────────────────┼──────────────────┤
│ ETH      │ 5.505375   │ 8.55138725 │ 0                 │ 14.1075          │
├──────────┼────────────┼────────────┼───────────────────┼──────────────────┤
│ LTC      │ 1.00964201 │ 8.78999599 │ 0                 │ 0                │
├──────────┼────────────┼────────────┼───────────────────┼──────────────────┤
│ USDT     │ 525        │ 1000       │ 0                 │ 0                │
└──────────┴────────────┴────────────┴───────────────────┴──────────────────┘
simnet > logs arby
2020-11-06 11:04:53.5353 [OpenDEX] trace: Creating ETH/BTC sell order with id arby-ETH/BTC-sell-order, quantity 8.46587337 and price 0.0290292
2020-11-06 11:04:53.5353 [OpenDEX] warn: 6 ALREADY_EXISTS: order with local id arby-ETH/BTC-sell-order already exists. Retrying in 5000ms.
2020-11-06 11:04:55.5555 [Centralized] trace: heartbeat from ETHBTC socket
2020-11-06 11:04:58.5858 [Centralized] info: Base asset balance 1000 and quote asset balance 1000
2020-11-06 11:04:58.5858 [OpenDEX] trace: Base asset balance 8.55138725 (maxinbound: 19.612875, maxoutbound: 8.55138725) and quote asset balance 2.81795123 (maxinbound: 2.13195826, maxoutbound: 2.76795123).
@kilrau kilrau assigned ghost Nov 6, 2020
@kilrau
Copy link
Contributor Author

kilrau commented Nov 9, 2020 •
edited
Loading

I have arby in this state on both of my simnet environments now, restart doesn't help. xud restart probably fixes this, but I don't want to do that now in order to debug this.

@kilrau
Copy link
Contributor Author

kilrau commented Nov 9, 2020

Confirmed: xud restart fixes this

@kilrau
Copy link
Contributor Author

kilrau commented Nov 10, 2020

Move this issue to the xud repo? I still have on environment up where this happens. cc @sangaman

@kilrau kilrau transferred this issue from ExchangeUnion/market-maker-bot Nov 12, 2020
@kilrau
Copy link
Contributor Author

kilrau commented Nov 12, 2020

Moved this over to xud since it very much looks like an xud issue.

@kilrau kilrau assigned sangaman and unassigned ghost Nov 20, 2020
@kilrau
Copy link
Contributor Author

kilrau commented Nov 20, 2020 •
edited
Loading

After speaking to @raladev , we think this is what happens and it's actually an issue on arby side @erkarl (hence moving issue back to arby repo):

  • ALREADY_EXISTS is arby trying to placeorder for an order that is already in the order book
  • we assume this can occur if arby somehow fails to cleanly remove all orders when e.g. restarting as part of the scheduled restarts
  • when coming up again it tries to place its first order but fails because the old order from pre-restart is still there

Handling this should be easy: since arby orders have rather unique id arby should switch to replace or remove+place if an order with the same id already exists on xud

@kilrau kilrau assigned ghost and unassigned sangaman Nov 20, 2020
@kilrau kilrau transferred this issue from ExchangeUnion/xud Nov 20, 2020
@kilrau kilrau added bug Something isn't working P1 top priority labels Nov 20, 2020
@ghost
Copy link

ghost commented Nov 20, 2020

After speaking to @raladev , we think this is what happens and it's actually an issue on arby side @erkarl (hence moving issue back to arby repo):

* `ALREADY_EXISTS` is arby trying to `placeorder` for an order that is already in the order book

* we assume this can occur if arby somehow fails to cleanly remove all orders when e.g. restarting as part of the scheduled restarts

* when coming up again it tries to place its first order but fails because the old order from pre-restart is still there

Handling this should be easy: since arby orders have rather unique id arby should simply replace or remove/place orders if an order with the same id already exists on xud

The current logic tries to replaceorder with the unique buy/sell ID. If replacing fails with NOT_FOUND error from xud then it will attempt to placeorder instead.

@kilrau
Copy link
Contributor Author

kilrau commented Nov 20, 2020

When issuing the first order arby uses placeorder though, correct? That's when it happens.

@kilrau
Copy link
Contributor Author

kilrau commented Dec 1, 2020

Above @erkarl

@kilrau kilrau added P2 mid priority and removed P1 top priority labels Dec 1, 2020
@kilrau
Copy link
Contributor Author

kilrau commented Dec 1, 2020

Decreasing prio though since this is an edge case.

@kilrau
Copy link
Contributor Author

kilrau commented Dec 14, 2020

I just did a couple of tests: stopped arby, manually created buy and sell orders with the arby-specific ids:
buy 1 ltc/btc .01 arby-LTC/BTC-buy-order
sell 1 ltc/btc .02 arby-LTC/BTC-sell-order
started arby again: it successfully replaced above orders. This verifies that #121 (comment) is not the case.

@kilrau
Copy link
Contributor Author

kilrau commented Dec 14, 2020

Taking this out of the December project for now, since it's a non-reproducible edge case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working P2 mid priority
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sangaman @kilrau