Dagstore migration failure during boost migration

[LexLuthr]

This problem was reported by @tmyuu during the SPX testing

Migration has been completed on mainnet.

Lotus version:

Daemon: 1.15.1-rc5+mainnet+git.913478b21+api1.5.0
Local: lotus-miner version 1.15.1-rc5+mainnet+git.913478b21
make note for something went wrong for me:

After running the migrate command, I found unable to access market (boostd) API from the storage miner. Due to missing the token file, I copied it manually then work fine. I also copied the dagstore directory in the same situation; however, it doesn't work fine. Therefore I'm currently running the initialize-all command.
The Web UI cannot be accessed directly from the Global IP. It looks like I'll need to use SSH tunnel or reverse proxy and so on. Error message is Error: Failed to fetch - check connection to Boost server.

[LexLuthr]

@tmyuu Did you keep the error you faced when migrating the Dagstore? Boost migrate should have done it automatically.

SSH tunnel for webui
https://boost.filecoin.io/upgrade-from-lotus-to-boost/migrate-a-lotus-markets-service-process-to-boost#web-ui

[tmyuu]

I ran the migrate command again in another directory, but I don't see anything like an error.
There were no errors in the previous migration, so I had thought it was completed without any problems at the moment.

Getting markets repo '/usr/local/filecoin/.market'
Initializing boost repo
Trying to connect to full node RPC
Checking full node sync status
Worker: 6039; Base: 1701096; Target: 1701096 (diff: 0)
State: complete; Current Epoch: 1701096; Todo: 0

Done!
Checking if repo exists at /usr/local/filecoin/.boost_
Checking full node version
Creating boost repo
Migrating datastore keys
Importing all legacy markets datastore keys under /deals/provider
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 428 legacy markets datastore keys
Imported 30124 legacy markets datastore keys under /deals/provider
Importing all legacy markets datastore keys under /retrievals/provider
Importing 616 legacy markets datastore keys
Imported 616 legacy markets datastore keys under /retrievals/provider
Importing all legacy markets datastore keys under /storagemarket
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 1024 legacy markets datastore keys
Importing 858 legacy markets datastore keys
Imported 4954 legacy markets datastore keys under /storagemarket
Migrating markets keystore
Migrating markets config
Adding miner address f0104671 to datastore
Migrating storage.json file
dagstore directory size: 404 kB
dagstore directory size is 404 kB. Copy [c] / Move [m] / Ignore [i]:
m
Moving /usr/local/filecoin/.market/dagstore to /usr/local/filecoin/.boost_/dagstore
Boost repo successfully created, you can now start boost with 'boostd -vv run'

[tmyuu]

Fortunately it did not have a fatal effect on boostd.

[LexLuthr]

Without the older dagstore, we will not see the older deals as well. Indexer will not work correctly for older deals.
Can you give a quick run down of what steps you took for the migration. I will try to reproduce this based on your steps.

[tmyuu]

I followed the steps here.
My mistake was that I didn't really care because not see the error. Relying on my memory, I ran the boostd dagstore list-shards command and it seemed strange that it shows 'new'. Then I tried the commands in order.
Especially after running index announce-all, looking at the xxx.full.idx seems to have disappeared.
I haven't been able to debug reliably because there is no log file.

[LexLuthr]

All the logs should go to the same file specified for GO_LOGS during market install. Can you check there?
Logs will help us understand what could have gone wrong in the first attempt.

From your screenshot, it looked like all previous deals are visible in the boost ui. We just need to check about indexing and dagstore.