Separate Graph into it's own Service

[gkrizek]

A very nice feature request would be to enable Sensei to run in a 'graph-only' mode and also let Sensei run pointing at a separate Sensei instance that's running the Graph.

Use Case:
This could enable more robust deployments of Sensei that essentially breaks down components into micro services. Thus
improving reliability, upgradability, and separation of duties.

Tech:
I think Sensei could exist in 3 states:

1. All-in-one: this is the default where there's 1 process for all things in Sensei (like exists today)
2. graph-only mode: On startup you could set a configuration option to make Sensei run only Graph services. No wallets, nodes, etc. Strictly there to take in Gossip Data, build the graph, and service API requests for graph data. It could expose an API for communication.
3. no-graph mode: On startup you could set a configuration option to point Sensei at a separate Sensei that is running in graph-only mode. This would mean that Sensei wouldn't do any gossip gathering and wouldn't have a local graph. Whenever it needs to do things like route calculation, it calls out to the configured Sensei that is running the graph only.

This is part of a large ideation around decomposability of Sensei. Does it make sense to move path finding to a micro service? What makes sense in this model and what doesn't?

Example Architectures:

[johncantrell97]

at first glance this makes sense to me. the idea is if you had multiple Sensei nodes you probably don't need a network graph on each one.

similar to how today the child nodes don't have their own graph and just take the one from the parent.

this is just having the ability to have that graph be external to the sensei node itself. should be pretty straight forward -- want to implement an "ExternalNetworkGraph" or something that can call out to an external service for it's routing needs.

i guess one thing to keep in mind is that the "graph-only" node will still need to be connected to the network in order to receive the gossip. are there any issues receiving gossip with no channels?

[gkrizek]

i guess one thing to keep in mind is that the "graph-only" node will still need to be connected to the network in order to receive the gossip. are there any issues receiving gossip with no channels?

Yes, from my understanding, the graph-only node would still need to have connections out to the network to receive gossip. However I'm almost positive channels are not required for this. Think about a brand new LND instance, it connects to 3 random peers to sync the graph without any channels. So this graph-only node could actually have like 15 peer connections or something to have a very well constructed graph.

Additionally, this also creates more optimization for the no-graph node. This no-graph node wouldn't have to have any peer connections with nodes it doesn't have channels with. So it can slim down the network requirements for the no-graph node as well.

[johncantrell97]

Yeah, pretty sure you're correct about the no channel thing. Though I have a vague memory of matt mentioning something about this before. I think it might be that a peer without channels doesn't have to gossip to you? Which I guess makes sense? Might have just been a technicality but in practice you will receive the gossip.

Yeah definitely another optimization for the no-graph node. It's probably a bit of over optimization but regardless I like the idea of having a routing sub-service that can run on it's own. I think there will be a handful of other sensei sub-services you might want to run standalone as well. So this might lay some groundwork for future services as well.

[gkrizek]

Ah, that could make sense. Probably worth doing a little bit of research and testing to see what the other implementations do currently. I guess the worst case is you could let the graph-only node still have channels for the sake of gossip.