Forward condition

[leobago]

It would be good to implement a forward condition that can be selected in the configuration file. This forward condition, as it name says, is only for "forwarding" data, so it does not apply to the block proposer, but only to other validators during block dispersal.

The forward condition can be any of the following three options:
a) Forward every single data sample received (current strategy)
b) Forward only rows/columns that are recoverable, so at least 50% of the data there
c) Forward only full rows/columns

[cskiraly]

Actually b) and c) are the same if we do recovery on the fly. We keep that as a parameter in code, but in practice we will do it, so the two are the same. We also forward only half of the row, even if the full row is available (again, parametrized in code).

I'm wondering if there are other meaningful conditions:

• Would it make sense to forward only if, say, 25% of the line is available?
• Would it make sense to make the forwarding condition based on full state across all lines and columns observed?

[leobago]

Well, they are the same only if you assume that we send only half a row/column even when we have the full one, which might be the case sometimes to save bandwidth but this is not written in stone.

In the case we do send full rows/columns when we have them (for whatever reason), then there is a clear difference in the failure rate.

Other conditions might be interesting to explore indeed.

[cskiraly]

What I was saying is that they are the same as a condition. Not because we send half line only (if turned on in code), but because if you have b), you can repair, and if you do (if turned on in code), c) is immediately satisfied.

[cskiraly]

But I'm afraid such policies would introduce a huge bootstrap problem. Maybe something like the number of nodes we are getting segments from is a better indicator.

[leobago]

I think I see where the confusion might come from. I meant b) forward if you received at least 50% of it, vs c) forward only if you received the full row. Regardless, of having repair on the fly, whether you received 50% or 100% says something about the overall failure rate.

So in the case, a node receives 75% of the data, the node will:

c) this is the most restrictive mode (probably too conservative), in which with the 75% I reconstruct the data up to 100% but I will not forward it because it did not receive 100% of it.

b) repair and forward because it received over 50% of it.

Then again, this depends on the failure model. So far we are assuming sample failures (although not sure how realistic that is).