Metric format

[barneydobson]

Fixes #54

Begins #49

OK this is a reasonably small PR, but quite an important one.

@dalonsoa for your context - we will run the model lots of times with different parameter values - and the thing that we will study in the paper (and something I hope many other potential users are going to be interested in using our package for) are various metrics. These metrics will in some way capture how good a job the synthetic network has done versus the real network. There are a few different 'categories' of metric (see #49 ) - e.g., whether comparing the 'shape' of the network, or flows at a critical location - etc. From a compute perspective, the metrics are essentially 'free information' about the method since, in comparison to the network derivation and simulation, they take basically no time to calculate (with the exception of some complicated graph metrics, we can discuss whether these are necessary in #50 ).

It would be great to get a review from both @dalonsoa and @cheginit on this one since ideally we will implement many many metrics, so I want to make sure they are done in a thought through format before commencing. In particular here one thing I'm not so sure on is the need for a BaseMetric - I was copying from graphfcn so I included it. But there is no parameter that the metric has to take (unlike graphfcn, which needs G), and instead a subset of 6 arguments that the metric might take (see below).

This PR has two example metrics (bias_flood_depth and kstest_betweenness) to demonstrate.

As with graphfcn I have put here an example to implement metrics using a register. We would iterate over metrics (provided in some configuration file) and pass a few arguments, something like this:

# Evaluate metrics
from swmmanywhere.metric_utilities import metrics
results = []
for metric in metrics:
    val = sm[metric](synthetic_results = synthetic_results, 
                     synthetic_subs = synthetic_subs,
                     synthetic_G = synthetic_G,
                     real_results = real_results,
                     real_subs = real_subs,
                     real_G = real_G)
    results.append(val)

[dalonsoa]

If speed is a concern, I would get rid of the BaseMetric and register just simple functions. Creating classes has some overhead (not huge, but some), and having them here adds nothing.

If you want to make sure that the metric functions really accept the right arguments, you could use the inspect module. This runs when registering the function, not when running it, so it has no overhead at runtime.

[barneydobson]

If speed is a concern, I would get rid of the BaseMetric and register just simple functions. Creating classes has some overhead (not huge, but some), and having them here adds nothing.

If you want to make sure that the metric functions really accept the right arguments, you could use the inspect module. This runs when registering the function, not when running it, so it has no overhead at runtime.

@dalonsoa OK thanks - I've now implemented I think this is a cleaner solution, hopefully you agree! Thanks for the tips

[dalonsoa]

This looks excellent 👍

[barneydobson]

@cheginit it seems like we're happy with the format of metrics, but in the interest of not starting out with dud metrics, can I just check that comparing the distribution of nx.betweenness_centrality of two graphs via a KS test is actually a semi reasonable thing to do?

[cheginit]

That's a loaded question 😄

First, regarding computing BC, networkx can be very slow (computing BC is computationally expensive in general), that's why I use networkit. Second, for comparing graphs in the context of optimization, there are more suitable metrics that we can choose from. For example, there is an interesting discussion here. You can also check out the distance measures or s-metric in networkx. It appears that there's a new backend for networkx that speeds up some slow operations in networkx, called graphblas

[barneydobson]

OK that's super helpful - though I'm going to bring it over to #50 since that's probably the best place to discuss loaded questions about graph comparisons ;)

[cheginit]

Other than the comments, it looks good to me.