Document how an SUT should be deployed in order to get reliable benchmarking

[davidkel]

If you aren't benchmarking a duplicate setup of your production network to determine it's max TPS for example, then you might be benchmarking an environment to determine things like

1. chaincode performance changes
2. fabric configuration changes
3. fabric code changes

In order to get a fair comparison, the environment that is running hyperledger fabric must be consistent in the resources it has at it's disposal. That means the same amount of CPU power as well as having as much opportunity for disk i/o and network i/o as possible. To ensure this the systems running the hyperledger fabric components must ensure that no other processes are using CPU, interacting with the disk or network, or it needs to be as minimal as possible and for each run the same processes should be running
Linux also does are fair amount of in memory file caching and as that can vary it could influence benchmark results.

Cloud VMs are generally not suitable but would a dedicated environment be suitable so long as the hypervisor config doesn't change ?
Could local VM technology be used to guarantee consistent resources ?