-
Notifications
You must be signed in to change notification settings - Fork 3
Road Map
The following types of bug fixes and enhancement requests will get the most priority.
Before getting started, one important concept is the "Workload Key". It is a JSON string that specifies what code to run. The key can also be encrypted into a text string that can be presented to an end user. For example, give a user two different keys and ask them which key has problematic concurrency? Which would has massively chatty SQL executions? Which has slow response times from requests to a backend? etc....
Would love to present a user and ask "Which of these two keys has a GC problem?" Both keys would reference the exact same code, but would have radically different JVM startup parameters for GC. THIS is how we can teach our IT people how to correctly parameterize our JVMs.
I don't think this should implement all features in a load generator. However, high-level features like "incremental load test" and "steady state" test should be available, with some basic parameters. At a minimum, I'd like end users to be able to configure these kinds of load plans:
https://jmeter-plugins.org/wiki/ThroughputShapingTimer/
https://jmeter-plugins.org/wiki/ConcurrencyThreadGroup/
We'd like to enabled both self-paced and instructor-led performance education. Do your best performance experts learn by just reading books? Or have that repeatedly been integral in diagnosing and massive production performance/stability issues? We'd like performance experts to use this to package and their experience to educate those who weren't fortunate enough to be in the room where it happens. Novice developers need to get their hands on contended sync blocks, slow unindexed SQL queries, slow backend requests, memory leaks, etc....so they know the tools required to diagnose the problems and implement optimizations.
Does your system's monitoring provide enough "actionable intelligence" to diagnose/fix perf problems? Are your people trained to use the monitoring tools?
Would like to enable load-test-in-a-box to run through a script of defects that run underneath a prospective monitoring system, to verify all the right diagnostic information is available. You could also assess your team's ability to understand/navigate the monitoring tool(s). The script might look like the following. This load specification language from Taurus could be used/stolen/emulated.
Run defect A for 5 minutes.
Pause for 1 minute
Run defect B for 10 minutes (perhaps include some load parameters).
In parallel, run defects F,T,X,R and Z for 10 minutes
....etc
Monitoring containerized platforms is a complex task. The above "Sample Performance Defects for Assessing Monitoring Systems" functionality would need to support both Docker and OpenShift. When load is applied, it would need to point to a load balancer.
Imagine creating some code with a sync block, then creating two workload keys that points to the exact same code. The first key would have a low amount of load. The second key would have more load -- just enough to raise concurrency to the point that degrades performance. Then you can present developers with these two keys and say "This is what happens when you introduce a sync block without load testing."
Need lots more load generator metrics for the end user, this is an absolute prerequisite for the "inter-cloud" enhancement below.
Many software performance books have been published with accompanying source code, like these:
https://github.com/vladmihalcea/high-performance-java-persistence
Would like the load-test-in-a-box to host the source code for these and other books.
Redeploying one component of your system to a different data center? Wouldn't it be great to use this to provide a quick picture of what performance should look like. Imagine a system with these components, all in 1 data center. A-->B-->D-->F,G,H
Would like this project to be able to quickly reconfigure a test system like this to run some components in one data center, and some in the other data center. That functionality would be a great help to answer questions like this: How would performance be impacted if we relocated component D to a separate data center? For example, below are three different configurations being evaluated. Each could be encoded in a different Workload Key. The load-test-in-a-box would need to reconfigure itself with each key. For example, say C-->D communications was JDBC. In the first example, component C would point to a JDBC URL of D, that resided on DC_1. In the second example, that same JDBC URL would point to DC_2. ....and this would make it trivial to switch between workload keys to compare performance.
| DC_1 | DC_2 |
|---|---|
| A | |
| B | |
| C | |
| D | |
| E | |
| F | |
| G |
| DC_1 | DC_2 |
|---|---|
| A | |
| B | |
| C | |
| D | |
| E | |
| F | |
| G |
| DC_1 | DC_2 |
|---|---|
| A | |
| B | |
| C | |
| D | |
| E | |
| F | |
| G |
How much of the CPU/RAM is cause by monitoring/load generation, and how much by your system under test?