I was halfway into a post about modeling value and why reliable tech skill is universally important, but I received several independent criticisms regarding being more concrete. In response I’m going to address measuring the success or failure of your experiments.
How to Measure Success and Failure
The short answer is, I can’t tell you. The more satisfying answer is that there’s an easy way to find out yourself. Success depends entirely on your definition, which you can freely choose to define before running tests. Here’s an example of something I plan on testing tonight on stream:
Waste or Value Discovered: By watching videos of my play, I see that I consistently miss a significant number of punish opportunities on knockdown.
Root Cause Analysis: To ensure that I’m treating a root cause and not a symptom, I turn to the 5 Whys.
- I always hard-read tech in place or no tech
- I prioritize hard punishes over consistent punishes
- It’s extremely low effort because it’s now ingrained in muscle memory
Prescription: I have to update my strategy and commit to consistency over hard punishes. To counteract the muscle memory, I’m going to have to heavily invest in focusing up and not going on autopilot after knockdown hits. This is what I think the 5 whys are telling me to do:
- Not a root cause
- Root cause – adjust strategy to increase priority of consistency
- Root cause – invest significant time into breaking muscle memory
Hypotheses: This is a leap of faith that hasn’t been proven yet. The hypothesis will be either validated or disproved based on experimentation. Mine is as follows:
Increasing my reaction-based punishes will deliver more value.
Now I have to create metrics, establish a baseline, define my success criteria, and come up with experiments.
Metric Creation: What is important here? I can think of a few things.
- % of punish attempts that cover a legitimate option
- % of punish attempts that are reaction-based vs. prediction-based
- Success % of reaction attempts
- Success % of prediction attempts
- Influence of stage / matchup on outcomes.
Obviously if I’m covering no options, I have a big problem. Based on the distribution of reaction vs prediction attempts, I should be able to maximize value. Note that this experiment (unlike focus) will likely have a LOT to do with matchup. I should measure this too.
Establish a Baseline: By watching videos, I can measure my baseline accurately. Remember that more points of data means more accurate measurements.
- Approximately 60% of punish attempts cover a legitimate option
- 10% of punish attempts are reaction-based vs. prediction-based
- Success % of reaction attempts – about 25%
- Success % of prediction attempts – about 10%
- Influence of stage / matchup on outcomes. (no data)
Success Criteria and Experiment Creation: Ultimately I want to increase the total value of punishes and follow-ups. But, on the road to getting there, I want to be able to think and make the right decisions based on the situation. Currently I’m just guessing in place, so from my perspective, any steps towards delivering on my goals are an objective improvement.
In this case, OKRs sound like a great tool. My objective is to increase consistency in punishes on knockdown. What are my Key Results?
Key Results: The point of OKRs is to set up sub-goals that, if completed, will demonstrate that I did everything I could to satisfy my objective of increasing consistency in punishes on knockdown. The 5 whys prescription is a great starting point and provides me with much-needed direction.
KR1: Follow up on reaction after ¾ of knockdowns (5 Whys Rx 3)
KR2: Successfully follow up on 25% of reactions
KR3: Increase hard-read success rate by 50%
Setting up strong KRs is my job, and my ability to create good experiments will determine whether I actually achieve my objective. The logic is that currently on knockdown I’m attempting hard-reads nearly 90% of the time, attacking in-place or missed techs. I want to invest heavily in not doing this to break the muscle memory first.
I’ll update this post tomorrow after measuring the data to determine the results. To find out sooner, join us live on the stream!
Subscribe to @bobbyscar for update notifications
Join the discussion at The Lean Melee Smashboards Thread
Watch our stream on twitch.tv – next broadcast indicated at the top of the page
Check out the new home for NorCal Melee Videos on YouTube
Email questions to firstname.lastname@example.org