At Yarno we make assumptions every day. From the micro “Learners will know to rollover a tooltip to view it” to the macro “delivering learning via a quiz format will benefit both learners and managers”. Most of the time they sound plausible and are accepted at face value.
It’s once we’ve accepted the assumption and ask, “Ok, so how are we going to measure it?” that things get tricky.
Announcing an assumption is cheap but testing it can be expensive.
Why test assumptions?
Since Mark and I started Yarno we’ve looked to research based methods for guidance.
Whether that was interviewing potential customers about their pain points Lean Canvas or digging into the learning science archives - we wanted the platform to be grounded in reality.
9 months ago our assumptions had no real consequence. But now that our first pilot has kicked off, the importance of working with assumptions that can be validated is of vital importance.
After all, we’re cognitively predisposed to finding evidence to support our existing beliefs. But we struggle to actively look for information that may contradict what we already believe.
Enter science
A few weeks ago we were discussing a few assumptions we’d made about Yarno, when our coding wizard Paul suggested we write down a few hypotheses, and (drumroll please)…test them!
We all agreed it was a great idea, however, we weren’t quite sure where to start.
So I did some research (thank you Barry O’Reilly and Ben Yoskovitz) and came across a useful way to structure hypotheses:
We believe that ( this capability )
will (do this action).
To validate this we will ( hypothesis test ).
We believe that (this capability)
What’s the assumption that we’re testing?
Will (do this action)
What are we expecting the user to do?
To validate this we will (hypothesis test)
How will we know that what we’re testing has succeeded or failed?
An example
We made an assumption early on that learners would want to answer questions every day, rather than letting them pile up and answering them in one hit.
We believe that >90% of learners will login every day to answer questions. To validate this we will average the login frequency metric in Mixpanel (analytics aggregator) at 1, 2 and 3-month intervals.
Our pilot hypotheses
Massaging our assumptions to fit this framework was challenging at first. Writing a statement that needs to be testable highlights problem areas immediately. Assumptions such as “using Yarno daily will improve learner’s knowledge” were on thin ice.
How would we test it? What does improvement look like? How would we set a baseline for knowledge?
Using the framework above I came up with 6 hypotheses to be tested in the pilot:
If-then
Once I’d defined the hypotheses I thought my work was done. But a comment on Barry’s post caught my eye. It was by Timothy Fitz, who suggested that one aspect that is often missed is what happens when you have the result of your hypothesis. If the hypothesis is true (or false), what does that actually mean? What happens next?
Timothy suggested an additional step when writing the hypothesis. An If-then statement "If true, we'll keep the feature, if false we'll delete the feature and work on an unrelated hypothesis.”
So I started thinking through If-then scenarios for each hypothesis. This was tricky, since it’s a hypothetical exercise and forced me to consider the value of the hypothesis. I quickly discovered that if a hypothesis is true but you have no meaningful way to use that result, it’s a waste of time. A few of my hypotheses didn’t cut the mustard at this point, and were binned.
However, 6 hypotheses did make the cut and are currently being tested. I’ll share the results once I have them!
Lastly, with what we know about the value of testing hypotheses, perhaps the quote from the start of the post should actually say:
Announcing an assumption is cheap and not testing it can be expensive.
Want to check out the results? Find them here!
