Testing our assumptions – the results are in!

Lachy Gray, 4 min read
dave-scientist-dancing@2x

We recently ran our first campaign on the Yarno platform with Mortgage Choice. It was a 7-week onboarding campaign for new starters with the overall objective of improving baseline knowledge and confidence.

In a previous post, I explained the many assumptions that the Yarno team made when designing the platform (we’re only human after all!).

We saw our first customer campaign as the perfect opportunity to test these.

So we defined 6 hypotheses and how we’d measure them. This was as much an exercise in writing and testing hypotheses as it was testing the Yarno platform. 

Drumroll, please…

And here are the results (with the If then column removed):

Let’s take a look at each hypothesis in turn.

1.1 Learners will log in every day to answer questions

We believed that >90% of learners would log in every day to answer questions. This was false. A core number of learners logged in daily, around 20%. Interestingly, this group was most likely to be at the top of the leaderboard.

The majority of learners in the campaign logged in every other day. A small number of learners waited a few days then answered many questions at once in an attempt to boost their score (this doesn’t work, by the way!).

The most popular time to answer questions was in the morning between 8am-11am. And surprisingly, the most popular day to answer questions was Friday (Monday was the least popular day).

1.2 Learners will spend time on the explanation screen

We believed that learners would spend >30 seconds on the explanation screen when they answered incorrectly. We were unable to properly test this hypothesis as we discovered that it’s super difficult to accurately identify time spent on a particular browser screen.

Consider a learner who starts reading the explanation and then switches browser windows to check their email. When they return to the explanation 20 minutes later, the browser tells us that they’ve spent 20 minutes looking at the explanation, which is incorrect.

We can, however, make some approximations based on time between opening the explanation screen and clicking the Next button. This was around 5 minutes.

1.3 Incorrect question spacing will increase retry performance

Yarno is built around a few core learning principles. One of them is spaced repetition, which is the idea that learning spaced over time is more likely to be retained than learning delivered in one hit.

So, correct and incorrect questions are represented to learners over time. Incorrect questions are presented in approximately 3 days from the first answer, and correct questions are presented in approximately 10 days.

We believed that an incorrect question spacing of 3 days would contribute to a >75% correct on second try accuracy by the end of the campaign. This was true, after 4 weeks second try accuracy was 73% and 77% at the end of the campaign.

In hindsight, this is a weak hypothesis as it’s difficult for us to attribute the outcome (whether achieved or not) to just the incorrect question spacing. For our next campaign, we’ll create some more specific hypotheses.

1.4 Learner performance will increase over the campaign duration

We believed that overall learner performance would increase by 5% over the 7-week campaign. This was true. At the end of the second week, performance was 68%. And at the end of the pilot, it had reached 85%.

This was a pleasing result as learner performance was a key metric. It showed an improvement in overall learner understanding throughout the campaign. Ideally, this knowledge would be reinforced over the coming months for maximum memory retention.

1.5 Learners will keep unanswered questions to a minimum

We believed learners would attempt to keep their unanswered question to <6 throughout the campaign. This was false. In hindsight, this hypothesis could have been more specific since we discovered that by using an average the results were skewed. Although the majority of learners kept their unanswered questions to <6, a small number of learners had unanswered questions much higher than this, which skewed the average.

Interestingly, those learners with high unanswered questions were consistent throughout the campaign. How to engage with this type of learner will be a key focus in future campaigns.

1.6 Learners will use the daily reminder email to answer questions

We believed that we would achieve a 90% Click Through Rate (CTR) on the answer questions button in the daily reminder email. We thought this was the fastest and easiest way for learners to answer their daily questions. However, this was false.

The average CTR for the reminder emails was 46%, a surprising result. Especially when considering the average email open rate was high, at 76%. The disparity between the CTR and the open rate suggests that learners were logging in independently of the reminder email. We hadn’t expected this behaviour and it’s definitely something we’ll be focusing on in future.

Where to from here?

Testing our assumptions was a fantastic exercise, in both writing and testing hypotheses and putting the Yarno platform through its paces. The platform needs to be flexible enough to cater for the many different ways learners will interact with it and the Mortgage Choice campaign was our very first customer test run.

Overall we’re happy with the results, as we have a bunch of learnings that we’ll be looking to consolidate and action in the coming weeks. We’ve identified some key areas for improvement, and are looking forward to testing these when our second campaign kicks off in May.

Mortgage Choice also considered the campaign a success. 88% of all questions asked were answered, and learner performance improved by 16% over the 7-week campaign.

Learners reported feeling more confident about and aware of Mortgage Choice products and partners, and felt better aware of key policies and procedures. A majority of learners also reported having a clear understanding of how their role relates to the company vision and strategy for 2017 and 2020.

We’ll be writing a case study on the Mortgage Choice campaign in the near future. If you’d like any information in the meantime please get in touch!

Lachy Gray

Lachy Gray

Lachy's our Managing Director. He's our resident rationalist and ideas man. He also reads way too many books for our liking.

More from Lachy Gray

We'd love to chat about how Yarno can benefit your business

Mark Eggers

Mark, our Head of Sales, will organise a no-obligation call with you to understand your business and any training challenges you’re facing. Too easy.