Planning is everything when it comes to CRO.
Many new time CRO practitioners will rush into conversion rate optimization, implementing test ideas with knee-jerk reactions, and then becoming frustrated when the tests do not perform. How do you set yourself up for success when beginning AB testing?
In this article, I’ll review how to build a CRO Roadmap that will get you results (almost!) every time 🙂
Step 1: Determine your test length
Before you run any type of tests, you’ll want to start by asking yourself, “How long do these tests need to run to reach significance?”
A good rule of thumb is to plan for a 10% increase in conversions through your tests. If your test results are more significant, you’ll simply save time. Planning for the worst, hoping for the best type of scenario.
The easiest way to do this is by using a significance calculator. Here’s an example calculation from GetDataDriven:
In this example, even at 200 conversions, we’ve only reached 86% significance. More likely, to get 95% confidence, you’re needing closer to double this many conversions.
It’s important to do this upfront just so you realize how much time CRO is going to take. AB testing is often a long-term – rather than a short-term – investment, one that continues to pay dividends after your CRO fees are finished.
Step 2: Determine your feedback loop
Once you know how long you’re going to give each test, you should set up what I call a “feedback loop.” This is how you’re going to adjust the test designs for each experiment that you are running.
For most CRO firms and experts, the formula looks like this:
- Generate a test hypothesis
- Let test run to 85-90% confidence
- Determine result
- Add new variation
You can adjust this depending on your goals. Some people prefer to let initial tests run longer so they can measure exact results. Other people will wait until they have a directional idea of where the test is headed, then wait to do a final measurement until the tests are combined.
Having a clear direction on how often you’re going to change tests and adjust results is very important. Otherwise, you will be running scatter-brained from test to test.
Step 3: Conduct your research
Having a solid research plan in place is essential.
User research should begin as soon as you start with your first “gut check” test idea. The most effective methods are:
- Qualaroo surveys: These are quick, easy and inexpensive. Asking customers why they didn’t complete a purchase is one of the best ways to get raw customer feedback.
- User Interviews: Speaking to 3-6 customers via the phone and conducting a more in depth interview is also extremely valuable. Once you’ve spoken to a few customers, you’ll get a better idea of what they value. You would be surprised how few interviews it takes before common themes start appearing. Usually, unless you are selling to a widely diverse niche, your customers will come from the same “tribe” and be similar in their likes and dislikes about your brand.
- Heatmaps/Eye Tracking: These tools will help you identify where users are spending the most time on the site.
User research should be happening at the same time that you are testing and is constant. You should always be asking people what they think to help generate more new test ideas.
And finally …
(4) Determine your significance
Some clients and CRO experts are much more exact in their testing than others.
In order to determine exactly how much your conversion rate changes, you will need to let the test run for at least 4 weeks, and preferably during a time that doesn’t reflect seasonality.
However, in many cases this is not achievable. Clients need to know quickly whether a design is going to work, and then simply run with it. For fast-moving companies, there is often too much going on to wait.
Make sure that you know how exact you want to be in measurement before you start. This will help you plan exactly how much time you need to invest into your CRO testing strategy.
