Monday, December 31, 2012

What Spending $252,000 On Conversion Rate Optimization Taught Me

ab testing

Over the last year and a half I spent a total of $252,000 on conversion rate optimization. That money was used to hire 3 firms: Conversion Rate Experts, Digital Telepathy, and Conversion XL.

All 3 firms produced a positive return on investment, but if I knew what I know now, I would have made a lot more money. Although conversion rate optimization (CRO) in it’s simplest form is finding out why your website visitors aren’t converting into customers… and then fixing it, there is a lot more to it.

Here’s what I learned by spending $252,000:

Lesson #1: Gather qualitative and quantitative data before you start testing

You probably already have some ideas on how you can boost your conversion rate, but before you start testing them out there are a few things you should note:

  1. Do not run test based off of your gut, you’ll end up running a ton of tests that fail. Instead make sure you base all your testing off of data.
  2. Numbers don’t paint the whole picture, for this reason you need qualitative data… so don’t be afraid to ask your current customers and potential customers questions.
  3. It’s better to have more data then less. For this reason you should spend a month gathering and analyzing data before you start testing.

When I first started paying CRO companies we just tested based off of our Google Analytics data. It wasted time and money as it didn’t produce an ROI. Since I have started gathering both quantitative and qualitative data and started making decisions based off it, the ROI has been huge. The $252,000 in spend has turned into millions in additional revenue.

Lessons #2: Do A/A tests before you do A/B tests

When I first started A/B testing there were instances in which the new variation had big increases in conversion… such as 30% or 40%. However when I analyzed revenue increase I didn’t see much if any increase at all.

Do you know why?

The software I was using for A/B tests was not great. This is why you should run an A/A test before you run an A/B test. Meaning you take your original variation of your landing page and test it against itself. After a few hundred conversions, if the conversion rates aren’t very similar, it means your software is probably off.

It’s really important that you run an A/A test first as this will help ensure that you won’t be wasting time with inaccurate software.

Lesson #3: Don’t expect increases on a monthly basis

When I got into A/B testing I expected increases in conversions on a regular basis. At least every other month.

Boy, was I wrong!

From my experiences you tend to only get a few big wins each year that drastically affect your revenue in a big way. Those wins typically make up for all of the fees you pay to consultants. This is really important for you to understand if you have cash flow issues, as you will be out a lot of money before you make it back. CRO is a long-term investment, not a short-term one.

In other words, don’t expect to make a return on your investment within the first 3 months. Expect to start seeing a return in about 6 months and by the end of 12 months you should be cash flow positive on your CRO investment.

Lesson #4: Multivariate tests never work… or at least for me

If you don’t know what a multivariate test is, check out this article.

In a multivariate test, a Web page is treated as a combination of elements (including headlines, images, buttons and text) that affect the conversion rate. Essentially, you decompose a Web page into distinct units and create variations of those units. For example, if your page is composed of a headline, an image and accompanying text, then you would create variations for each of them. To illustrate the example, let’s assume you make the following variations:

  • Headline: headline 1 and headline 2
  • Text: text 1 and text 2
  • Image: image 1 and image 2

Now that you know what it is, I recommend that you stay away from them. Every time I’ve taken the winning elements from each multivariate test and made them the default version, I never seen the increase in conversion that the testing tool is showing that I should get. I’ve tried this with multiple tools and have had statistical significant results, but the results never equate into huge revenue increases.

You can try out multivariate tests, but I personally never find them to work in my favor.

Lesson #5: Don’t optimize for conversions, optimize for revenue

Most CRO consultants focus on increasing your conversion rate, but they don’t focus on increasing your revenue. Which at the end of the day, is all that matters.

When you are running tests, you’ll quickly get an understanding of how a conversion decrease can increase revenue. The quickest way to do this is to increases you prices.

For example, assume that you sell flowers online. Out of every 1000 people that visit your website, 5% convert into a paid customer. And because you charge $10 for each flower you sell, you end up making $500 in revenue for every thousand visitors.

Now lets assume you decide to increase your prices to $20 per flower. Due to your increased prices, now out of every 1000 people that visit your website, only 3% convert into a paid customer. In this scenario you make $600 in revenue for every thousand people that visit your website. Even though you conversion rate went down from 5% to 3%, you still were able to make more money by increasing your prices.

So when you are working with CRO consultants, have them focus on optimizing your revenue, not conversion rates.

Lesson #6: Focus on macro conversions, not micro conversions

The difference between macro and micro conversions is that macro focuses on the big picture while micro focuses on the small picture.

For example, a macro conversion would be how many people end up buying your product. A micro conversion would be optimizing how many people click the “add to cart” button or view your “pricing page”. As you already know, just because someone added something to their cart or viewed your pricing page, it doesn’t mean they will purchase your product.

When running A/B tests don’t run tests that will boost your micro conversions, as that will not guarantee a boost in your macro conversions. Focus on macro conversions such as increasing the total number of sales, instead of optimizing how many people view your pricing page.

Lesson #7: Drastic changes = drastic results

Once you optimize your conversions by making all of the major changes, you’ll notice that small tweaks stop having huge impacts on your conversion rate. From headlines to button colors, these small tweaks will stop having huge impacts on your conversion rate.

It’s not that those small tweaks aren’t important, it’s more so that you’ve taken care of all of the low hanging fruit that is stopping people from converting. At this point you’re best chance of boosting your conversion rates, or more importantly revenue, is to make drastic changes.

From changing up your signup process, to forcing people to signup for a free account before you upsell them, you have to make drastic changes.

Many of these changes won’t work out, but some will have a positive impact. Just get creative as that is the trick to boosting revenue. For example, a drastic change I made that tripled my contact requests was changing my contact page to an infographic.

Lesson #8: Don’t forget to optimize your backend for conversions

When you think of optimizing conversions, what comes to mind? The concept of turning more visitors into customers, right?

Although that is CRO, it doesn’t mean it has to stop there. What about increase the lifetime value of your customers? Like getting them to spend more money with you or getting them to refer their friends to you.

There are a lot of things you can do to boost your backend conversions, so don’t just focus on the front end. In many cases it is easier to optimize your backend than front end, so focus on both.

CRO consultants from my experience love working on the front end of your website, but they can also do wonders for your backend… so make them work on both.

Lesson #9: Consultants aren’t miracle workers, they need direction

That’s right, CRO consultants aren’t miracle workers. Just because you are paying someone 6 figures a year to help you boost your conversions, it doesn’t mean they will actually produce results.

If you want to get the most out of your CRO consultant, here are a few things I recommend doing:

  • Require that you have a call at least once every 2 weeks.
  • Assuming you have enough traffic, make it a requirement that you have to run at least 2 tests a month. It’s a numbers game…
  • Make them gather and analyze quantitative and qualitative data every quarter. What your customers have to say change over time.
  • Have them focus their efforts on creating wireframes and writing copy. Most CRO consultants are slow at design, so might as well have them focus their time on what they are best at.
  • Don’t expect your consultant to come up with all of the ideas. You know your business better than anyone else, so make sure you throw your ideas out there. Our biggest conversion increases came from ideas my business partner and I had.
  • Make sure you are working with a consultant who is good at executing. What I’ve experienced is that tons of consultants understand the concepts of CRO, but many of them suck at execution. If they suck at execution, nothing will get done.
  • Once you find a consultant you really like working with, prepay them for future work and ask for a discount. You should optimize your spend. We do it with all of the consultants we work with and it easily saves us over 20% a year.

Lesson #10: Just because you had huge wins, doesn’t mean you will see large revenue increases

Even if you are using good A/B testing software, focusing on optimizing revenue, and measuring macro conversions instead of micro, it doesn’t mean things will go the way you want.

Over the last 1.5 years I’ve noticed a trend that just because a test says it increases your revenue by 30%, doesn’t mean it will maintain that increase in the longrun. I am not 100% sure why, and nor are the consultants I work with, as they have seen this happen too.

Even with statistically significant tests, those 30% revenue lifts tend to be 15% lifts in the long run. My best guess is that there are other variables that come into play, such as the quality or volume of your traffic changing over time.

This doesn’t mean you should discount those tests or stop testing. Instead this means that you need to constantly test and work on optimizing your conversions/revenue. It’s a never ending game.


Have you started optimizing your conversions yet? If not, I hope this post encourages you to start, as CRO can help you make millions.

Before you can start, the one thing you’ll need to have is traffic volume and conversions. If you have less than 10,000 monthly visitors or $200,000 in yearly income, it maybe hard to optimize your conversions.

On the flip side if you have over $500,000 in yearly revenue, then you should consider making CRO a line item expense. Like how you would constantly pay a book keeper or accountant each year, you should constantly pay a CRO consultant.

What have you learned by optimizing your website for conversions?


Michael Pragnell Thomas P. Mac Mahon Frank O\'Halloran Daniel Hajj Aboumrad Philippe Varin

No comments:

Post a Comment