A/B Testing And Your Website

abtestingwebsiteEvery month here at Breaking Even, we try to have a loose theme. This theme determines a series of blog posts, our monthly email newsletter subject, and gives us an opportunity to dive into a concept we think people should care about.

How many of you have said one of the following things?

“My customers would never be interested in that.”

“My website visitors can get to my shopping cart just fine.”

“I never go to our website on my phone.”

We ALL say them (I am including myself here) because, after years in our business, we have a pretty good idea of what our customers want and don’t want.

Thing is, we don’t know what anyone thinks. Ever.

We can guess. But we can’t know.

We can really only manage what we measure.

So in order to get an accurate view of what is and isn’t happening is present the same set of customers with two almost identical options with one variable change (ex: color of a button) and see how people react. This concept is called ‘A/B Testing’ and is this month’s theme! If you want to know some basics about A/B testing, I wrote a lovely blog post about it previously. Click here to read it.

OK, so what can you test on your website?

  • Color of a button
  • Position of a button
  • Size of a button
  • Writing on a button
  • Lead photo on a page
  • Headline of a page
  • Size of product photos
  • Featured product photo
  • Length of a form
  • Color of text
  • Text in your menu
  • Position of your menu
  • Size of writing in your menu
  • Number of photos in slideshow
  • Order of photos in slideshow
  • Changing a step in an order/checkout process (ex: wording on a particular page)
  • Changing position of sidebar (or adding a second sidebar)
  • Changing position of item in sidebar/footer
  • Changing color of sidebar background
  • Length of testimonials
  • Size of social media sharing buttons
  • Position of social media sharing buttons
  • Lots more!

I wanted to make that list somewhat ridiculously long but I think you get my point.

What should you test?

Deciding what you are going to test to start off with may be as simple as deciding what’s important to you. For example, if email signups are important, you may test an email signup positioned in header area versus sidebar and see how this change affects signups.

You may want to start with testing something ‘controversial’. Maybe everyone in your company argues about whether your products should have a plain wood or a plain white background.

When should you test?

You obviously need to test long enough to collect some data, so when and how long you run your test may depend on how much web traffic you get, among other factors. That said you probably don’t want to do this during a really important time, say, during the holiday rush. Think long enough to get data to make a decision and not so long that it annoys everyone.

How often should you test?

You don’t want to be in testing mode all the time, so plan your year accordingly. Quarterly tests may feel often enough, or twice yearly may be more than enough. Just put it on your calendar/agenda as something to do at a regular interval.

How do you test?

This is probably your biggest question: how do you do split testing on a live site?

You have three options:

    1. Using your current website software (ex: Wordpress) with Google Analytics goals. There is a robust discussion and how-to here: http://www.wpbeginner.com/wp-tutorials/how-to-ab-split-testing-in-wordpress-using-google-analytics/ Note: this is somewhat limited as, for example, it would be somewhat impossible to use this method to change, say, menu text as it works best on main page content.
    2. Use an add-on to your current website software. So there are tons of Wordpress plugins that allow you to, say, split test headlines “Title Split Testing For Wordpress”, or do more general changes. Some ideas here: http://wplift.com/ab-testing-in-wordpress (Note: just because a plugin works, doesn’t mean it’ll work on ALL websites so test test test! Want to know how to pick a plugin? Read this.)
    3. Use a third party A/B testing service like Optimizely or VWO. Check out their complete guides to A/B Testing, here for VRO and here for Optimizely (not affiliate links). Note: Remember if someone makes something easy for you, you’ll likely have to pay them.

If you’d like to learn more about A/B testing this month, subscribe to both our blog and monthly email newsletter. 

A Few Reasons That Didn’t Work

You know that thing you did… the one that didn’t work?

No, I don’t know about it exactly. I just know I have a few things I’ve done that didn’t work and assumed you had at least one too.

So why didn’t your last commercial/coupon/event/blog post/insert-thing-here work?

You took the ‘doing homework’ shortcut by surveying your friends and not your customers.

Whatever you do, don’t ask your friends what they think of your idea. Because they will say your new haircut is awesome, right?

Your website, much like my asymmetrical haircut may have been cool... back in the 90s.

Me in the 1990s. You’re welcome.

Your friends will lie to your face because they love you. Before you pour a lot of time and money into something, you need some unbiased, ideally stranger, opinions. It’ll probably be more involved (re: expensive) than asking your friends over pizza but it’s better than the money you lose chasing a bad idea, right?

(A great way to get some feedback if you are shy is to use paid ads like Facebook or Google to test messages. This is part of what paid ads are for!)

You already tried it before and it didn’t work that time either.

Sometimes we really really want something to work. But it doesn’t.

So think “Is this like that time I…?” And if it is, and if that time things didn’t go so well, there better be a lot more about it that is different than what it has in common with your last mediocre (or terrible) initiative.

You can tell people to pivot over and over... but that couch still won't get up the stairs.

You can tell people to pivot over and over… but that couch still won’t get up the stairs.

You didn’t tell enough people about it.

Let’s say you think email is amazing. Well, your customers are tweeting, Facebooking, pinning, tumbling, blogging, and doing all kinds of other technological and non-technological ‘ings’ to get their information. So the more ways you get the word out and the bigger your audience, the better this is going to go.


You picked a bad time.

Let’s say you’re OKCupid and part of your coverage area is experiencing devastating floods with thousands of people stranded. Not a good time to say…



Sometimes your idea is good but it’s badly timed. If you did all your homework, tried something new, and told a lot of people, this is probably at play.

So truthfully, was your last bad idea one of these things? And do you have a tendency of repeating any of these patterns?

Marketing Monday: A/B Testing

The idea of A/B testing has long been used with designers but definitely has applications with online marketing as well. It was my friend Lynn Cyr, a user interface designer, who introduced me to the concept via a blog.

A/B testing is simply testing two versions of the same email newsletter, landing page, contact form, etc. and see which one performs better.

Here is a simple example from CampaignMonitor.com (click here to get the full post from the site):

There's not much difference in this email campaign except the words used in the link. Guess which one did better?

There’s not much difference in this email campaign except the words used in the link. Guess which one did better?

So I am going to kill the suspense here: Campaign A had a higher response rate. Here’s the graph for those of you who like a visual:

You can lie to yourself about how effective something is but your numbers won't lie for you.

You can lie to yourself about how effective something is but your numbers won’t lie for you.

And you may think, great, nerds the world over can figure out how more people can fill out a contact form. What does this have to do with me?

What if I told you you could test a new website configuration to sell more products?

ABTests.com has visual case studies you can learn from and be inspired to try yourself.

ABTests.com has visual case studies you can learn from and be inspired to try yourself.

In the example above, adding a ‘Popular Products’ feature increased sales with those products while people were looking at related products (AKA cross-selling). This example makes sense but not all of the case studies seem logical. Here’s one with interesting results from ABTests.com:

You'd think the cell phone would speak better to active technology saavy folks on the go (at least I would!) but it didn't. That's why we test, right?

You’d think the cell phone would speak better to active technology saavy folks on the go (at least I would!) but it didn’t. That’s why we test, right?

Here’s the thing: As business owners, we can not assume we know what our customer thinks. Scientists run experiments and psychologists conduct studies precisely because, while we can come up with a prediction, we always have to test it. But besides the quest for knowledge, why A/B test your Facebook page, website, email newsletter, blog, or anything?

1) More sales with less work.
Let’s say you spend an hour setting up Version A and Version B of something. Wait a month while data collects. Look at the numbers, do what works. Even if you only increased sales by 5%, that’s more money for the same amount of effort, that means more money for the same amount of website traffic you are already getting.

2) It’s not expensive.
If you want to DIY, companies like Visual Site Optimizer offer a 30 day free trial (and less than $30/month after for small businesses) which allow you to set up your own tests and track the results. Or you can have your favorite nerd set up something for you. Even if you pay a nerd like me $75/hour for 1-2 hours work, you’ll more than make that money back with your improved results.

3) It means better results over time.
So you find out your customers like one website layout over another. Guess what you’ll consider when you redesign your email newsletter? Exactly. A/B testing means you get better at reaching your target customer over time, further improving what you are trying to do both on and off your website.

Your website visitors can tell you so much without saying a word. And while we’d all prefer to think our websites are perfect, we can always do it a bit better. A/B Testing just helps you get to that better level quicker.

For more information about A/B testing including setup, check out this article from Smashing Magazine.

Or contact me if you’d like me to set up an A/B test for your contact page, newsletter, Facebook ad, or whatever!