A/B Testing TIPS

SEE ALL TIPS

GET THE LATEST TIPS EACH WEEK

Get product resources, conference updates, along with access to an exclusive Slack channel by subscribing below.

Are A/B Tests Ethical?

 “A/B tests are easy and seem harmless, but many consumers become disturbed when they find out they’re being tested without knowing it. Some argue that A/B testing tracks along the same ethical lines as a product launch; others believe organizations must be transparent about their testing even if it seems harmless.” Hal Conick explores the various ethical implications of A/B testing and suggests that the best way to address it may be to simply be transparent with your users. (via @HalConick)

OPEN

How to Choose the Right Testing Software For Your Business

A/B testing is great to get an idea of how your customers will react to your product. However, in order to make the best use out of the information you get from A/B testing you’ll likely need to do some fairly involved data analysis. Don’t worry, there are several tools that can help you work through this data analysis. David Zheng reviews seven A/B testing tools that are “highly successful, fully supported, and come with a dedicated team invested in making your experience flawless.” (via @Nansida)

OPEN

Guidelines for A/B Testing

When she worked at Etsy, Emily Robinson “benefited from a very robust A/B testing system” that took care of many aspects of A/B testing. Even with that, she found she still spent the majority of her time working on experiments. She realized that “generating numbers is easy; generating numbers you should trust is hard!”  and that “there are many ways A/B Testing can go wrong, but most of them won’t be obvious.” Emily shares 12 guidelines that will help you “guard against some common mistakes and set you up for success.” (via @robinson_es)

OPEN

How to do A/B testing: A checklist you’ll want to bookmark

A/B testing can be valuable because it helps you see how your customers will react to differences in your product. “But A/B tests can also be complex. If you’re not careful, you could make incorrect assumptions about what people like and what makes them click — decisions that could easily misinform other parts of your strategy.” Lindsay Kolowich provides a 15 step checklist you can use to split test your marketing material and aspects of your product. (via @lkolow)

OPEN

The complete guide to A/B testing

A/B Testing is a pretty familiar term to product people these days, but it’s always good to refresh your memory once in a while. The folks at VWO put together this guide to A/B testing which explains what A/B testing is, why you would use it, what you should use it for, how you might do it, and shares some case studies of organizations who have done it. (via @VWO)

OPEN

7 A/B Testing Blunders That Even Experts Make

There are pitfalls to the practice of A/B testing. Neil Patel lists a bunch of potential blunders such as switching the test off after you’ve reached a statistically significant number, believing that variations are always going to be game changers and testing too many variables at once.

OPEN

A or B

For a more generalized overview of A/B testing, it’s good to go to the folks at NPR. The team explores how the practice is invading if that’s not too strong a word, every part of our lives. Ultimately, the question is whether we are seeing real improvements around us or if the world is getting more effective at telling us stories.

OPEN

3 Real-Life Examples of Incredibly Successful A/B Tests

To help you figure out how A/B testing can have an impact, Robin Johnson has detailed 3 interesting examples. Finding a variation that drives up revenue by 43% like Electronic Arts did, truly underlines the importance of this practice.

OPEN

71 A/B Testing Ideas

Almost anything can be A/B tested online. The team at Optimizely have come up with 71 in fact, including copy, visual media and calls to action. The question remains however; what are you going to do with the answers?

OPEN

Related Tips