This post is part of the ongoing series of reading reflections on HCI papers and articles. You can find the other posts ‘here.’
Putting A/B Testing in Its Place, Nielsen and Norman Group
Available online at https://www.nngroup.com/articles/putting-ab-testing-in-its-place/
A/B testing is a method of comparing two versions of a website or application to determine which one performs better. The two versions are called “A” and “B” and are usually referred to as “treatments” or “variants.” This way, multiple variants can be tested to optimize for a chosen metric (e.g. click-through rate, conversion rate, etc.) that best reflects the business goal. Additionally, they are inexpensive and fast to implement and can be used to test a wide variety of design changes, including changes to the layout, content, and functionality of a website and support validating the effectiveness of usability improvements.
But, A/B testing is not a substitute for qualitative user research. It’s also not a good way to test usability improvements, as it doesn’t provide any insights into why users behave the way they do. Therefore, A/B testing is best used in conjunction with qualitative user research to validate the effectiveness of usability improvements and to test the impact of design changes on a site’s conversion rates or other relevant metrics.
Though I felt the article was dismissive of A/B testing, as it tries to optimize for a business’ central goal (conversion rates, or other metrics), I think using A/B testing with a more sophisticated usability metric, that takes a holistic approach to the user experience, business goals - both macro and micro would be a better approach, but that would increase the distance between the user’s actions and its interpretation, further reducing its insights into the user’s behavior.