Qualitative vs quantitative: from VS to &
If it is possible to A/B test it, why do we need to use qualitative research?
Since there are other precise and numerable data sources (like user metrics), why should we even bother? The truth is that qualitative research gives back data impossible to get in any other way. It can answer “why” while other methods only answer “what,” “how,” “when,” and “where” (Karina, “Qualitative Research Is Always Biased“).
Qualitative research unveils the details: shows how people are using products, why and with which differences. That’s why qualitative research – together with field-based ones like ethnography – is the answer to deliver uncommon and different solutions.
In order to understand why qualitative research is really needed to complete the user research puzzle, here are six reasons for why applying it should be useful in addition to A/B tests.
2_To create hypotheses to be tested: In order to provide meaningful insights, qualitative user research is actually the best tool. With the obtained insights it’s possible to generate hypotheses to test with the A/B method. Obviously A/B test can be the starting point, but qualitative research is going to be faster and more effective. Even during an advanced stage of the project, talking to users is the right tool to adjust the path.
3_A/B test can’t validate features: Not all the features can be A/B tested with meaningful results. Usually, features have so little usage that in order to get an enough consistent sample, the test has to be run forever. Moreover, features have interaction effects that include the ability to react in relation to another user’s action. Even a simple function as “like”, could impact on both people involved in this cross-user situation: A/B test hasn’t the chance to filter it correctly.
4_Unveil new opportunities: In order to expand the reach of a product or service and jump into new customer bases, there is the need to identify those new users’ needs and verify if the solution works for them. This cluster isn’t using the product/service yet, so there isn’t the chance to A/B testing. In this case qualitative research can identify these opportunities talking to prospective customer in order to test prototypes of the new solution.
“A/B testing can yield incremental improvements, but qualitative research is required to jump an entirely new level”
(Jens-Fabian Goetzmann, “Why PMs Need Qualitative Research”).
5_Faster answers: Even if talking to user involves all the existing communication problems, it’s way easier to get a qualitative feedback talking to them instead of actually building an A/B testing. It is much faster and cheaper to build a prototype rather than the whole experience. Moreover, when A/B testing, the hypothesis has to be as specific as possible, making the changes as small as possible. Bigger changes affect multiple aspects of the product/service at once, and there is no chance to be punctual with the quantitative research.
6_Stay humble: Working on a project for a long time gives the feeling of knowing what users need, sometimes having the sensation to know it better than the subjects. This isn’t true. Talking to users on a regular basis provides an indispensable checks and keeps everything on a ground level.