The potentially biased responses of members in private online communities can be a concern for businesses hoping to use these platforms for insight and co-creation. In my role as Director of Research, I am frequently asked to explain the ins and outs of what makes online communities tick and how to best use them to explore research questions. It occurred to me that, although I usually do this for our clients, others out there might have similar questions. I would like to share a response I helped to write recently, addressing the concern that feedback from community members might be biased.
In our 10+ years of experience we have found that members across all types of communities (B2B, B2C, various industry verticals, etc.) do provide constructive, actionable and appropriately critical feedback. Online communities are a highly engaging and interactive method where qualities such as transparency, reciprocity (between researchers and participants), and collaboration are the norm. It is because communities are branded, open and relationship-based that they generate so many rich and insightful contributions from deeply committed customers. However, the very things that make communities a valuable resource also introduce the potential for systematic error, or bias. These concerns are understandable and drive the fundamental questions all of us wrestle with in using online communities as a research methodology.
To get an overview of the various forms of bias with which researchers contend in online communities (and other social media driven methods) you may find it useful to read our recent report, Leaving Our Comfort Zone: 21st Century Market Research. In this position paper we show how the “weaknesses” of communities can actually strengthen validity, delineating eight tradeoffs researchers need to think through to make informed and effective decisions. We spell out the risks for various kinds of bias, the rewards associated with a collaborative and engaging approach (which are often overlooked), and provide some examples of how our clients have worked through some of these issues.
We have also conducted an empirical study exploring the relationship between concept test ratings and tenure in community (the concern being that members’ feedback would grow increasingly positive the longer they participated in a given community). Looking across multiple online communities, we found this to be untrue: If anything community members become more discerning over time, and so you may want to read the report to get the details.
And finally, in the early years, our clients did a great deal of parallel testing to ensure that our methodology produced research comparable with traditional methods. We cannot disclose their proprietary results, but in over 25 different cases, our clients found no significant differences in outcome. Community members and traditional research participants rate concepts similarly: Top features and bottom features are consistent.
Thanks for your great question and keep them coming!