Sentiment analysis is getting more and more attention all over the world in many different areas of business. What was previously the realm of market researchers alone is now directly affecting areas such as finance and investing. While sentiment analysis in and of itself is certainly not new for researchers and has been a central pillar to their value offering for many decades, the ways in which it can be measured and the accuracy of that data are evolving fairly quickly. In the past and still most generally, when a market research company receives open-ended data, it is sent for manual or semi-automated processing to identify the important keywords and overall sentiment of the response. While it can be somewhat costly and time-consuming, open-ended coding is a very important part of research that, while most companies try to minimize, cannot be done away with. After all, while ranking-style questions and precise numerical inputs are much easier to quantify and deal with, the double-edged sword is that the structured data you obtain is only going to be as accurate as the researchers is able to foresee when creating the answer codes or subquestions.
Certainly open-ended responses are not going anywhere and well they shouldn’t. The data that is obtained from them is incredibly valuable simply because the respondent is far more able to give a free and candid response without the constraints of a closed-ended question type. For the researcher, that data is far more problematic precisely because of its freedom: it is not easily quantified into a precise number, it is more prone to garbage data, and responses must be processed in a secondary step, as opposed to a closed-ended question type. The problem with closed-ended ranking-style questions, however, is that they may try to too-precisely quantify the sentiment of a respondent into the way that the researcher wants them to think. A respondent asked a question like “On a 1 to 5 scale, where 1 is not at all tasty and 5 is completely delicious, how would you rate our chocolate?” is forced to give a response that fits into the expected scale of the researcher. For the researcher, the obtained data is much easier to deal with simply because of its constrained nature, but it is an easy and dangerous trap to believe that a respondent can accurately convert his feelings into the precise input type demanded by a closed-ended question. It’s for this reason that greater use of open-ended responses should be embraced, but to identify their information and sentiment in realtime is a great necessity.
Another fantastic benefit to automated sentiment analysis is the ability to examine sentiment at the macro level (the general social media trends), the micro level (your specific survey), or both. For example, if you ran a survey with a nationally representative sample to find out how well your company’s chocolate is being received, you may wish to compare that with the sentiment data you obtain from social media about your company’s chocolate. If the results are greatly incongruent, it may be a good idea to figure out why.
Whether looking at the specific data of your survey or the data you see in the social media universe, with the right sentiment analysis tools, it’s possible to not only see and identify the trends and influencers, but actually use the information to produce actionable results. At the micro level, combining sentiment analysis with the quantifiable information in the rest of your survey produces a fuller spectrum of data in your final deliverable. When this micro level data is used in combination with macro social media data, a “bigger picture” is shown in your data.
Researchers need to begin filling in the “bigger picture” in order to stay relevant: combining the micro view with the macro view paints a much broader portrait than either one by itself. Of course, this is not relevant in every type of survey. Very specific topics or niche discussions are not necessarily well represented in the social media universe. However, more efficient sentiment analysis at the survey level as well as the public internet on a set of topics of keywords are absolutely critical to remain competitive and offer as broad a data deliverable as possible. Researchers should allow as much flexibility as possible to respondents, but they must also be confident in the efficiency and accuracy of the data that they collect. The industry as a whole should be embracing the new technological capabilities that can be used to complement or upgrade their existing data collection methods, but as with all new things, it is better to adapt step-by-step and verify the new additions to be confident in the final data.