Five basic questions to ask before leaping into low cost social media monitoring

I just finished testing two low cost (<$50.00/mo) social media monitoring tools. They were both easy to use with clean user interfaces. Both had some nice features, especially around reaching back out to people making comments in social media. However, running these two services side by side brought home some issues with these kinds of offerings – specifically in the area of coverage, sentiment analysis, and analytics. Note that I am not naming names because I believe the issues I ran into are not unique to these specific tools, but to the low cost social media monitoring market, in general. Some of these issues will also apply to higher priced offerings!

What I did:

I ran an analysis using the term “advanced analytics” including the term itself as well as companies (as additional topics) in the space. I made sure to be as specific and clear as I could be, since I knew topics were keyword based. I let the services run side by side for several weeks, interacting with the tools on a regular basis.

What I noticed:

1. Topic specification. Tools will vary in how the end user can input what he or she is looking for. Some will let you try to refine your keyword search (and these are keyword based, they won’t let you discover topics per se), other won’t. Some will only allow search across all languages, others will allow the user to specify the language. Find out how you can be sure that you are getting what you are searching for. For example, does the tool allow you be very specific about words that should and should not be included (i.e. SAS is the name of an analytics company and also an airline)? Since these low cost tools often don’t provide a way to build a thesaurus, you need to be careful.
2. Completeness of Coverage. The coverage varied tremendously between the services. Nor was the coverage the same for the same day for the same company name that I was tracking (and I was pretty specific, although see #1 above). In fact it seemed to vary by at least an order of magnitude. I even compared this manually in twitter streams. When I asked, I was told by one company that if they weren’t picking up everything, it must be a bug and it should be reported (!?). The other company told me all of my content came to me in one big fire hose, because there had been a problem with it, before (!?). In both cases, there still seemed to be a problem with the completeness of content. The amount of content just didn’t add up between the two services. In fact, one company told me that since I was on a trial, I wasn’t getting all of the content – yet even with the firehose effect, the numbers didn’t make sense. Oh. And don’t forget to ask if the service can pull in message boards, and which message boards (i.e. public vs. private). For an analysis, all of these content issues might mean that completeness of buzz might be misrepresented which can lead to problems.
3. Duplicate Handling. What about the amount of buzz? I thought that part of my content counting discrepancy might be due to how the company was dealing with duplicates. So beware. Some companies count duplicates (such as retweets) as buzz and some do not. However, be sure to ask when duplicate content is considered duplicate and when it is not. One company told me that retweets are not counted in buzz, but are included in the tag cloud (!?).
4. Sentiment analysis. The reality is that most of the low cost tools are not that good at analyzing sentiment. Even though the company will tell you they are 75% accurate the reality is more like 50%. Case in point: on one offering, one job listing was rated positive and another job posting listed as negative. In looking at the two postings, it wasn’t clear why (shouldn’t a job post be neutral anyway) Note, however, that many of these tools provide a means to change the sentiment from +/-/neutral (if they don’t then don’t buy it). So, if sentiment is a big deal to you then be prepared to wade through the content and change sentiment, if need be. Also ask how the company does sentiment analysis and find out at what level it does the analysis (article, sentence, phrase)
5. Analysis. Be prepared to ask a lot of questions about how the company is doing its analysis. For example, sometimes I could not map the total buzz to other analytics numbers (was it duplicates handling or something else). Additionally, some social media monitoring tools will break down buzz by gender. How is this determined? Some companies determine gender based on a name algorithm, while others use profile information from facebook or twitter (obviously not a complete view of buzz by gender, since not all information sources are twitter and facebook like). Additionally, some of the tools will only show a percentage (a no-no in this case), while others may show the number and the percent. Ditto with geolocation information. If the data is incomplete (and isn’t representative of the whole) then there could be a problem with using it for analytical purposes.

What this means

Certainly, the lure of low cost social media platforms is strong, especially for small and medium businesses. However, I would caution people to do their homework and ask the right questions, before purchasing even a low cost product. I would also suggest testing a few products (running them side by side for the same time period, even if you have to pay for it for a month or so) to compare the tools in terms of coverage, sentiment, and analysis.

The reality is that you can end up with an analysis that is completely wrong if you don’t ask the right questions of the service provider. The amount of buzz might not be what you think it is, how your company compares to another company might be wrong based on how you specified the company name, the sentiment might be entirely wrong if you don’t check it, and the analysis may be misleading unless you understand how it was put together.

Five Analytics Predictions for 2011

In 2011 analytics will take center stage as a key trend because companies are at a tipping point with the volume of data they have and their urgent need to do something about it. So, with 2010 now past and 2011 to look forward to, I wanted to take the opportunity to submit my predictions (no pun intended) regarding the analytics and advanced analytics market.

Advanced Analytics gains more steam. Advanced Analytics was hot last year and will remain so in 2011. Growth will come from at least three different sources. First, advanced analytics will increase its footprint in large enterprises. A number of predictive and advanced analytics vendors tried to make their tools easier to use in 2009-2010. In 2011 expect new users in companies already deploying the technology to come on board. Second, more companies will begin to purchase the technology because they see it as a way to increase top line revenue while gaining deeper insights about their customers. Finally, small and mid sized companies will get into the act, looking for lower cost and user -friendly tools.
Social Media Monitoring Shake Out. The social media monitoring and analysis market is one crowded and confused space, with close to 200 vendors competing across no cost, low cost, and enterprise-cost solution classes. Expect 2011 to be a year of folding and consolidation with at least a third of these companies tanking. Before this happens, expect new entrants to the market for low cost social media monitoring platforms and everyone screaming for attention.
Discovery Rules. Text Analytics will become a main stream technology as more companies begin to finally understand the difference between simply searching information and actually discovering insight. Part of this will be due to the impact of social media monitoring services that utilize text analytics to discover, rather than simply search social media to find topics and patterns in unstructured data. However, innovative companies will continue to build text analytics solutions to do more than just analyze social media.
Sentiment Analysis is Supplanted by other Measures. Building on prediction #3, by the end of 2011 sentiment analysis won’t be the be all and end all of social media monitoring. Yes, it is important, but the reality is that most low cost social media monitoring vendors don’t do it well. They may tell you that they get 75-80% accuracy, but it ain’t so. In fact, it is probably more like 30-40%. After many users have gotten burned by not questioning sentiment scores, they will begin to look for other meaningful measures.
Data in the cloud continues to expand as well as BI SaaS. Expect there to still be a lot of discussion around data in the cloud. However, business analytics vendors will continue to launch SaaS BI solutions and companies will continue to buy the solutions, especially small and mid sized companies that find the SaaS model a good alternative to some pricey enterprise solutions. Expect to see at least ten more vendors enter the market.

On-premise becomes a new word. This last prediction is not really related to analytics (hence the 5 rather than 6 predictions), but I couldn’t resist. People will continue to use the term, “on-premise”, rather than “on-premises” when referring to cloud computing even though it is incorrect. This will continue to drive many people crazy since premise means “a proposition supporting or helping to support a conclusion” (dictionary.com) rather than a singular form of premises. Those of us in the know will finally give up correcting everyone else.

Follow

Get every new post delivered to your Inbox.

Join 1,189 other followers