Over the past several weeks, I’ve been briefed by a number of text analytics vendors and companies in partnership with text analytics vendors about syndicated services that make use of text analytics. Of course, syndicated services such as brand monitoring and news services that make use this technology to some degree have been around for a while. But, how about some of the newer services?

 illumin8

An interesting example of this is illumin8, which is being offered by Elsevier, in partnership with Netbase. The service is targeted at R&D knowledge workers looking to solve technical and business problems. According to Elsevier, knowledge workers spend more time per week trying to discover relevant content relating to a particular problem area than analyzing that information (5.5 hours/week accessing vs. 4.7 hours/week analyzing). These workers are usually using a google-like search engine. I think everyone can agree that the google-like search engine is not ideal for research purposes, so I won’t belabor the point here. In the case of the R&D knowledge worker, often one goal is to gather information relating to a particular problem, finding products that solve that problem, as well as understanding the approach used to solve the problem.

Elsevier has aggregated 5 billion business sources, 3 million full text articles, 33 million scientific records, and 21 million patents as the source of information for this service. Using the Netbase semantic index, Elsevier crawls through the information and extracts solutions that solve a problem and the approaches used to address a specific issue. In this way, R&D can help answer the following questions:

• Solutions that exist to solve a problem
• New applications and processes that might exist to help solve a problem
• Information about what competitors are doing in the particular problem space
• What the experts are saying about a particular problem area

Here is a screen shot of what an end-user might see using this service. In this example, the user is interested in solving the problem of fuel efficiency in boats. He or she wants to see what products and approaches are out on the market to address this problem and what companies are providing these solutions. The user enters the topic (boats) and the benefit (fuel efficiency) in the search box and gets back information that is organized in a logical way. In this example, you can see that query returns information about products that address the problem as well as the companies that make the products, organizations that deal with energy, as well approaches to solving the problem (drag, stroke, etc). These are ranked. Users can then drill down on any of these areas to get snippets (and full text) associated with areas that he/she is interested in analyzing.

During the demo, I asked to see what would happen if we input “text analytics” as the problem space in the search box. I was actually impressed that what was returned was a good set of information about the players, organizations dealing with text analytics and other information about it. The service is not inexpensive, but it does cull a lot of information.

Syndicated Services

I believe that the number of syndicated services using text analytics will continue to grow. We’re certainly seeing action in the brand monitoring space on this front. Vendors are also getting into the act. Expert System, for example, has its own service that is targeted at the auto industry. I believe that other vendors may get into the act if they determine that the financial benefits of offering syndicated services (as opposed to SaaS offerings) makes sense.

I add my thoughts to others who have blogged on the Summit. See the Seth Grimes posting, for all of the other comments.

• I too agree that it was great to see a large number of end users at the Summit, this year. I was especially interested in the fact that a number of them were in the investigation phase. And, what were many investigating? You got it - Voice of the Customer and the closely intertwined area of sentiment analysis.
• VoC was a major theme. In fact, I was overwhelmed by the number of talks in the area. I thought that the presentation about what Gaylord Hotels is doing with text analytics and VoC was extremely interesting. Tony Bodoh took the audience through a journey beginning with the fact that before text analytics it would take the company weeks to even see customer comments. He said that the working pilot the company did in conjunction with Clarabridge took only 10 weeks. He reported benefits in process improvements, value-oriented marketing, and facility improvements. He even told us about reticular activating systems! Go look that one up.
• I also met a number of people from new start-ups in the sentiment space, each taking a slightly different angle. I wonder if sentiment analysis will become a confusing space in the near future.
• There was some discussion at the Summit about text analytics and Web 2.0. I would have like to hear more about this, as text analytics will be important in Web 2.0
• And speaking of important, another interesting concept was brought up several times - the idea that text analytics will morph to become part of something bigger. I don’t want to say component, although others were.
• I was hoping to hear more about text analytics and content management. At one point, during the expert panel, I had the chance to ask the audience if anyone was deploying their text analytics in conjunction with their content management systems. A handful responded affirmatively. Unfortunately, I didn’t catch up with them. If you’re reading this and are deploying text analytics with your ECM system, I’d like to hear from you.

All in all though, I was impressed with the Summit.

I recently had the chance to get an update on what SPSS is up to in text analytics.  It was an interesting conversation for several reasons:

• First, it highlighted an important point about text analytics – which we know but is worth repeating – which is that the analysis of unstructured data can be more useful, in many scenarios, when accompanied by structured data.
• Second, it got me thinking more about social media/network analysis, which prompted the question on the recent “four questions about innovations in analysis” blog I recently posted.

A few words of background.   SPSS’s goal is help its customers analyze everything about data associated with people – behavior, attitudes, and so on to help an organization understand anyone it interacts with.  In fact, Olivier Jouve, VP of Corporate Development at SPSS was quite clear that SPSS is not a BI company.  Rather, SPSS software helps to enable what SPSS refers to as the “Predictive Enterprise”.   The Predictive Enterprise makes use of analytics (not simply reports) to help manage multiple dimensions across the enterprise including customer intimacy, product placement, and even operational issues such as fraud. 

SPSS offers a suite of text-mining products that is based on 25 years of research in the application of natural language processing (NLP) technologies. In 2002, SPSS bought LexiQuest™, a linguistics-based text-mining company, intending to combine LexiQuest’s extraction capabilities with SPSS’s data-mining capabilities in order to strengthen the company’s position in predictive analytics. All of SPSS’s text-analytics products now share this same core linguistic functionality.

It’s not just about text

While the market for text analytics has moved out of the early adopter stage, depending on what type of analysis you’re trying to accomplish, it often is not just about the text.

For example, consider the following churn scenario:  A telecommunications company is concerned about churn.  The company realizes that it has a wealth of information at its disposal to help predict churn.  On the structured data side it has collected demographic information, usage information, trouble ticket, and product information about each of its customers.  On the unstructured side, it also has collected call center notes, emails, and customer satisfaction surveys.  The company decides to invest in text analytics software that can sift through its call center notes, emails, and survey notes. At the end of the exercise, the company has some great insight into customer complaints that it can certainly act on.  However, it has not exactly gotten the information it might need to solve the churn problem.  In order to do this, it is probably more useful to marry the unstructured information from the call centers and surveys and emails to an actual customer and all of the structured information about that customer.  This way, using some predictive modeling the company can train its system to zero in on those customers that are likely to drop its service and make the right decisions to help retain them.  

According to SPSS many of its customers have seen upwards of a 50% reduction in churn by combining data mining with text mining.

Social media is becoming an important source of information for companies

What about other forms of media such as blogs, message threads, etc.?  SPSS is also moving into social network/media analysis because as Olivier said, “The number of people participating in Web 2.0 activities is growing rapidly across all age groups, and businesses are using the direct influence they have traditionally had over customers’ decisions about their products.  Peer to Peer networks are now a trusted source of insight and information.”   This is quite true.  Our recent Hurwitz & Associates survey confirmed that companies do plan to make use of the information found in various kinds of social networks, even if they don’t think they are making use of text analytics.  One interesting point on this front is that blogs, message boards, etc. do provide a great source of information of customer sentiment, opinions, etc. The challenge will be mapping this kind of information back to the other information that a company keeps about its customers, and making sense of the behaviors.  I’ll look forward to hearing more about what SPSS is doing to help solve this problem.

Several weeks ago, Hurwitz & Associates deployed a short survey entitled, “Four questions about innovations in analysis”.  Well, the results and they are quite interesting!

THE SURVEY

First, a few words about the survey itself and who responded to the survey.

1. We wanted to make the survey short and sweet.  We were interested in what  kinds of analytical technology companies thought were important and specifically how companies were using text analytics to analyze unstructured information.  Finally, since there has been a lot of buzz about analyzing social media we asked about this, as well
2. Let me say up front that given the nature of our list, I would categorize most of the respondents to the survey as fairly technology savvy.  In all,  61 people responded to the survey, 32% of these respondents were from high technology companies.  The verticals included professional services, followed by manufacturing, financial/insurance, healthcare and pharmaceutical. There were also some responses from governmental agencies, telecommunications and energy companies.  So, while the results are unscientific in terms of a random sample across all companies, they probably do reflect the intentions of potential early adopters, although not in a statistically significant manner.
3. In analyzing the results, I first looked at the overall picture and then examined individual verticals as well as filtered the results by other attributes (such as those using text analytics vs. those not using the technology) to get a feel for what these companies were thinking about and whether one group was different from another.  These subgroups are of course, quite small and the results should be viewed accordingly.

THE RESULTS

 We first asked all of the respondents to rate a number of technologies in terms of importance to their companies.  Figure 1 shows the results.  Overall, most of these technologies were at least somewhat important to this technology savvy group, with query and reporting leading the pack.  This isn’t surprising.  Interestingly, OLAP data cubes appeared to be the least important analytical technology – at least with this group of respondents.  Other technologies, such as performance management, predictive modeling, and visualization ranked fairly high, as well.  Again not surprisingly, text analytics ranked lower than some of the other technologies probably since it is just moving out of the early adopter stage.  Some of the respondents, from smaller firms, had no idea what any of these technologies were.  And, in terms of text analytics, one company commented, ” yeekes, this must be big time company kind of stuff. Way up in the clouds here, come down to earth.” They, no doubt, are still using Excel and Access for their analytical needs.  Other smaller companies were very interested in “non-cube” technologies such as some of the visualization products on the market today.

What does this mean?  I would speculate that it means that companies that have found value in perhaps one or more analytical technologies are more apt to try out others.  Success breeds success.  We’ve seen this trend in the text analytics space.  When a company deploys the technology and finds success, other parts of the organization are quick to jump on board.  These technology savvy companies no doubt are gaining benefits from these technologies and are willing to invest in them because they provide significant value to the company.  Those that either haven’t invested in the technologies, or are caught up in one problem –say getting their data into a warehouse- haven’t had the opportunity to experience the benefits of these other technologies, for budgetary or other reasons and so they are not that important to their companies right now.

Text Analytics

Question 2 asked which statement best describes your company’s use of text analytics?  The results here indicate a surprisingly high percent – 32.7%- actually using the technology today.  Again, the reason behind this is no doubt due to the fact that the companies polled are probably very technology savvy.  I believe this is significantly higher than the overall adoption rate of text analytics, which Hurwitz & Associates estimated at 10-15% of large enterprises in 2007. On the other hand, almost 45% either didn’t know what the technology was or didn’t plan to use it in the next 12-18 months.  It is important to note that these results were industry independent.

We asked respondents that were using text analytics or planning to use it. what they were planning to use it for.  An important point is that the majority of respondents were planning to use the technology in more than one application area.  Not surprisingly, Voice of the Customer and Competitive Intelligence/Brand Intelligence led the space with over 60% of the respondents cited these two areas.  Next up on the application front was e-Discovery, an area that is starting to gain a lot of attention. 

 Let’s slice and dice this a bit….

 First up, those respondents deploying text analytics.  These respondents came from a range of industries.  In this group, companies are using text analytics for (in order of importance)  Competitive/Brand intelligence, Voice of the Customer, e-discovery, quality and early warning and fraud.  The respondents also cited other areas their companies are using text analytics for including spend analysis, analyzing company metrics, product lifecycle management, and mining business rules. Analysis of social media was very important to this group. 

Those who were planning to deploy text analytics in the next 12-18 months also viewed query and reporting as most important to their company.  Interestingly, the top application this group is considering is Voice of the Customer followed by competitive intelligence/brand image.   This tracks with the primary research study we performed last year, indicating that Voice of the Customer should continue to be a hot area for text analytics implementations for at least the next year or so. Analysis of social networks and media was of high interest to this group.

Even those who did not have plans for text analytics were still interested in social networks – indicating that companies are starting to think about using unstructured information to gain insight about customers.  These companies may be considering brand monitoring companies to serve its needs, or perhaps haven’t even gotten that far in the thought process, yet.

The Upshot?

• Broadly speaking, companies that believe that innovative analysis technologies such as predictive modeling, text analytics, and even performance management are very important, generally rated all analytical technologies as having a higher value than those companies that did not believe they were very important. 
• Text analytics continues to grow in importance, at least with this group of respondents.  Companies deploying the technology today are doing so across several application areas.  Companies that plan to deploy the technology are also looking at several application areas.  Voice of the Customer will continue to be a hot area for the foreseeable future.  E-discovery is also making noise.
• Companies across the board expressed interest in making use of social media and social networks.

Several months ago, I wrote a piece about Voice of the Customer (VoC) and how text analytics companies are getting good traction from this application.  I recently got an update from Sid Banerjee, CEO of Clarabridge that further supports this observation.  Clarabridge, an independent text analytics vendor is garnering much success with what it is referring to as Customer Experience Intelligence (CEI).  This is really another term for a Voice of the Customer with an emphasis on analyzing the experience – from the customer’s point of view.

The Chief Customer Officer wants to know

It goes without saying that customer experience matters to companies. Companies want to know what their customers are thinking about their products and services.  New positions, such as the Chief Customer Officer have even been developed at some larger companies to address this issue.  The traditional method of manually reading through surveys, emails, etc to answer questions about customer experience is time consuming and not effective.  I remember reading through this kind of information myself to try to determine why customers were dropping a service and I’ll be the first to say that the process was not pretty.  Text analytics enables companies to sift through this massive amount of information much more efficiently and effectively and derive valuable insight from it.   Clarabridge is getting good traction from groups concerned with enterprise feedback management, CRM including call centers, tech support and the Chief Customer officer her/himself, to help analyze structured and unstructured information to understand customer experience.  These types of deployments have helped fuel the company’s rapid growth.

Clarabridge Navigator

While text analytics technology helps companies gain valuable insight about their customers, companies deploying the technology have faced challenges, regardless of the vendor they have chosen to work with.  Our Hurwitz & Associates research last year highlighted a number of challenges including taxonomy and rule development.  End-users complained that taxonomy development was often a very frustrating experience.   End-users also stated that rule development was often difficult and time consuming.

Earlier this year, Clarabridge released Clarabridge Navigator.  Navigator is an interface for business users to help them build taxonomies and categories more effectively.   The interface enables end-users to iteratively explore categories that they build.  The example Sid used in our discussion utilized hotel related data.  Suppose I’ve created a category (in Navigator) called bed comfort.  I’ve keyed off of words such as comfort and bed and, using Navigator, I can see the kinds of text that gets returned off of these words.  Perhaps when I am exploring this I see that people are talking about other words such as queen and king.  I can then incorporate the concept of bed size into my categorization.  In this way the end-user can interactively and iteratively explore text data and develop a better categorization scheme.  Navigator provides the same functionality for sentiment.  The Clarabridge product comes with over ten thousand words that are sentiment related.  However, if the user wants to explore the data and change the defaults, it can be done via Navigator. 

If the categories are correct, this makes rule development more efficient.  Instead of running the data through the rules and hoping it makes sense, this gives end-users a more effective way to develop rules.  This doesn’t negate the need for people involvement in the taxonomy and rule development process, but it is one step in making the process easier – and enabling text analytics to become more mainstream.

When I first started to research the text analytics industry about three years ago, people didn’t understand why I was so interested in this nascent marketplace.  I had always anticipated how important it would be.  I remember working at Bell Laboratories, in the 1980s, analyzing structured data using advanced data mining techniques, trying to predict customer actions.  I knew then, that this structured data was only a piece of the puzzle and that unstructured data needed to be part of the mix.  Since that time technology and computing power have obviously come a long way and text analytics has become an increasingly mature market.

 Over the past several years, I’ve gotten to know many of the vendors in the space, talked to many customers deploying the technology, and in 2007 published a benchmark study entitled, “Text Analytics: The Road to Understanding Your Company’s Unstructured Data.” Three key findings from our research have been borne out in the market:

·        Voice of the Customer is a key application area.  Our research indicated that end-users placed a high priority on customer focused applications, with75% of respondents planning to deploy customer care related solutions.   Today, many of the vendors in this space have gained great traction in this area.

·        Text analytics moves beyond business intelligence.  While early adopters of the technology deployed text analytics together with data mining and BI, and this is still a very popular means of deploying the technology, our study indicated that end-users were interested in deploying text analytics in conjunction with both BI and content management.  Since then, I’ve seen more evidence in the market for this kind of deployment.

·        The market will continue to consolidate. At the time the report was published, ClearForest and Inxight had recently been acquired. I predicted that additional market consolidation would take place; and it has.

Since the study was published in August, 2007, and due to the factors mentioned above, the market has reached the next level of maturity.  For these reasons, I believe it is time to provide another benchmark.  I am looking for a few sponsors who would like to partner with our firm in bringing valuable market research to the end-user community.  If you are interested, please contact Carol Caliendo at carol.caliendo@hurwitz.com or 617-454-1030.

I recently had an interesting conversation about classification and search with James Zubok, CFO of Brainware, Inc. Brainware is a Virginia based company that was once part of SER Systems AG, a former German ECM company.  Brainware provides products that help companies extract information from unstructured and semi-structured documents, such as invoices, order forms, contracts, etc. without using templates.  The company also offers some interesting classification and search technology and this is what our conversation focused on.

We discussed two different, but interrelated technologies that Brainware has developed; one a search engine based on n-grams and another,  a classification engine that uses neural networks. Brainware offers both enterprise and desktop editions of each.  I received a demo of the desktop version of the products and now have both running on my laptop.

A Search example

On the desktop search side, the product, called Globalbrain Personal Edition, differs from many other search products on the market in that it does not make use of keyword search.  Rather, it searches are natural language based, using a patented n-gram approach.  When indexing a word, the word is parsed into three parts and then a vector is created.  For example, the word sample would be parsed as sam, amp, mpl, etc.  According to Brainware, this three-letter snippet approach makes the search engine language independent.   The capability provided by Brainware lets users search, not simply on key words, but on whole paragraphs. For example, I have many documents (in various formats) on my desktop that deal with all of the companies I speak with.  Say, I want to find some documents relating to specific challenges companies faced in deploying their text analytics solutions.  Rather than simply inputting “text analytics” and “challenges”, I can type in a phrase or even a paragraph with the wording I’m looking for.  This returns a much more targeted set of documents. 

A Classification example

On the desktop classification front, the product is very easy to use.  I simply loaded the software which provided me a user interface where I could develop classes and then train my system to automatically classify documents based on a few training examples. As I mentioned, I have many documents on my desktop that deal with various technology areas and I might want to classify them in an intelligent manner for some research I’m planning.  So, I created several classes: text analytics, visualization, and MDM. I simply created these classes and then dragged documents that I felt fell into each category onto those classes.  I trained the system on these examples. 

Brainware provides a visual interface that lets me view how “good” the learned set is via a series of points in three-dimensional space.  The closer together the points (from the same class) are on the plot, the better the classification will be.  Also, the more separate the various class points are, the better the classification.  In my classification test, the visualization and the MDM documents were tightly clustered, while the text analytics information was not.  In any event, I then ran the classifier over the rest of my documents (supplying a few parameters) and the system automatically classified what it could.  It also gave me a list of documents that it couldn’t classify, but suggested the appropriate categories. I could then just drag those documents into the appropriate categories and run the classifier again. I should add that it did a good job of suggesting the right class for the documents it put in the unclassified category.

Brainware on an enterprise level

The enterprise edition of the product combines the search and classification capabilities and lets users search and classify over 400 different document types. 

Now, Brainware isn’t planning to compete with Google, Yahoo!, Fast, etc.  Rather, the company sees its search as a complement to these inverted index approaches.  The idea would be to embed its search into other applications that deal with archiving, document management, or e-discovery, to name a few.  The classification piece could also be embedded into the appropriate applications.  I asked if the company was in discussions with content management providers and service providers that store emails and documents.  It would seem to me that this software would be a natural complement to some of these systems. My understanding is that the company is looking for partnerships in the area.  Brainware currently has a partnership with Attensity, a text analytics provider, to help classify and search documents as part of the text analytics process.

I’m interested to see what will develop with this company.

Monday, SAS announced that it had purchased Teragram, a privately held natural language processing (NLP) company, for an undisclosed sum.  Teragram is now a SAS company meaning that the Teragram brand will be maintained.  Its solutions and OEM business will be retained. 

A good move for SAS

This acquisition is a good move for SAS for a number of reasons.  First, SAS had partnered with Inxight to supply text analytics software components for its text mining solution and, of course, Inxight has been acquired by Business Objects (and subsequently SAP).  It was just a matter of time before SAS would have to replace these capabilities and Teragram is a logical choice because of its NLP technology.   More importantly, in my discussion with SAS and Teragram about the acquisition, it was clear that the purchase is more than just a move to replace technology components.  The purchase is actually quite strategic in nature.  Teragram technology can be used to enhance rather than simply replace existing capabilities.  

Structured and Unstructured Data are Not Separate Domains

 SAS’s strategy is to use both structured and unstructured data in analysis and to integrate it for descriptive and predictive modeling.  The company’s aim is to provide users with a seamless deployment of predictive model results and improve the consistency and accuracy of enterprise intelligence. 

The folks at SAS believe that structured and unstructured data have typically been viewed as two separate silos that can be joined together and then analyzed. Traditionally, unstructured data is the realm content management systems; structured data is the realm of BI.  SAS believes that both of these data sources should be brought together earlier in the analysis process and utilized as a joint asset.  Unstructured information can be categorized and indexed and even extracted and integrated more intelligently.  This approach makes a lot of sense to me and Teragram has technology to help make this happen.  

While a primary focus of the acquisition is to more seamlessly integrate structured and unstructured data into BI, SAS also mentioned a few other exciting concepts during our discussion.

• SAS hopes to use Teragram’s NLP capabilities to make BI more pervasive.  For example, by combining “SAS business intelligence, data integration and advanced analytics with Teragram’s NLP technologies to deliver answers to search queries in seconds”.  This appears to be an extension of the Teragram Direct Answers solution.
• Mobile BI
• Real time alerting

 And we shouldn’t forget that Teragram will also provide SAS with a much needed search capability.  I’m definitely looking forward to hearing more about integration and roll-out plans in the near future.

Data makes the world go ‘round and a lot of this critical data resides in databases.  This is why it is important to monitor database activity in order to prevent attacks and invasions. 

I recently had an interesting discussion with Rani Osnat, VP of Marketing at Sentrigo Software about its database monitoring product called Hedgehog and its recently announced 2.0 release.  For those of you who aren’t familiar with Sentrigo, it is an Israeli/US based company, founded in 2006, that focuses on real time database activity monitoring and intrusion prevention.  The company released its first product, Hedgehog 1.0 in 2007.