With all of the research I’ve been doing for our latest book:  Cloud Computing for Dummies, I’ve noticed something very disturbing.  Maybe it’s because I come from a telecommunications background, that this bothers me so much – but has anyone else noticed that people are misusing the word premise when describing aspects of the cloud?  I keep reading articles and blogs where an author refers to an “on premise” solution.  The proper term is premises as in – on your premises (see below).

From Dictionary.com

Premise:  a proposition supporting or helping to support a conclusion.

Premises:  a tract of land including its buildings.

 Even vendors in the space are making this mistake.  It’s appalling.  I could list dozens of examples of this error.  Has the definition of the word changed and I’m missing something?  Or, has the word been used incorrectly so many times that it doesn’t matter anymore?

I was at the EMC writer’s conference  this past Friday, speaking on Text Analytics and ECM.  The idea behind the conference is very cool.  EMC brings together writers and bloggers, from all over the world, to discuss topics relevant to content management.  All of the sessions were great.  We discussed Cloud, Web 2.0, Sharepoint, Text Analytics, and e-Discovery. 

 I want to focus here on the e-Discovery discussion, since e-Discovery has been showing up on my top text analytics applications list for several years.  There are a growing number of vendors looking to address this problem (although not all of them may be making use of text analytics yet) including large companies like EMC, IBM, Digital Iron Mountain, Microsoft and smaller providers such as Zylab.

 Ralph Losey gave the presentation. He is a defense lawyer, by training, but over the years has focused on e-Discovery.  Losey has written a number of books on the topic and he writes a blog called e-Discovery Team.  An interesting fellow!

 His point was that “The failure of American business to adopt ECM is destroying the American system of justice.”  Why?  His argument went something like this:

• You can’t find the truth if you can’t find the evidence.  As the amount of digital data explodes, it is harder to find the information companies need to defend themselves.  This is because the events surrounding the case might have occurred a year or more in the past, and the data is buried in files or email.  I don’t think anyone will argue with this fact. 
• According to Losey, most trial lawyers are luddites, implying that they don’t get technology.  Lawyers aren’t trained this way so they are not going to push for ECM systems, since they might not even know what they are.  And corporate America is putting off decisions to purchase ECM systems that could actually help organize some of the content and make it more findable.
• Meanwhile, the cost of litigation is skyrocketing.  Since it is so expensive, many companies don’t go to court and they look to private arbitration.  Why spend $2M in e-Discovery when you can settle for $3M?  Losey pointed to one example, in the Fannie Mae securities litigation (2009), where it cost $6M (or 9% of the annual budget of the Office of Federal Housing Enterprise Oversight) to comply with ONE subpoena. This involved about 660,000 emails. 
• According to Losey, it costs about $5 to process one computer file for e-Discovery.  This is because the file needs to be reviewed for relevance, privilege, and confidentiality. 

 Can the American justice system be saved? 

 So, can e-Discovery tools be used to help save the justice system as we know it?  Here are a few points to ponder:

• Losey seems to believe that the e-Discovery process may be hard to automate since it requires a skilled eye to determine whether an email (or any file for that matter) is admissible in court. 
• I’m not even sure how much corporate email is actually being stored in content management systems – even when companies have content management systems. It’s a massive amount of data.
•  And, then there is the issue of how much email companies will want to save to begin with.  Some will store it all because they want a record.  Others seem to be moving away from email altogether.  For example, one person in the group told us that his Bank of America financial advisor can no longer communicate with him via email!  This opens up a whole different can of worms, which is not worth going into here. 
• Then there is the issue of changing vocabularies between different departments in companies, people not using certain phrases once they get media attention, etc. etc.

Before jumping to any conclusions let’s look at what vendors can do.  According to EMC, the email overload problem can be addressed.  The first thing to do is to de-duplicate emails that could be stored in a content management system.  Think about it.  You get an email and 20 people are copied on it. Or, you forward someone an email and they don’t necessarily delete it. These emails would pile up.  De-duplicating emails would go a long way in reducing the amount of content in the ECM.  Then there is the matter of classifying these emails.  That could be done.  Some of this classification would be straight-forward.  And, the system might be able to be trained to look for those emails that might be privileged, and classify these accordingly, but this would no doubt still require human intervention, to help with the process.  Of course, terminology will change, as well and people will have to stay on top of this. 

The upshot is that there are certainly hurdles to overcome to put advanced classification and text analytics in place to help in e-Discovery.  However, as the amount of digital information keeps piling up, something has to be done.  In this case, the value certainly would seem to outweigh the cost of business as usual.

I just got back from the Text Analytics Summit and it was a very good conference.  I’ve been attending the Summit for the last three years and it has gotten better every year.  This year, it seemed like there were a lot more end users and the conference had more of a business oriented approach than in previous years.  Don’t get me wrong- there were still technical discussions, but I liked the balance.

A major theme this year, as in previous years, was on Voice of the Customer applications.  That is to be expected, in some ways, because it is still a hot application area and most of the vendors at the conference (including Attensity, Clarabridge, Lexalytics, SAS, and SPSS) focus on it in one form or another.  This year, there was a lot of discussion about using social media for text analytics and VoC  kinds of applications.  Social media meaning blogs, twitter, and even social networks.  The issue of sentiment analysis was discussed at length since it is a hard problem.  Sarcasm, irony, the element of surprise, and dealing with sentiment at the feature level were all discussed.  I was glad to hear it, because it is very important.  SAS also made an announcement about some of its new features around sentiment analysis.  I’ll blog about that in a few days.

Although there was a heavy focus on the VoC type applications, we did hear from Ernst & Young on fraud applications.  This was interesting because it showed how human expertise, in terms of understanding certain phrases that might appear in fraud, might be used to help automate fraud detection.  Biogen Inc also presented on its use of text analytics in life sciences and biomedical research.  We also heard what Monster and Facebook are doing with text analytics, which was quite interesting.  I would have liked to hear more about what is happening with text analytics in media and publishing and e-Discovery.  It would have also been useful to hear more about how text analytics is being incorporated into a broader range of applications.  I’m seeing (and Sue Feldman, from IDC, noted this too) a large number of services springing up that use text analytics.  This spans everything from new product innovation to providing real time insight to traders.  As these services, along with the SaaS model continue to explode, it would be useful to hear more about them next year.

Other observations

Here are some other observations on topics that I found interesting.

• Bringing people into the equation.  While text analytics is very useful technology, it needs people to make it work.  The technology itself is not Nirvana.  In fact, it can most useful when a person works together with the technology to make it zing.  While people who use the technology obviously know this (there is work that has to be done by people to make text analytics work),  I think that  people beginning the process need to be aware of this too, for many reasons.  Not only are people necessary to make the technology work, the cultural component is also critical, as it is in the adoption of any new technology.  Having said this, there was discussion on the end user panel about how companies were making use of the SaaS model (or at least services), since it wasn’t working out for IT (not quite sure why – either they didn’t have the time or didn’t have the skills).
• Managing expectations. This came up on some of the panels and in a few talks.  There were two interesting comments worth noting.  First, Chris Jones, from Intuit said that some people believe that text analytics will tell you what to do, so expectations need to be set properly.  In other words, people need to understand that text analytics will uncover issues and even the root cause of the issues, but it is up to a company to figure out what to do with that information.   Second, there was an interesting discussion around the notion of the 85% accuracy that text analytics might provide.  The end user panel was quite lively on this topic. I was especially taken with comments from Chris Bowmann, a former school superintendent of the Lafourche Parish School Board, and how he had used text analytics to try to help keep kids in school. He used the technology to cull through disciplinary records to see what patterns were emerging.  Very interesting.  Yes, as he pointed out, text analytics may not be 100% accurate, but think of what 85% can get you!
• Search needs to incorporate more text analytics. There were two good search talks on the agenda:  Usama Fayyad, CEO of Open Insights who spoke about text analytics and web advertising as well as how text analytics might be used to help search “get things done” (i.e. like book a trip).  The other speaker on the topic was Daniel Tunkelang from Endeca, who talked about text analytics and exploratory search. There were a number of comments from people in the audience as well about services like Wolfram Apha.
• Content Management.  I was happy to see more about enterprise content management this year and see more people in the audience who were interested in it.  There was even a talk about it from Lou Jordano from EMC. 

 I think anyone who attended the conference would agree that text analytics has definitely hit the main stream.

 Several weeks ago, in preparation for the Text Analytics Summit, I deployed a short survey about the state of text analytics.  I supplemented the end-user survey with vendor interviews.   Here are some of the end-user findings. 

First, a few words about the survey itself and who responded to the survey.

• I wanted to make the survey short and sweet.  I was interested in company’s plans for text analytics and whether the economy was affecting these plans. 

• Let me say up front that given the topic of the survey and our list, I would categorize most of the respondents as fairly analytical and technology savvy.  Approximately 50 companies responded to the survey – split evenly between those companies that were deploying the technology and those that were not (note that this is a self selecting sample and does not imply that 50% of companies are currently using text analytics).  The respondents represent a good mix across a number of verticals including computing, telecommunications, education, pharmaceuticals, financial services, government, and CPG.  There were also a few market researchers in the mix.  Likewise, there was a mix of companies of various sizes. 
• Here’s my caveat:  I would not view the respondents as a scientific sample and I would view these results as qualitative.  That said, many of the results paralleled results from previous surveys.  So, while the results are unscientific, in terms of a random sample and size, I believe they probably do reflect what many companies are doing in this space.

Results:

Kinds of applications, implementation schemes

I asked those respondents that were deploying text analytics, what kinds of applications they were using it for.  The results were not surprising.  The top three responses:  Voice of the Customer (VoC), Competitive Intelligence, and eDiscovery, were also in the top three the last time I asked the question. Additionally, many of the respondents were deploying more than one type of application (i.e. VoC and quality analysis).  This was a pattern that also emerged in a study I did on text analytics back in 2007. Once a company gains value from one implementation, it then sees the wider value of the technology (and realizes that it has a vast amount of information that needs to be analyzed).

 I asked those companies that were planning to deploy the technology, the top applications being considered.  In this case, VoC and Competitive Intelligence were again in the top two.  Brand Management and Product R&D were tied for third.  This is not surprising.  Companies are quite concerned with customer feedback and any issues that impact customer retention.  Companies want to understand what competitors are up to and how their brand is being perceived in the market.  Likewise, they are also trying to get smarter about how they develop products and services to be more cost effective and more market focused.

How Text Analytics is being deployed

I also wanted to find out how companies were deploying the technology. In particular, we’ve heard a lot this past year about organizations utilizing text analytics in a Software as a Service (SaaS) model.  This model has become particularly attractive in the Voice of the Customer/Competitive Intelligence/Brand Management area for several reasons.  For one thing, this kind of analysis might involve some sort of external information source such as news feeds and blog postings.  Even product R&D would draw from external sources such as trade journals, news about competitive products, and patent files.  Additionally, the SaaS model generally has a different price point that enterprise solutions.

In fact, SaaS was the model of choice for implementing the technology.  The SasS model does offer the flexibility and price point that many companies are looking for, especially in some of the above-mentioned areas.  However, that is not to say that companies are not deploying text analytics in other ways (note the values on the X axis).  Interestingly, companies are starting to deploy text analytics in conjunction with their content management systems.  I think we will see more of this as the technology continues to become more mainstream. 

Just as an FYI, all of the companies that had deployed text analytics stated that the implementations either met or exceeded their expectations.  And, close to 60% stated that text analytics had actually exceeded expectations. 

 What about those companies that aren’t deploying the technology?

Equally important to understanding the market are those companies that are not deploying text analytics.  I asked those companies if they had any plans to utilize the technology.  Eleven percent stated that plans had been put on hold due to funding constraints.  Twenty-eight percent stated that they had no plans to implement the technology.  Another 28% stated that they planned to implement the technology in the next year and 33% said they planned to implement it in the next few years. 

Reasons cited for not implementing the technology included not understanding enough about text analytics to implement it.  Other companies just never considered implementing it, or had other analytic projects on the radar.

What about the economy?

There have been numerous articles written about whether certain technologies are recession proof, with various BI related technology vendors stating/hoping/praying that their technology falls in to this category.  And certainly, companies do feel the need to gain insight about operational efficiency, their customers, the market, and the competition with, perhaps a greater urgency than in the past.  This has helped keep business analytics vendors moving forward in this economy.

The 11% number is relatively small.  However, I wonder what part of the 61% that said that they would be deploying it in the future, might actually fall into the hold category.  When I asked text analytics vendors (mostly private companies) whether the economy was impacting the sales cycle, they pretty much said the same thing.  Existing customers were not dropping projects (there is too much value there, as supported by this survey).  However, sales cycles are longer (companies are not necessarily rushing) and potential clients may be looking for creative financing and contracting options. 

I am participating in an analyst panel at the Text Analytics Summit in June.  I have more to say about the topic, as I am sure, do the other analysts who will be participating.

I recently had the opportunity to speak with Richard Turner, VP of Marketing, at Content Analyst.  Content Analyst was originally part of SAIC and spun out about 5 years ago.  The company provides content management, eDiscovery, and content workflow solutions to its clients – primarily as an OEM solutions.

The tool set is called CAAT (Content Analyst Analytic Technology).  It includes five core features:

• Concept search: uses concepts rather than key words to search through documents.
• Dynamic Clustering: classifies documents into clusters.
• Categorization: classifies documents into user-defined categories.
• Summarization: identifies conceptually definitive sentences in documents.
• Foreign Language: identifies the language in a document and can work across any language.

Let me just briefly touch upon concept search and dynamic clustering and categorization.  Concept search is interesting because when most people think search, they think key word search.  However, key words may not give you what you’re looking for.  For example, I might be interested in finding documents that deal with banks.  However, the document might not state the word bank explicitly.  Rather, words like finance, money, and so on might occur in the document.  So, if you don’t insert the right word into the search engine, you will not get back all relevant documents.  Concept search allows you to search on the concept (not keyword) “bank” so you get back documents related to the concept even if they don’t contain the exact word.  CAAT learns the word bank in a given set of documents from words like “money”, “exchange rate”, etc.  It also learns that the word bank (as in financial institution) is not the same as the bank on the side of a river becuase of other terms in the document (such as money, transfer, or ATM).

Dynamic clustering enables the user to organize documents into categories based on content called clusters.  You can also categorize documents by using examples that fall into a certain cluster and then train the system to recognize similar documents that could fall into the same category.  You literally tag the document as belonging to a certain category and then give the system examples of other documents that are similar to this to train on.  In eDiscovery applications, this can help dramatically cut down the amount of time needed to find the right documents.  In the past, this was done manually, which obviously could be very time intensive. 

How do they do it?

The company uses a technique called Latent Semantic Indexing (LSI), along with other patented technology, to help it accomplish all of this.  Here is a good link that explains LSI.  The upshot is that LSI uses a vector representation of the information found in the documents to analyze the term space in a document.  Essentially, it removes the grammar, then counts and weights (e.g. how often a word appears on a page or in a document, etc.) the occurrence of the terms in the document.  It does this across all of the documents, and actually collapses the matrix using a technique patented at Bell Labs.  The more negative a term, the greater its distance from a page.  Since the approach is mathematical, there is no need to put together a dictionary or thesauri.  And, it’s this mathematical approach that makes the system language independent.

Some people have argued this technique can’t scale because the matrix would be too large and it would be hard to keep this in-memory.  However, when I asked the folks at Content Analyst about this they told me that they have been working on the problem and that CAAT contains a number of features to optimize memory and throughput.  The company regularly works with ligitation clients who might get 1-2 TB of information from opposing counsel and they are using CAAT for clustering, categorization, and search.  The company also works with organizations that have created indexes of 45+ million (>8 TB) documents.  That’s a lot of data!

Conceptual Search and Classification and Content Management

Aside from eDiscovery, Content Analyst is also being used in application such as improving search in media and publishing and of course, government applications.  The company is also looking into other application areas.

Concept search is definitely a big step up from keyword search and is important for any kind of document that might be stored in a content management system.  Automatic classification and clustering would also be huge (as would summarization and foreigh language recognition).  This could move Content Analyst into other areas including analyzing claims, medical records, and customer feedback.  Content management vendors such as IBM and EMC are definitely moving in the direction of providing more intelligence in their content management products.  This makes perfect sense, since a lot of unstructured information is stored in these systems.  Hopefully, other content management vendors will catch up soon.

Attensity Corporation announced today that it is joining forces with two other text analytics/information management companies, Living e and Empolis, to form a new company called Attensity Group focused on creating business user applications on top of sophisticated data and semantic analytics technology.  According to Michelle de Haaff, CMO, the new combined company has software deployed in more than 500 installations worldwide and is in 250 of the Fortune 1000 companies.  

The combined company offerings include the following applications:

·        Voice of the Customer/Market Intelligence:  analysis of unstructured information from internal and external sources in order to understand customer sentiment, root cause and issues.  This helps to answer questions such as:  What are the top three complaints about my service?  How do I stack up against the competition?

·        E-service:  creates portals for online service content and includes guided search and escalation paths for both end customers and service agents.  This helps to answer questions such as: How do I change my password?  What workarounds have helped other customers? 

·        Automated Response Management: allows timely and accurate responses to incoming communications. This helps companies to answer inbound customer inquiries leveraging the same knowledge base used for E-Service.

·        Research and Discovery:  provides advanced search and classification of internal and external data, enabling early detection of issues in corporate and legal processes.  This helps to answer questions such as:  What pending legislation could affect our launch?  Is my patent unique?

·        Intelligence Analysis:  analyzes human intelligence and public intelligence for the purpose of identifying and preventing criminal activity.  This helps to answer question such as:  Who is the leader of this cell?  How is money being moved?

Attensity Group will also provide custom applications.  It will sell in the Americas under the banner: Attensity, An Attensity Group Company and in Europe under the banner: Empolis, An Attensity Group Company.

Does it make sense?

In a nutshell, the combination of these companies elevates Attensity Group out of niche application provider status to $40M to $50M mid-sized company.  The combined company is backed by Aeris Holdings.

Is this a viable move?   Attensity Group claims that companies want applications, not platforms that they don’t know what to do with.  I think that the move makes sense because companies do want specific, straight-forward solutions to their problems.  Look at how popular the SaaS model has been for Voice of the Customer solutions that utilize text analytics.  And, the traction that Attensity has gotten with its VoC solutions.  Additionally, small companies have a hard time making it as platform providers.  Text Analytics platforms are best left to substantially larger companies such as IBM or EMC that offer text analytics in conjunction with platforms (or as part of it).  These companies have built a sustainable ecosystem around their platforms.   

It also doesn’t hurt that Attensity Group has some solid legs to stand on. 

I attended the IBM Business Analytics and Optimization (BAO) briefing yesterday at the IBM Research facility in Hawthorne, NY.   At the meeting, IBM executives from Software, Global Business Services, and Research (yes, Research) announced its new consulting organization, which will be led by Fred Balboni.   The initiative includes 4000 GBS consultants working together with the Software Group and Research to deliver solutions to customers dedicated to advanced business analytics and business optimization. The initiative builds off of IBM’s Smarter Planet . 

IBM believes that there is a great opportunity for companies that can take all of the information they are being inundated with and use it effectively.  According to IBM (based on a recent study), only 13% of companies are utilizing analytics to their advantage.  The business drivers behind the new practice include the fact that companies are being pressured to make decisions smarter and faster.  Optimization is key as well as the ability for organizations to become more predictive.  In fact, the word predictive was used a lot yesterday. 

According to IBM, with an instrumented data explosion, powerful software will be needed to manage this information, analyze it, and act on it.  This goes beyond business intelligence and business process management, to what IBM terms business analytics and optimization.  BAO operationalizes this information via advanced analytics and optimization.  This means that advanced analytics operating on lots of data will be part of solutions that are sold to customers.  BAO will go to market with industry specific applications

‘Been doing this for years

IBM was quick to point out that they have been delivering solutions like this to customers for a number of years Here are a few examples:

·        The Sentinel Group , an organization that provides healthcare anti-fraud and abuse services, uses IBM software and advanced analytics to predict insurance fraud.

·        The Fire Department of New York is using IBM software and advanced analytics to “ build a state of the art system for collecting and sharing data in real-time that can potentially prevent fires and protect firefighters and other first responders when a fire occurs”.

·        The Operational Risk data exchange (ORX) is using IBM to help its 35 member banks better analyze operational loss data from across the banking industry.  This work is being done in conjunction with IBM Research.

These solutions were built in conjunction with the members of IBM Research who have been pioneering new techniques for analyzing data.  This is a group of 200 mathematicians and other quantitative scientists.  In fact, according to IBM, IBM research has been part of a very large number of client engagements.  A few years back, the company formalized the bridge between GBS and Research via the Center for Business Optimization.  The new consulting organization is yet a further outgrowth of this. 

The Dawn of a New Era

The new organization will provide consulting services in the following areas:

·        Strategy

·        Biz Intelligence and Business Performance Management

·        Advanced Analytics and Optimization

·        Enterprise info management

·        Enterprise Content management

It was significant that the meeting was held at the Research Labs.  We lunched with researchers, met with Brenda Dietrich, VP of Research, and saw a number of solution demos that utilized intellectual property from Research.  IBM believes that its research strength will help to differentiate it from competitors.

The research organization is doing some interesting work in many areas of data analysis including mining blogs, sentiment analysis, and machine learning and predictive analysis.  While there are researchers on the team that are more traditional and measure success based on how many papers they publish, there are a large number that get excited about solving real problems for real customers.   Brenda Dietrich requires that each lab participate in real-world work. 

Look, I get excited about business analytics, it’s in my blood.  I agree that world of data is changing and companies that make the most effective use of information will come out ahead. I’ve been saying this for years.   I’m glad that IBM is taking the bull by the horns.  I like that Research is involved. 

It will be interesting to see how effectively IBM can take its IP and reuse it and make it scale across different customers in different industries in order to solve complex problems.  According to IBM, once a specific piece of IP is used several times, they can effectively make it work across other solutions.  On a side note, it will also be interesting to see how this IP might make its way into the Cognos Platform.  That is not the thrust of this announcement (which is more GBS centric), but is worth mentioning.  

Can people learn to innovate?  It may depend on how you define innovation.  When most people think of innovation, they think invention – like developing a post-it note. But, innovation can be something different.  Innovation can be as simple as using something old in a new way or as complex as inventing something… well really complex.  In the technology world, innovation can be about taking a product to market and extending its life.  Or, about finding new markets for technology that you already have.  

The folks at Invention Machine believe that you can learn to innovate, regardless of the type of innovation.  In fact, they believe that innovation can be injected everywhere in the product life cycle – from planning and research to design to preventing and fixing defects.  The goal is to make innovation “repeatable and sustainable”.  They have crafted an interesting solution that blends best practices in innovation with text analytics software so that the software can actually act as a subject matter expert. 

Best practices in innovation include text analytics

How does it work?  The solution- called Goldfire- provides an innovation framework with a semantic engine that allows you to mine internal documents, patents, and external literature to find answers to questions.  It supports the following innovation categories:

·        Analyze a market: enables the user to cull through the literature, extracting relevant information in order to understand a technology better

·        Develop a Product:  enables the user to design a new or hybrid system, providing best practices frameworks for this along with the ability to analyze patents and other documents relevant to the functionality of the product.

·        Improve a System: enables users to diagnose and fix a problem and resolve contradictions in an existing system. 

·        Risk Management: provides predictive failure analysis

·        Leverage Intellectual property: enables users to cull through patents

Right now, the company is focused on the manufacturing sector only.  For example, assume you are interested in developing a new product.   You would log into the Invention Machine solution and select the task (on the left hand tool bar) design a system. You would see some steps involved with designing a new system.  These are illustrated in the diagram below.

Here, you can see some best practices concerning what is involved in designing a system and then the steps you would take to do this.  One step is to explore the opportunity space.  The company uses Natural Language Processing techniques to rank concept retrieval requests relative to the question being asked. For example, say you’re interested in designing a new coffee maker and you need a component to heat water.   If you ask, “What heats water?” the result “The anode heats water” would be ranked higher than “Water is heated on a stove” because semantically, the first answer is more accurate.  Note that here the user is interested in the function of heating water and the anode is a component that can help to do this.  The text analytics engine will make use of selective substations for accurate extended concept retrieval.  Selective means only the words that are precise substitutes are used for automatic rephrasing.

The order of words makes a difference in what would come back in a search query.  Using the coffee maker example, assume the user is interested in getting power to the switch on the machine.  If the user asks, “How to power the switch?” and “How to switch the power?” the software recognizes the difference in concepts expressed by the different word orders.  This is because the semantic approach used by Invention Machine looks at the word roles and the relationships implied by the structure of language rather than a simple keyword or Bayesian approach.    

Can you learn to innovate?

I have to say that I have not seen anything quite like this before.  I’ve seen text analytics used in R&D to search the literature and patents, but not wrapped in best practices. 

Invention Machine’s semantic approach is interesting because it helps users focus on components and functions and, at one level, this is a part of what innovation is all about.  You could argue that components and functions are relevant to any industry.  If I am building something or even delivering a service, I am interested in what components I would need to meet a specific function.  Right now Invention Machines is focused on manufacturing, but you could certainly see how the application could be extended to other domains. 

I’ve recently noticed a small buzz building about the notion of “real time” text analytics.  In fact, I’ve come across several vendors talking about it in relation to customer experience and financial trading.  The idea is that these companies analyze a lot of  unstructured data quickly and provide real time  information to the people who need it.  Of course, real time can mean something different depending on the context.  It might mean continuously monitoring customer feedback from multiple sources to improve customer retention.  This could mean analyzing information on an hourly basis.  Or it might mean millisecond response time analysis in the case of monitoring current events to use for trading purposes.   In the first example, millisecond response time may not be necessary.  In the case of financial trading and other activities, it can make a difference.

One vendor that offers this kind of analytical power in a SaaS model is Psydex.  Robin Bloor and I recently had the opportunity to speak to Rob Usey and Don Simpson about Psydex.    Robin has also written about this company in his blog.

Psydex analyzes huge amounts of unstructured feeds to assess the impact of news events.  The company takes in and analyzes feeds from various news sources like Thomson Reuters, Dow Jones, Associated Press, Business Wire as well as social networking sites like Twitter to extract useful information.   It can even pull in TV news feeds and text messaging.  Latency is less than 20 milliseconds to query decades of content.    The secret sauce is the company’s ability to organize streaming content in-memory and around time.  The goal is when an event hits, rather than taking hours or minutes to get information to the person who needs to know, it takes seconds. 

How it works

Psydex organizes information around semantic topics.  These topics are built using rules that represent events, people, themes, places, and so on.  For example oil might be a topic.  The topic oil might include oil, crude oil and the price of oil. Another topic, such as Oil Problem, might incorporate this topic as well as any information relating to spills, explosions, etc. 

The company uses a proprietary grid-based indexing scheme for organizing content in memory with topic models stored separately. These topics are then analyzed for trends and patterns.   Specifically, Psydex uses all of its information to establish a baseline for normal topic noise levels.  The company tracks these topics and can detect when a statistically significant deviation occurs.  The screen shot below illustrates this idea.  This shot shows what a Psydex user might see when a plane crash hits the news wires.  In this case, it is about the plane crash in Japan earlier this week.

In this example, one semantic topic is plane crash.  The topic was built using a rule that includes phrases such as plane is down,plane crashed,plane crash,jet has crashed,helicopter crashed,helicopter crash,plane down,jet crashed,jet crash,plane went down, just crashed,plane has crashed,airplane has crashed,helicopter has crashed,jet has crashed,airliner has crashed,plane just crashed,plan, crashes,flight+crashes,flight+crashed – you get the idea.   There is also another topic called Japan.

The view is a five day hourly view.  Immediately you see a spike for Japan and then two other spikes beginning at 1800 hours.   You can also see the associated content stating that a plane went down in flames.  This content is coming from multiple sources.  You can also see that the noise level is up significantly for the Plane Crash topic.   The user interface also allows you to see potential related topics. In this case, Japan is a related topic.  These are topics that were found in the same articles and/or time slots.  The spikes for the two topics together are shown in blue. 

While this plot shows hourly spikes, the company can take this time interval down to the second.   Psydex sent over a log from yesterday that showed the story about the fedex crash hit the wire at 18:21:47.  Psydex signals were generated at 18:21:48. 

Real time BI and Real Time Text Analytics

There has been a lot of hype about real time BI (dealing with structured data) over the past few years and its use in operational systems.  And, then came complex event processing (again, structured data).   And, now real time text analytics.  You can imagine some good use cases for analyzing news-related text in real time.  Trading is obviously a use case that might require some really fast response time.  Government applications might be another.  Psydex would argue that another use case might be brand management because companies might want to use some piece of news or chatter to update their online advertising campaign.  Or, be the first to know if something negative is being said about your company.    Of course, there are other scenarios in which continuously analyzing text other than news feeds as part of an operational process might be useful, as well.

I was at a Verizon Wireless store late last month.  My mission was to enable my blackberry phone to receive emails while I was in the Bahamas.  I had already spent about a half hour on the phone with a Verizon wireless customer service representative who told me that:

1. My Subscriber Identity Module (SIM) card wasn’t activated 
2. I needed to change my billing plan in order to cost-effectively receive email while traveling. 
3. I would have to go to the Verizon Wireless store to do this.

So, on a Sunday a few weeks back, I trudged down to the store.  As I waited for my name to come to the top of the digital queue, I started thinking about how Verizon Wireless is dealing with service management.  As many of you know, Hurwitz & Associates is in the process of writing our latest book, Service Management for Dummies.  I figured that while I was waiting, at least I could be thinking about something useful.   Actually, that isn’t entirely true.  I was actually too aggravated while I was waiting there to think clearly about it.  I thought more about service management once the whole incident was over.

What is Service Management?

First, a very simple primer on service management.  We define a service as

“A purposeful activity carried out for the benefit of a known target.” 

In this case, I was the target and the activity was to get my emails coming onto my phone when I was overseas.   Each service has inputs and outputs.  The internal activity of the service (i.e. what happens when I asked my question) transforms the inputs into the outcome and it typically involves some skilled participants as well as assets and tools that are used to execute the service.   My request triggered a series of activities which theoretically should have resulted in a positive outcome. 

We define service management as:

“The management of a service to ensure that it meets the critical outcomes the customer values and the stakeholders want to provide.”

In other words, meeting customer expectations is key to a successful service management strategy.  The delivery of a service that meets these expectations can be complex as it often involves people, process, and various layers of technology.   The management of a single service such as the one I described, can also involve multiple component layers.  However, if the goal is to optimize the service this means that every link in the chain needs to work.

The technology to support service management ranges from service desks to monitoring systems to data center optimization.  Here, I am just focusing primarily on the service desk technology and the people and processes that go along with it.  The service desk provides a single point of contact for customers and internal users to report any issues they may have with the IT service.

What Verizon Wireless needs to Improve

As it turned out, it took me over an hour to get my SIM card activated and a new billing plan put on my account.  And, as it turned out, I couldn’t receive my email in the Bahamas anyway because of some new frequency in the Bahamas that didn’t work with my model phone.  That took another 30 minutes to figure out once I was in the Bahamas.  And, customer service never changed my billing plan back to my original plan to reflect this fact, even though I had asked them to.  That took another 20 minutes to resolve once I got back. Whew! 

So, what should this company do better in terms of service desk management and customer support? 

1. Service Desk Queue.   When I came into the store, I entered my name and information into Verizon’s service desk system at an in-store kiosk.  I put my name on the technical queue and subsequently found out that my problem was really a customer service related problem.  Then, I had to put my name on the customer service queue.  While all of the people on the technical support and sales queue were serviced, I sat there waiting (about 45 minutes) for a customer support person.  There were two customer service reps in the store but the other one was taking a break.  The queue needs to be monitored.  If someone is in the queue for too long (like more than 10 minutes) then either the person “on break” needs to come off of break or a technical service person who was free should have helped – at least to install the SIM card. This monitoring is directly related to KPIs and SLAs described in bullet 4.  
2. Training.  Once I finally got to speak to a customer service representative she couldn’t figure out how to make the billing change for me.  She had to call over her supervisor and they had a long discussion about it.  What happened to all of that training that Verizon advertises?
3. Service Desk Notification.  Two things here. 
   • As I mentioned, I first called a customer service representative who directed me to the store.  When my service didn’t work, I called a customer service rep (the international one) again.  When I had a billing question I called yet another customer service representative.  Yet, none of these people seemed to know about any of my calls or visits.  The service desk should be able to track this. It would have moved things along much faster and improved those metrics.
   • While I was waiting for the customer service representative to help me, another man was talking to a rep and said that he wanted to get a new LG phone.  The rep said that he could sell the man one, but wouldn’t activate it because there was a problem with the phone.  When the man questioned this, the rep confessed that he didn’t know what the problem was but said it would be fixed within 48-72 hours and the man should contact the store for updates.  Wrong on several counts.  First, the rep should have known what the problem was. The service desk had notified him about the expected time to repair, but didn’t say much about the problem.   The man already owned the phone – perhaps he was having the same problem.  Second, the rep should have said that he would notify the guy.  This would be easy enough if the guy had a text message feature on his phone.  At least he could have offered.
4. Key Performance Indicators.  Verizon needs to put some business focused key performance indicators in place that would help them understand whether they are meeting customer expectations.  This shouldn’t be too difficult since it appears to have a service desk.  Obvious metrics include time to service, time spent servicing (by type of issue).  Others might include average time to resolve an incident,  Customer support turnover and I could go on and on.  It would also be helpful if the customer could answer a few customer satisfaction questions at the kiosk on the way out.   That way they could correlate dropped service with incident management.
5. The Knowledge base.  Two different customer service representatives at Verizon didn’t know that I couldn’t access my email using my model Blackberry in the Bahamas.  Verizon needs to deploy a knowledge base that stores information like this.  If it has a knowledge base, it needs to update it and train its people how to use it.

Clearly, my expectations for service were not met.