I’ve been writing about text analytics for a number of years, now. Many of my blogs have included survey findings and vendor offerings in the space.  I’ve also provided a number of use cases for text analytics; many of which have revolved around voice of the customer, market intelligence, e-discovery, and fraud.  While these are all extremely valuable, there are a number of other very beneficial use cases for the technology and I believe it is important to put them out there, too.

Last week, I spoke with Daniel Mayer, a product-marketing manager, at TEMIS about the publishing landscape and how text analytics can be used in both the editorial and the new product development parts of the publishing business.  It’s an interesting and significant use of the technology.

First a little background.  I don’t believe that it comes as a surprise to anyone that publishing, as we used to know it has changed dramatically.  Mainstream newspapers and magazines have given way to desktop publishing and the Internet as economics have changed the game.  Chris Anderson wrote about this back in 2004, in Wired, in an article he called “The Long Tail” (it has since become a book).  Some of the results include:

• Increased Competition.  There are more entrants, more content and more choice on the Internet and much of it is free.

• Mass market vs. narrow market.  Additionally, whereas the successful newspapers and magazines of the past targeted a general audience, the Internet economically enables more narrow appeal publications.  
• Social, Real time.  Social network sites, like twitter, are fast becoming an important source of real time news. 

All of this has caused mainstream publishers to rethink their strategies in order to survive.  In particular, publishers realize that content needs to be richer, interactive, timely, and relevant.

Consider the following example.  A plane crashes over a large river, close to an airport.  The editor in charge of the story wants to write about the crash itself, and also wants to include historical information about the cause of plane crashes (e.g. time of year, time of day, equipment malfunction, pilot error, etc based on other plane crashes for the past 40 years) to enrich the story.  Traditionally, publishers have annotated documents with key words and dates.   Typically, this was a manual process and not all documents were thoroughly tagged.  Past annotations might not meet current expectations. Even if the documents were tagged, they might have been tagged only at a high level (e.g. plane crash), so that the editor is overwhelmed with information.   This means that it might be very difficult her to find similar stories, much less analyze what happened in other relevant crashes.  

Using text analytics, all historical documents could be culled for relevant entities, concepts, and relationships to create a much more enriched annotation scheme.  Information about the plane crash such as location, type of planes involved, dates, times, and causes could be extracted from the text.  This information would be stored as enriched metadata about the articles and used when needed.  The Luxid Platform offered by TEMIS would also suggest topics close to the given topic.  What does this do? 

• It improves the productivity of the editor.  The editor has a complete set of information that he or she can easily navigate.  Additionally, if text analytics can extract relationships such as cause this can be analyzed and used to enrich a story.
• It provides new opportunities for publishers.  For example, Luxid would enable the publisher to provide the consumer with links to similar articles or set up alerts when new, similar content is created, as well as tools to better navigate data or analyze it (this might be used by fee based subscription services).  It also enables publishers to create targeted microsites and topical pages, which might be of interest to consumers.

Under many current schemes, advertisers pay online publishers.  Enhancing navigation means more visits, more page views, and a more focused audience, which can lead to more advertising revenue for the publisher.  Publishers, in some cases, are trying to go even further, by transforming readers into sales leads and receiving a commission from sales. There are other models that publishers are exploring, as well.  Additionally, text analytics could enable publishers to re-package content, on the fly (called content repurposing), which might lead to additional revenue opportunities such as selling content to brand sponsors that might resell it.  The possibilities are numerous.

I am interested in other compelling use cases for the technology.

IBM made a number of announcements last week at IOD regarding new products/offerings to help companies analyze content.  One was Cognos Content Analytics, which enables organizations to analyze unstructured data alongside structured data.  It also looks like IBM may be releasing a “voice of the customer” type service to help companies understand what is being said about them in the “cloud” (i.e. blogs, message boards, and the like).  Stay tuned on that front, it is currently being “previewed”.

I was particularly interested in a new product called IBM Infosphere Content Assessment, because I thought it was an interesting use of text analytics technology.  The product uses content analytics (IBM’s term for text analytics) to analyze “content in the wild”.  This means that a user can take the software, run it over servers that might contain terabytes (or even petabytes) of data to understand what is being stored on servers.  Here are some of the potential use cases for this kind of product:

• Decommission data.  Once you understand the data that is on a server, you might choose to decommission it, thereby freeing up storage space
• Records enablement.   Infosphere Content Assessment can also be used to identify what records need to go into a records management system for a record retention program
• E-Discovery.  Of course, this technology could also be used in litigation, investigation, and audit.  It can analyze unstructured content on servers which can help to discover information that may be used in legal matters or information that needs to meet certain audit requirements for compliance.

The reality is that the majority of companies don’t formally manage their content.  It is simply stored on file servers.  The IBM product team’s view is that companies can “acknowledge the chaos”, but use the software to understand what is there and gain control over the content.  I had not seen a product positioned quite this way before and I thought it was a good use of the content analysis software that IBM has developed.

If anyone else knows of software like this, please let me know.

Last Friday, SAS announced that it was partnering with Teradata and Elder Research Inc. (a data mining consultancy) to open a Business Analytics Innovation Centre.  According to the press release,

“ Recognising the growing need and challenges businesses face driving operational analytics across enterprises, SAS and Teradata are planning to establish a centralised “think tank” where customers can discuss analytic best practices with domain and subject-matter experts, and quickly test or implement innovative models that uncover unique insights for optimising business operations.”

The center will include a lab for pilot programs, analytic workshops and proof of concept for customers.  I was excited about the announcement, because it further validated the fact that business analytics continues to gain steam in the market. I had a few questions, however, that I sent to SAS.  Here are the responses. 

Q. Is this a physical center or a virtual center?  If physical – where is it located and how will it be staffed?  If virtual, how will it be operationalized?

R. The Business Analytics Innovation Center will be based at SAS headquarters in Cary, North Carolina.  We will offer customer meetings, workshops and projects out of the Center. 

Q. Will there be consulting services around actually deploying analytics into organizations?  In other words, is it business action oriented or more research oriented?

R.  The Business Analytics Innovation Center will offer consulting services around how best to deploy analytics into organizations, as well as conduct research-based activities to help businesses improve operational efficiency. 

Q.  Should we expect to hear more announcements from SAS around business analytics, similar to what has been happening with IBM?

R.  As the leader in business analytics software and services, SAS continues to make advances in its business analytics offerings. You can expect to hear more from SAS in this area in 2010

I’m looking forward to 2010!

I just got back from the IBM Information on Demand (IOD) conference in Las Vegas.  A key message was that the future is in analytics and predictive analytics at that.  IBM has already invested $12B ($8B acquisitions, $4B organic growth) in analytics since 2005.  Its recent purchase of SPSS has enabled the company to put a stake in the ground regarding leading the analytics charge.

Predictive analytics uses historical data to try to predict what might happen in the future.  There are different technologies that can help you to do this including data mining and statistical modeling.  For example, a wireless telecommunications company might try to predict churn by analyzing the historical data associated with customers who disconnected the service vs. those that did not.  Attributes that might serve as predictors include dropped calls, calling volume (in network, out of network), demographic information, and so on.  An insurance company might try to predict future fraud using past claims that where the outcome is known.  Adam Gartenberg’s blog describes more examples of this.  IBM plans to make predictive analytics more pervasive in several ways. 

• Making models easier to build. It will make predictive modeling tools easier to use for those who build the models.  A good example of this is the SPSS PASW Modeler product that uses a visual paradigm to build various kinds of models.  I stopped by the SPSS booth at the show and saw the software at the demo area and it is nice with lots of feature/functionality built into it.  Training is available (and I would argue necessary), for example, to understand when you might want to use a certain kind of model. 
• Embedding the predictive model in a process.  Here, the predictive model would become part of a business process. For example, a predictive model might be built into a claims analysis process.  The model determines the characteristics and predictors of claims that might be classified as fraudulent.  As the claims come through the process, those that are suspicious, based on the model, would get kicked out for further examination.  

So, given these two approaches, can predictive analytics become pervasive? 

In the case of making predictive modeling tools easier to use, the question isn’t whether someone can use a tool, but whether he or she can use it correctly.   The goal of a tool like PASW is to enable business users to build advanced models. Could a BI power user who is accustomed to slicing and dicing and shaking and baking data effectively use a tool like this?  Possibly, if they have the right thought process and they pay attention to the part of the training that describes what type of technique to use for what type of problem.  It is a good goal.  Time will be the judge.

As for embedding predictive analytics in business processes; this is already starting to happen and here is where the possibility of making prediction more pervasive gets exciting.  For example, telecommunications companies can embed predictive analytics into a call center application to understand an action that a customer might take.  A call center representative can make use of the results of the model (without understanding the model or what it does).  He or she is simply fed information, from the model, (in real time) to help service a customer most effectively.   The model can be created by a skilled analytics person, but deployed in such a way that it can help a lot of other people across an organization.  One key will be the ability to integrate a model into the actual code and culture behind a business process.

Look, I don’t have a crystal ball (little predictive modeling humor there), but I am very excited about the possibilities of predictive modeling.  I did this kind of modeling for years at Bell Laboratories, way back when, and it is great to see it finally gaining traction in the marketplace.  Predictive analytics can be a truly powerful weapon in the right hands.

IBM has dominated a good deal of the news in the business analytics world, recently. On Friday, it completed the purchase of SPSS and solidified its position in predictive analytics.  This is certainly the biggest leg of a recent three-prong attack on the analytics market that also includes:

• Purchasing Red Pill.  Red Pill is a privately-held company headquartered in Singapore that provides advanced customer analytics services -  especially in the business process outsourcing arena.  The company has talent in the area of advanced data modeling and simulation for various verticals such as financial services and telecommunications. 
• Opening a series of solutions centers focused on advanced analytics.  There are currently four centers operating now: in New York (announced last week), Berlin, Beijing, and Tokyo.  Others are planned for Washington D.C. and London. 

Of course, there is a good deal of organizational (and technology) integration that needs to be done to get all of the pieces working together (and working together) with all of the other software purchases IBM has made recently.  But what is compelling to me is the size of the effort that IBM is putting forth.  The company obviously sees an important market opportunity in the advanced analytics market.  Why?  I can think of at least four reasons:

• More Data and different kinds of data.  As the amount of data continues to expand, companies are finally realizing that they can use this data for competitive advantage, if they can analyze it properly.  This data includes traditional structured data as well as data from sensors and other instruments that pump out a lot of data, and of course, all of that unstructured data that can be found both within and outside of a company.
• Computing power.  The computing power now exists to actually analyze this information.  This includes analyzing unstructured information along with utilizing complex algorithms to analyze massive amounts of structured data. And, with the advent of cloud computing, if companies are willing to put their data into the cloud, the compute power increases.
• The power of analytics.  Sure, not everyone at every company understands what a predictive model is, much less how to build one.  However, a critical mass of companies have come to realize the power that advanced analytics, such as predictive analysis can provide.  For example, insurance companies are predicting fraud, telecommunications companies are predicting churn.  When a company utilizes a new technique with success, it is often more willing to try other new analytical techniques. 
• The analysis can be operationalized.  Predictive models have been around for decades.  The difference is that 1) the compute power exists and 2) the results of the models can be utilized in operations.  I remember developing models to predict churn many years ago, but the problem was that it was difficult to actually put these models in to operation.  This is changing.  For example, companies are using advanced analytics in call centers.  When a customer calls, an agent knows if that customer might be likely to disconnect a service.  The agent can utilize this information, along with recommendations for new service to try to retain the customer. 

 So, as someone who is passionate about data analysis, it is good to see that it is finally gaining the traction it deserves.

I had an interesting chat with Roman Stanek, CEO of Good Data last week about the debate over data security and reliability in the cloud.  For those of you who are not familiar with Good Data, it provides a collaborative business analytics platform as a SaaS offering.

The upshot of the discussion was something like this:

 The argument over data security and reliability in the cloud is the wrong argument.    It’s not just about moving your existing data to the cloud.  It’s about using the cloud to provide a different level of functionality, capability, and service than you could obtain using a traditional premises solution- even if you move that solution to the “hosted” cloud. 

What does this mean?  First, companies should not simply be asking the question,  “should I move my data to the cloud?”  They should be thinking about new capabilities the cloud provides as part of the decision making process.  For example, Good Data touts its collaborative capabilities and its ability to do mash ups and certain kinds of benchmarking (utilizing external information) as differentiators to standard premises-based BI solutions.  This leads to the second point that a hosted BI solution is a different animal than a SaaS solution. For example, a user of Good Data could pull in information from other SaaS solutions (such as Salesforce.com) as part of the analysis process.  This might be difficult with a vanilla hosted solution.

 So, when BI users think about moving to the public cloud they need to assess the risk vs. the reward of the move.  If they are able to perform certain analysis that they couldn’t perform via a premises model and this analysis is valuable, then any perceived or real risk might be worth it.

Visualization can change the way that we look at data and information.   If that data contains a geographic/geospatial component then utilizing location information can help provide a new layer of insight for certain kinds of analysis.  Location intelligence is the integration and analysis of visual geographic/geospatial information as part of the decision making process.  A few examples where this might be useful include:

• Analyzing marketing activity
• Analyzing sales activity
• Analyzing crime patterns
• Analyzing utility outages
• Analyzing  military options

I had the opportunity to meet with the team from SpatialKey the other week.  SpatialKey offers a location intelligence solution, targeted at decision makers, in a Software as a Service (SaaS) model.  The offering is part of Universal Mind, a consulting company that specializes in design and usability and had done a lot of work on dashboards, Geographic Information Systems, and the like.  Based on its experience, it developed a cloud-based service to help people utilize geographic information more effectively. 

According to the company, all the user needs to get started is a CSV file with their data. Files must contain an address, which SpatialKey will geocode, or latitude and longitude for mapping purposes.  It can contain any other structured data component.   Here is a screen shot from the system.  It shows approximately 1000 real estate transactions from the Sacramento, California area that were reported over a five day period. 

There are several points to note in this figure.  First, the data can be represented as a heat map, meaning areas where there are large number of transactions appear in red, lower numbers in green.   Second, the software gives the user the ability to add visualization pods, which are graphics (on the left) that drill down into the information.  The service also allows you to incrementally add other data sets, so you can visualize patterns.  For example, you might choose to add crime rates or foreclosure rates on top of the real estate transactions to understand the area better.  The system also provides filtering capabilities through pop ups and other sliders. 

SpatialKey has just moved out of beta and into trial.  The company does not intend to compete with traditional BI vendors.  Rather, its intention is to provide a lightweight alternative to traditional BI and GIS systems.  The idea would be to simply export data from different sources (either your company data stores or even other cloud sources such as Salesforce.com) and allow end users to analyze it via a cloud model.

 The future of data is more data.  Location intelligence solutions will continue to become important as the number of devices, such as RFID and other sensors continue to explode.   As these devices spew yet even more data into organizations, people will want a better way to analyze this information.  It makes sense to include geographic visualization as part of the business analytics arsenal.

Over the past few days, I got a chance to speak to two different companies in the business analytics space about data in the cloud.  One was a SaaS provider, the other an enterprise software vendor.  Two vendors, two different stories that illustrate the jury is still very much out regarding how end users feel about putting their sensitive data in the cloud.

The SaaS provider runs its operation in the Amazon EC2 cloud (and no, I do not believe that the company was using Amazon’s new Virtual Private Cloud services).  Interestingly, the company said that even organizations in the public sector were starting to get comfortable with the level of security and reliability of the cloud.  In fact, the company said that the security and reliability of a cloud data center was, more often than not, better than the security and reliability of the infrastructure on a customer’s premises.    This is an argument I have heard before.

The enterprise software vendor also provides a cloud-like option to its customers.  This company told me that 80% of its customers did not want to keep their data in a cloud environment because of security concerns.  These customers are analyzing some pretty sensitive data about customers, revenue, and the like.

Considerations

When you think about data in the cloud, it is important to think about it from at least 2 perspectives:  Yours and the cloud provider.  Let’s say you are a mid sized company running a business analytics application in the cloud.  From your perspective, the amount of data that you are storing and processing in this service may not that great.  However, your SaaS provider might have five thousand customers.  In fact, it may be running its application across many servers.  It may house your data and the 4999 other companies it calls its clients on multiple database servers.   Once your company’s data is in the SaaS provider’s database, it may exist there with data from other companies.  The concern, of course, is that your data is in a shared environment that you don’t control.   The SaaS provider will tell you that since this is their business, they have a higher level of skill around issues such as security and reliability than might exist in your own company.  And this may be true, depending on your company.  Each organization needs to evaluate its own needs and issues and make a decision for itself.

Here are some issues to consider about security and reliability:

Data Security

o       Different kinds of data require different levels of security.  There are huge numbers of issues associated with security –including transporting the data securely to the cloud, as well as data access and data leakage .  (those interested should check out a very interesting paper that looks at potential threats from “non-provider affiliated malicious parties” by Ristenpar, Tromer, Shacham, and Savage.)

o       Along with this are controls over your data that need to be addressed.  These include controls to ensure data integrity such as completeness, accuracy, and reasonableness?  There are processing controls to ensure that data remains accurate. And, there also need to be output controls in place. And of course, there needs to be controls over the actual transport of data from your company to the cloud.

o       There are also data compliance issues to think about.  These might include retention as well as issues such as cross country data transfer.

o       Data ownership – Who owns your data once it goes into the cloud?  Some service providers might want to take your data, merge it with other data and do some analysis.

Reliability/Availability

o       Availability:  A provider might state that its servers are available  99.999% of the time, but read the contract.  Does this uptime include scheduled maintenance?

o       Business continuity plans.  If you cloud provider’s data center goes down, what plans are in place to get your data back up and available again.  For example, a SaaS vendor might tell you that they back up data every day, but it might take several days to get the back up onto systems in another facility.

o       Loss of data. What provisions are in your contract if something happens and your providers loses your data?

o       Contract termination-   How will data be returned if the contract is terminated?

o       Vendor Lock-in – If you create applications with one cloud vendor and then decide to move to another vendor, you need to find out how difficult it will be to move your data from one to the next.

It’s been a while since I’ve written a blog because I’ve spent most of the summer working on our Cloud Computing for Dummies Book along with Judith Hurwitz, Robin Bloor and Marcia Kaufman.  I did, however, post a short blog about premise vs. premises, which I urge anyone involved in Cloud Computing to read.

In any event, the book will be out in the early fall and I have to say that writing it was, for the most part, time well spent.  We did a lot of research for the book and I think that readers will find it very helpful as they try to sort out how and why and what of the cloud.  Judith recently posted a blog about 10 things she learned about the cloud while writing this book.   In keeping with this theme, here are five key considerations for managing the cloud.   I’ll touch on each briefly.  Obviously, we go into much more detail about Management in the book. Note that a number of these areas are still in their infancy. 

• Quality of Service.  I’ve noticed that when many cloud vendors address managing the cloud, they only talk about how to manage resources over a virtualized infrastructure – specifically about self service provisioning and about some sort of automated resource allocation. They’re not necessarily talking about fixing problems, providing service level agreements, or managing security. In other words, they’re not talking about managing the quality of the service they are providing.  However, a key element of managing the cloud is ensuring Quality of Service (QOS) which itself includes a host of issues such as availability, reliability, scalability, maintainability, integrity, security, and all of the other “ends in a y” words.     At a minimum, it is important for companies that use a cloud provider to have visibility into the services so that they can measure and monitor what is going on and whether their providers are meeting any SLAs that have been put in place.  Of course, negotiating these SLAs in another important consideration.
• Governance.   This area is still pretty much in its infancy.  Governance defines who is responsible for what and the policies and procedures people or groups need to follow to make sure business goals and objectives are met. Cloud governance requires governing your own infrastructure as well as infrastructure that you don’t totally control.  This includes understanding risk (such as compliance risk, contract risk, interoperability risk, billing risks, etc. etc.) as well as ensuring performance goals.  A key aspect of a governance strategy will be to put together the right group to interface with both internal and external providers to make sure that policies and procedures get enforced.
• Standards.  Another nascent area.     A standard is an agreed upon approach for doing something.  Cloud standards are needed to ensure interoperability, portability, and integration.  There are a number of organizations and informal groups that are addressing standards issues in the cloud environment.  Some of these organizations have been around for years; others are relatively new.  Several of these organizations have gotten together to create a cloud standards coordination wiki so each group can post their work in one spot.  You can find this wiki at www.cloud-standards.org
• Security and Privacy.  This topic has received a lot of attention by various groups such as the Cloud Security Alliance. The same principles that apply to security on your own premises will apply in the cloud.  This includes identity management to ensure that only authorized persons are allowed to access assets, as well as the ability to determine legitimate from illegitimate activity.  A huge area of concern is protecting data in the cloud.  This includes dealing with compliance issues associated, for example, with cross border data movement. 
• Dealing with Data.  Closely related to data security is the issue of ensuring that proper controls are in place for issues like co-mingling of data or secondary use of data (e.g. for marketing purposes).  This includes auditability of data in the cloud (yet another emerging area).    In addition to audit and control is also the issue of how vendors are storing and accessing the massive amount of data that is being stored in the cloud.  This has generated new ways of thinking about database management systems and other data stores.

 Cloud manageability is a big, complex, and evolving subject and clearly I’ve only given you a taste of some of the issues involved.   I would love to hear your thoughts about the subject.

With all of the research I’ve been doing for our latest book:  Cloud Computing for Dummies, I’ve noticed something very disturbing.  Maybe it’s because I come from a telecommunications background, that this bothers me so much – but has anyone else noticed that people are misusing the word premise when describing aspects of the cloud?  I keep reading articles and blogs where an author refers to an “on premise” solution.  The proper term is premises as in – on your premises (see below).

From Dictionary.com

Premise:  a proposition supporting or helping to support a conclusion.

Premises:  a tract of land including its buildings.

 Even vendors in the space are making this mistake.  It’s appalling.  I could list dozens of examples of this error.  Has the definition of the word changed and I’m missing something?  Or, has the word been used incorrectly so many times that it doesn’t matter anymore?