Are you ready for IBM Watson?

This week marks the one year anniversary of the IBM Watson computer system succeeding at Jeopardy!. Since then, IBM has gotten a lot of interest in Watson.  Companies want one of those.

But what exactly is Watson and what makes it unique?  What does it mean to have a Watson?  And, how is commercial Watson different from Jeopardy Watson?

What is Watson and why is it unique?

Watson is a new class of analytic solution

Watson is a set of technologies that processes and analyzes massive amounts of both structured and unstructured data in a unique way.   One statistic given at the recent IOD conference is that Watson can process and analyze information from 200 million books in three seconds.  While Watson is very advanced it uses technologies that are commercially available with some “secret sauce” technologies that IBM Research has either enhanced or developed.  It combines software technologies from big data, content and predictive analytics, and industry specific software to make it work.

Watson includes several core pieces of technology that make it unique

So what is this secret sauce?  Watson understands natural language, generates and evaluates hypotheses, and adapts and learns.

First, Watson uses Natural Language Processing (NLP). NLP is a very broad and complex field, which has developed over the last ten to twenty years. The goals of NLP are to derive meaning from text. NLP generally makes use of linguistic concepts such as grammatical structures and parts of speech.  It breaks apart sentences and extracts information such as entities, concepts, and relationships.  IBM is using a set of annotators to extract information like symptoms, age, location, and so on.

So, NLP by itself is not new, however, Watson is processing vast amounts of this unstructured data quickly, using an architecture designed for this.

Second, Watson works by generating hypotheses which are potential answers to a question.  It is trained by feeding question and answer (Q/A) data into the system. In other words, it is shown representative questions and learns from the supplied answers.  This is called evidence based learning.  The goal is to generate a model that can produce a confidence score (think logistic regression with a bunch of attributes).  Watson would start with a generic statistical model and then look at the first Q/A and use that to tweak coefficients. As it gains more evidence it continues to tweak the coefficients until it can “say” confidence is high.  Training Watson is key since what is really happening is that the trainers are building statistical models that are scored.  At the end of the training, Watson has a system that has feature vectors and models so that eventually it can use the model to probabilistically score the answers.   The key here is something that Jeopardy! did not showcase – which is that it is not deterministic (i.e. using rules).  Watson is probabilistic and that makes it dynamic.

When Watson generates a hypothesis it then scores the hypothesis based on the evidence.   Its goal is to get the right answer for the right reason.  (So, theoretically, if there are 5 symptoms that must be positive for a certain disease and 4 that must be negative and Watson only has 4 of the 9 pieces of information, it could ask for more.) The hypothesis with the highest score is presented.   By the end the analysis, Watson is confident when it knows the answer and when it doesn’t know the answer.

Here’s an example.  Suppose you go in to see your doctor because you are not feeling well.  Specifically, you might have heart palpitations, fatigue, hair loss, and muscle weakness.  You decide to go see a doctor to determine if there is something wrong with your thyroid or if it is something else.  If your doctor has access to a Watson system then he could use it to help advise him regarding your diagnosis.  In this case, Watson would already have ingested and curated all of the information in books and journals associated with thyroid disease.  It also has the diagnosis and related information from other patients from this hospital and other doctors in the practice from the electronic medical records of prior cases that it has in its data banks.  Based on the first set of symptoms you might report it would generate a hypothesis along with probabilities associated with the hypothesis (i.e. 60% hyperthyroidism, 40% anxiety, etc.).  It might then ask for more information.  As it is fed this information, i.e. example patient history, Watson would continue to refine its hypothesis along with the probability of the hypothesis being correct.  After it is given all of the information and it iterates through it and presents the diagnosis with the highest confidence level, the physician would use this information to help assist him in making the diagnosis and developing a treatment plan.  If Watson doesn’t know the answer, it will state that it has does not have an answer or doesn’t have enough information to provide an answer.

IBM likens the process of training a Watson to teaching a child how to learn.  A child can read a book to learn.  However, he can also learn by a teacher asking questions and reinforcing the answers about that text.

Can I buy a Watson?

Watson will be offered in the cloud in an “as a service” model.  Since Watson is in its own class, let’s call this Watson as a Service (WaaS).  Since Watson’s knowledge is essentially built in tiers, the idea is that IBM will provide the basic core knowledge in a particular WaaS solution space, say all of the corpus about a particular subject – like diabetes – and then different users could build on this.

For example, in September IBM announced an agreement to create the first commercial applications of Watson with WellPoint – a health benefits company. Under the agreement, WellPoint will develop and launch Watson-based solutions to help improve patient care. IBM will develop the base Watson healthcare technology on which WellPoint’s solution will run.  Last month, Cedars-Sinai signed on with WellPoint to help develop an oncology solution using Watson.  Cedars-Sinai’s oncology experts will help develop recommendations on appropriate clinical content for the WellPoint health care solutions. They will assist in the evaluation and testing of these tools.  In fact, these oncologists will “enter hypothetical patient scenarios, evaluate the proposed treatment options generated by IBM Watson, and provide guidance on how to improve the content and utility of the treatment options provided to the physicians.”  Wow.

Moving forward, picture potentially large numbers of core knowledge bases that are trained and available for particular companies to build upon.  This would be available in a public cloud model and potentially a private one as well, but with IBM involvement.  This might include Watsons for law or financial planning or even politics (just kidding) – any area where there is a huge corpus of information that people need to wrap their arms around in order to make better decisions.

IBM is now working with its partners to figure out what the user interface for these Watsons- as a Service might look like.  Will Watson ask the questions?  Can end-users, say doctors, put in their own information and Watson will use it?  This remains to be seen.

Ready for Watson?

In the meantime, IBM recently rolled out its “Ready for Watson.”  The idea is that a move to Watson might not be a linear progression.  It depends on the business  problem that companies are looking to solve.  So IBM has tagged certain of its products as “ready” to be incorporated into a Watson solution.  IBM Content and Predictive Analytics for Healthcare is one example of this.  It combines IBM’s content analytics and predictive analytics solutions that are components of Watson.  Therefore, if a company used this solution it could migrate it to a Watson-as a Service deployment down the road.

So happy anniversary IBM Watson!  You have many people excited and some people a little bit scared.  For myself, I am excited to see where Watson is on its first anniversary and am looking forward to see what progress it has made on its second anniversary.

Four reasons why the time is right for IBM to tackle Advanced Analytics

IBM has dominated a good deal of the news in the business analytics world, recently. On Friday, it completed the purchase of SPSS and solidified its position in predictive analytics.  This is certainly the biggest leg of a recent three-prong attack on the analytics market that also includes:

  • Purchasing Red Pill.  Red Pill is a privately-held company headquartered in Singapore that provides advanced customer analytics services -  especially in the business process outsourcing arena.  The company has talent in the area of advanced data modeling and simulation for various verticals such as financial services and telecommunications. 
  • Opening a series of solutions centers focused on advanced analytics.  There are currently four centers operating now: in New York (announced last week), Berlin, Beijing, and Tokyo.  Others are planned for Washington D.C. and London. 

Of course, there is a good deal of organizational (and technology) integration that needs to be done to get all of the pieces working together (and working together) with all of the other software purchases IBM has made recently.  But what is compelling to me is the size of the effort that IBM is putting forth.  The company obviously sees an important market opportunity in the advanced analytics market.  Why?  I can think of at least four reasons:

  • More Data and different kinds of data.  As the amount of data continues to expand, companies are finally realizing that they can use this data for competitive advantage, if they can analyze it properly.  This data includes traditional structured data as well as data from sensors and other instruments that pump out a lot of data, and of course, all of that unstructured data that can be found both within and outside of a company.
  • Computing power.  The computing power now exists to actually analyze this information.  This includes analyzing unstructured information along with utilizing complex algorithms to analyze massive amounts of structured data. And, with the advent of cloud computing, if companies are willing to put their data into the cloud, the compute power increases.
  • The power of analytics.  Sure, not everyone at every company understands what a predictive model is, much less how to build one.  However, a critical mass of companies have come to realize the power that advanced analytics, such as predictive analysis can provide.  For example, insurance companies are predicting fraud, telecommunications companies are predicting churn.  When a company utilizes a new technique with success, it is often more willing to try other new analytical techniques. 
  • The analysis can be operationalized.  Predictive models have been around for decades.  The difference is that 1) the compute power exists and 2) the results of the models can be utilized in operations.  I remember developing models to predict churn many years ago, but the problem was that it was difficult to actually put these models in to operation.  This is changing.  For example, companies are using advanced analytics in call centers.  When a customer calls, an agent knows if that customer might be likely to disconnect a service.  The agent can utilize this information, along with recommendations for new service to try to retain the customer. 

 So, as someone who is passionate about data analysis, it is good to see that it is finally gaining the traction it deserves.

IBM Business Analytics and Optimization – The Dawn of New Era

I attended the IBM Business Analytics and Optimization (BAO) briefing yesterday at the IBM Research facility in Hawthorne, NY.   At the meeting, IBM executives from Software, Global Business Services, and Research (yes, Research) announced its new consulting organization, which will be led by Fred Balboni.   The initiative includes 4000 GBS consultants working together with the Software Group and Research to deliver solutions to customers dedicated to advanced business analytics and business optimization. The initiative builds off of IBM’s Smarter Planet . 

 

IBM believes that there is a great opportunity for companies that can take all of the information they are being inundated with and use it effectively.  According to IBM (based on a recent study), only 13% of companies are utilizing analytics to their advantage.  The business drivers behind the new practice include the fact that companies are being pressured to make decisions smarter and faster.  Optimization is key as well as the ability for organizations to become more predictive.  In fact, the word predictive was used a lot yesterday. 

 

According to IBM, with an instrumented data explosion, powerful software will be needed to manage this information, analyze it, and act on it.  This goes beyond business intelligence and business process management, to what IBM terms business analytics and optimization.  BAO operationalizes this information via advanced analytics and optimization.  This means that advanced analytics operating on lots of data will be part of solutions that are sold to customers.  BAO will go to market with industry specific applications

 

‘Been doing this for years

 

IBM was quick to point out that they have been delivering solutions like this to customers for a number of years Here are a few examples:

 

·        The Sentinel Group , an organization that provides healthcare anti-fraud and abuse services, uses IBM software and advanced analytics to predict insurance fraud.

·        The Fire Department of New York is using IBM software and advanced analytics to “ build a state of the art system for collecting and sharing data in real-time that can potentially prevent fires and protect firefighters and other first responders when a fire occurs”.

·        The Operational Risk data exchange (ORX) is using IBM to help its 35 member banks better analyze operational loss data from across the banking industry.  This work is being done in conjunction with IBM Research.

 

These solutions were built in conjunction with the members of IBM Research who have been pioneering new techniques for analyzing data.  This is a group of 200 mathematicians and other quantitative scientists.  In fact, according to IBM, IBM research has been part of a very large number of client engagements.  A few years back, the company formalized the bridge between GBS and Research via the Center for Business Optimization.  The new consulting organization is yet a further outgrowth of this. 

 

The Dawn of a New Era

 

The new organization will provide consulting services in the following areas:

·        Strategy

·        Biz Intelligence and Business Performance Management

·        Advanced Analytics and Optimization

·        Enterprise info management

·        Enterprise Content management

 

It was significant that the meeting was held at the Research Labs.  We lunched with researchers, met with Brenda Dietrich, VP of Research, and saw a number of solution demos that utilized intellectual property from Research.  IBM believes that its research strength will help to differentiate it from competitors.

 

The research organization is doing some interesting work in many areas of data analysis including mining blogs, sentiment analysis, and machine learning and predictive analysis.  While there are researchers on the team that are more traditional and measure success based on how many papers they publish, there are a large number that get excited about solving real problems for real customers.   Brenda Dietrich requires that each lab participate in real-world work. 

 

Look, I get excited about business analytics, it’s in my blood.  I agree that world of data is changing and companies that make the most effective use of information will come out ahead. I’ve been saying this for years.   I’m glad that IBM is taking the bull by the horns.  I like that Research is involved. 

 

It will be interesting to see how effectively IBM can take its IP and reuse it and make it scale across different customers in different industries in order to solve complex problems.  According to IBM, once a specific piece of IP is used several times, they can effectively make it work across other solutions.  On a side note, it will also be interesting to see how this IP might make its way into the Cognos Platform.  That is not the thrust of this announcement (which is more GBS centric), but is worth mentioning.  

 

Redefining Innovation?

Can people learn to innovate?  It may depend on how you define innovation.  When most people think of innovation, they think invention – like developing a post-it note. But, innovation can be something different.  Innovation can be as simple as using something old in a new way or as complex as inventing something… well really complex.  In the technology world, innovation can be about taking a product to market and extending its life.  Or, about finding new markets for technology that you already have.  

 

 

The folks at Invention Machine believe that you can learn to innovate, regardless of the type of innovation.  In fact, they believe that innovation can be injected everywhere in the product life cycle – from planning and research to design to preventing and fixing defects.  The goal is to make innovation “repeatable and sustainable”.  They have crafted an interesting solution that blends best practices in innovation with text analytics software so that the software can actually act as a subject matter expert. 

 

Best practices in innovation include text analytics

 

How does it work?  The solution- called Goldfire- provides an innovation framework with a semantic engine that allows you to mine internal documents, patents, and external literature to find answers to questions.  It supports the following innovation categories:

 

·        Analyze a market: enables the user to cull through the literature, extracting relevant information in order to understand a technology better

·        Develop a Product:  enables the user to design a new or hybrid system, providing best practices frameworks for this along with the ability to analyze patents and other documents relevant to the functionality of the product.

·        Improve a System: enables users to diagnose and fix a problem and resolve contradictions in an existing system. 

·        Risk Management: provides predictive failure analysis

·        Leverage Intellectual property: enables users to cull through patents

 

Right now, the company is focused on the manufacturing sector only.  For example, assume you are interested in developing a new product.   You would log into the Invention Machine solution and select the task (on the left hand tool bar) design a system. You would see some steps involved with designing a new system.  These are illustrated in the diagram below.

slide12

Here, you can see some best practices concerning what is involved in designing a system and then the steps you would take to do this.  One step is to explore the opportunity space.  The company uses Natural Language Processing techniques to rank concept retrieval requests relative to the question being asked. For example, say you’re interested in designing a new coffee maker and you need a component to heat water.   If you ask, “What heats water?” the result “The anode heats water” would be ranked higher than “Water is heated on a stove” because semantically, the first answer is more accurate.  Note that here the user is interested in the function of heating water and the anode is a component that can help to do this.  The text analytics engine will make use of selective substations for accurate extended concept retrieval.  Selective means only the words that are precise substitutes are used for automatic rephrasing.

 

 

The order of words makes a difference in what would come back in a search query.  Using the coffee maker example, assume the user is interested in getting power to the switch on the machine.  If the user asks, “How to power the switch?” and “How to switch the power?” the software recognizes the difference in concepts expressed by the different word orders.  This is because the semantic approach used by Invention Machine looks at the word roles and the relationships implied by the structure of language rather than a simple keyword or Bayesian approach.    

 

Can you learn to innovate?

 

I have to say that I have not seen anything quite like this before.  I’ve seen text analytics used in R&D to search the literature and patents, but not wrapped in best practices. 

Invention Machine’s semantic approach is interesting because it helps users focus on components and functions and, at one level, this is a part of what innovation is all about.  You could argue that components and functions are relevant to any industry.  If I am building something or even delivering a service, I am interested in what components I would need to meet a specific function.  Right now Invention Machines is focused on manufacturing, but you could certainly see how the application could be extended to other domains. 

Follow

Get every new post delivered to your Inbox.

Join 1,189 other followers