Lets talk about Natural Language Processing

“Handle them carefully, for words have more power than atom bombs.” — Pearl Strachan Hurd

I am fairly sure, that when Pearl Strachan said those words, she did not have text mining & analytics in mind, but nevertheless her words still ring true, especially in today’s world of conversational AI & NLP ( Natural Language Processing)!

The interest in conversational bots (chat bots) and digital virtual assistants, is growing and growing fast. Enterprises are increasingly getting eager to explore bot-enabling their business processes. Though, as with all fields of AI — conversational AI is evolving at a rapid pace, and the focus is shifting from purely NLP led chat bots, to more reliance on rich messages ( cards, and buttons in the chat interface — offering higher accuracy etc.) — a trend underlined by Wit.ai, acquired by Facebook in 2015, sunsetting their Bot Engine in early 2016. ( btw, the hybrid of rich messages + NLP has been something that we have been focused on from the start :) #shamelessplug )

But nevertheless, Natural Language Processing still remains a very pertinent and key component of any conversational AI setup, so lets talk about :

  • What is Natural Language Processing & potential uses
  • Some NLP Libraries for Developers
  • Services/ platforms/Tools to setup & use NLP for Chat bots

What is Natural Language Processing

Natural Language Processing, as the name suggests is essentially a field of computer science ( more pertinently artificial intelligence) that deals with processing human language, to be able to analyze, understand and derive information from it ( so that subsequent actions can be taken).

NLP Libraries — Open Source Libraries

There are many libraries available that make it easier for developers to build the NLP applications. Listing down a few that we have used, and some of the prominent ones :

NLP Services & Tools for Chat bots

Circling back to our original point about use of NLP for Chat bots, if you don’t want to delve into libraries & code, there are many NLP Tools and Services available.

While we have mostly used Microsoft LUIS ( Language Understanding) , some of the other popular ones are Dialogflow ( Api.ai post acquisition by Google), Amazon Lex, Wit.ai ( Facebook) etc.

Some of them might be more suited towards building chat bot systems — Amazon Lex for instance, which provides an integrated setup for multi-turn conversations, with provision for prompts for filling the relevant ‘slots’ towards fulfilling the ‘intent’. Microsoft LUIS on the other hand focuses on the classical NLP aspects of extracting the ‘intent’ and ‘entity’ from the utterance. Others like Wit.ai, Dialogflow etc. provide a combination, if you will.

Every tools/services has its own terminology, but essentially in terms of base NLP:

  • Utterance is the spoken or the typed phrase that the system receives from the user, that invokes the intent. For instance ‘Book a flight from London to New Delhi for tomorrow’.
  • Intent — The intent over here is Booking a flight.
  • Entities are the specifics of the intent, included in the utterance that help us take a particular action. For instance ‘London’ and ‘New Delhi’ are Locations ( and could be made hierarchical as $Location -> $FromLocation and $ToLocation, for instance) and ‘tomorrow’ indicates $datetime.

Depending on what the particular focus is for your solution & the relevant use cases, and also maybe what are the other tools you are already using, you can choose the relevant tool or service. I am just jotting down some observations that you might find useful:

  • Modular or tightly coupled : The degree of modularity varies across these tools. For instance with Microsoft, its a fairly modular system where Microsoft LUIS assist with NLP, all the chatbot related aspects (Fulfillment calls, channel integration etc.) sit with the Bot Framework and the Bing Speech Api converts spoke language requests into text. Amazon Lex on the other hand, has a speech recognition engine in it, that has been trained on telephony audio (8Khz sampling rate). And while Amazon Lex provides access to Enterprises Applications through Amazon Lambda, Dialogflow on the other hand allows for webhooks for fulfilment ( without or with slot filling).
  • Analytics & Monitoring : Dialogflow provides a ready dashboard which provides usage data and the NLU data in terms of intent sessions, count, exit percentage etc. Microsoft Bot Framework that integrates with an Azure App Insights ( requires configuration) for Bot and Usage analytics, and in case of Amazon Lex it can be integrated with Amazon Cloudwatch ( monitoring dashboard pre-integrated with Amazon Lex) for metrics and monitoring. Wit.ai, like all others can be integrated with third party analytics platforms.
  • Channels supported: While both Dialogflow & Microsoft ( Bot Framework) integrate with 14–15 odd channels, including your webpage, Wit.ai allows for integration with facebook messenger, your webpage and your app, and for others one would need to use third party API integration platforms like APIANT etc. Amazon Lex supports only 4 — Facebook, Kik, Twilio SMS and Slack.
  • Languages supported : While most the platforms support quite a number of languages — LUIS supports 13 languages, Dialogflow supports 15 root languages, Wit.ai supports i think 5+ key languages and Amazon Lex currently supports only US English — which might just be all that you need.
  • Ecosystem : Microsoft LUIS, Amazon Lex and Dialogflow hugely benefit from the developer and product ecosystems of their parent companies. For instance if you are already using Bot Framework, maybe LUIS is the obvious choice, or if you are on Amazon Connect for your call center then Amazon Lex is probably a no brainer.

Each tool has its own set of unique features as well — for instance Dialogflow has the option of SmallTalk to provide responses to casual conversations — so cool & wit.ai provides an element of sentiment analysis ( though rather crude and heavily dependent on the number of examples you provide) etc.

But in terms of base NLP all of them, more or less are similar, allowing for features like pre-built & custom defined entities. Whilst one has ‘list’ of entities & ‘phrase list’ ( similar but not synonyms) in Microsoft LUIS, we have essentially the same things in ‘ keywords’ and ‘free-text’ for entities in Wit.ai. Dialogflow has prebuilt agents, while Microsoft LUIS also has pre-built domain models.

At some point, I will try and have parallel implementations and see how they fair on the F1 scores ( geometric mean of precision and recall). But in the interim, do hope this has helped you gain a slightly better understanding of NLP. Do leave your comments & thoughts — including any aspects that I might have missed. I will be happy to incorporate them.

Disclaimers: The above post in no way claims any copyright to any of the images or literature presented.

Technology Enthusiast. Startup Founder. Movie Buff. Foodie. Product Manager. Mechie at heart. Artificial Intelligence. Design. Carpentry.