Natural Language Processing (NLP) is a field of Machine Learning that focuses on processing text written in human languages. Specifically, NLP algorithms enable machines to understand and process human language and documents.
Some of the key tasks in NLP involve understanding grammar, semantics, and syntax. To start this process, lemmatization and stemming convert raw text to a normalized text (e.g. converting “is” to “be” and “cities” to “city”). Further, preprocessing of text is often necessary before applying NLP algorithms. This involves removing many common words, called stop words, and tokenizing text in the form of sequences of sentences and words. Parsers play a central role in NLP by tagging words with a finite set of types. Part-of-Speech (POS) taggers specify the grammatical role of a word in a sentence and Named-Entity Recognition (NER) parsers tag words that fall within a small set of entities such as people, places, companies, dates, times, money, percentage, and geo-political monuments.
Semantic Role Labeling (SRL) breaks down a sentence into noun phrases (NP) and verb phrases (VP) by forming a semantic tree. The main challenge in modern NLP is deciphering meaning, aka Natural Language Understanding (NLU), and figuring out what the machine should “say” back, aka Natural Language Generation (NLG). Both of these require machines to learn the ambiguities of human languages and need to be trained to handle new tasks and express concepts. NLU and NLG play major roles in the creation of intelligent Chatbots with human-like conversational capabilities.
In recent years, due to the unprecedented success of Deep Learning architectures and algorithms, almost all NLP tasks can be performed with Deep Neural Networks (DNNs) (in particular, Recurrent Neural Networks (RNNs), which are ideal for modeling sequences). Word2vec algorithms, introduced by Google researchers, have tremendously contributed to the understanding of context and meaning for NLU. Word embedding algorithms and DNNs have seen great success in areas like the sentiment analysis of text and documents.
Challenge: How to effectively use complex technical manuals in real time to solve problems.
Solution: AI Software Agent assists user in natural language Q&A for desired answer.
Natural Language Understanding technology enables an intelligent software agent to do the grunt work of word by word reading, capturing the semantic meaning of the text (e.g. technical manuals) and providing instructions to an operator using real time natural language questions and answers. It has a unique capability to overcome the “how do I know what to ask?” obstacle. Its unique ‘pre read’ function gives visuals to the diversity and topic prevalence of the underlying information, and suggests starter questions, prompting a user to pursue a path of reasoning until reaching the desired answer for real time problem solving.
With the internet, ubiquitous computing, cloud platforms, and Big Data, information overload has become an undeniable day-to-day challenge for knowledge workers of all industries. With the rapid advancement of neural NLP research, can AI bring much needed relief to the vast reading load professionals face, in order to remain competitive? The aggressive race among tech giants and leading universities have continued to push the limit of Machine Reading Comprehension (MRC). In fact, the ability for machines to accurately answer questions from supporting documents recently exceeded human performance (based on the GLUE metric).
So, one might ask why fund managers are still spending an average of 90 minutes out of each busy day reading about market events. Is it finally the day of nerd utopia, when college students can pass their course exams by simply consulting their mobile app. Unfortunately, having overworked professionals relying on intelligent agents to feed them information for making critical decisions is still far from market reality. Beside the inevitable trust barrier, humans simply do not learn new subjects out of the blue without being fed the right context.
BigRio delivers a reading assistant solution that integrates the state-of-the-art MRC technology with a pre-read tool. Our MeD Watch application, developed in collaboration with some of the largest Pharmaceutical companies in the world, gives the user the ability to browse and dive into the content, resulting in an instant understanding of the diversity and prevalence of the concepts expressed in the underlying corpus. Our unique insight extraction engine predicts relevant user questions and provides a FAQ style cheat sheet to trigger a sustained exchange between the user and our conversational agent, which has ingested the text information in entirety. The dialogue represents an exercise of validation or revision in pursuit of a hypothesis generated in the user’s mind.
Our MRC technology is open-domain, in the sense that it can work on any knowledge domain with or without a light weight fine tuning step, for example, academic papers, technical manual, financial reports, etc. Our product design was fueled by the latest study in psychology with a focus on the theory of learning. With the grunt work of manual reading eliminated, we foresee a marked acceleration in personnel training, formal education, managerial decision making, and any other area of endeavor involving insight extraction from a large content source.
Semantic search plays an important role in improving the performance and efficiency of search engines in eCommerce sites. One way to improve semantic search is to extract key phrases that convey the core meanings of description and characteristics of items in retail. At BigRio, we have some of the leading experts that have developed powerful algorithms for the improvement of NLP for applications such as semantic search and computing sentiment signals at production scale. Our team has extensive expertise and years of experience in development and deployment of production scale enterprise NLP software.