NLP made simple...

I am a learner. I am a kid at this.Well, I could say beginner but if you know me, you would accept the word kid without any thought.I still do write "hello world" or "hello Lakshmi" as my first program in any project no matter what language is used..and I get really happy happy for rest of the week if it worked!! Thus, I am keeping this article as well simple so it is understood by everyone. NLP which is natural language processing is not be mistaken as "Neural Linguistic programming" in our digital era. Neural Linguistic Programming is more part of healing and used as a therapeutic purposes.

Natural Language Processing deals with "natural language" of humans. Our languages, what we speak, write as a language. What we try to convey is analyzed by a bunch of nuts-bolts so that the computer can understand it like another human. Let me give an example - You tell your phone " I need coffee". It is not a command like we used to give to a computer program like "print", "copy file A to B" etc., It is more in a lingual format - "I need a coffee".

Now, computer if it is not smart enough, it will not do anything. But your phone being a 'smart' phone understands that -

I = You the person speaking, need - i.e. want, Coffee - something. It also tries to analyze on what it should do when such a statement is heard. I am jumping ahead. Let me step back.

Now, we know NLP deals with human speaking language. Ah!! okay.. NLP has a brother (I love brothers) "NLU" - Natural Language Understanding. NLP can deal with human language but NLU is for "speech" or "speaking" mode. We will stick to NLP here though I am using a speech command as an example. It is easy to explain.

NLP sits on top of lot of algorithms written for machine learning and advanced deep learning purposes. The algorithms were used initially for data processing and by data scientists. They still are. The algorithms have different purposes and depending on what we want to achieve, we use them, tweak them and so on.

I would take a puppy training as an example but I know babies better than puppies. So, when a baby is still learning, it understands to recognize the voice of mom, dad and each one separately. It also understood then what you meant by saying "No" and "Yes". Isnt it? It took the word "No" saw that when you say "No", probably it had a result A, vs. when you said "Yes". It also starts learning the tone of your voice.It knows a tone of time-out from the tone of 'love you'. It knows when to googoo at you and when to scream at you. Our human mind is one of the smartest machines in the world. We process the information from different sensory and other inputs billions of them per second to arrive at a conclusion a spontaneous one too!!

Similarly, think of a program which can process all the words we use, say English to keep it simple, knows what is it we are communicating. That - it achieves by bunch of trained NLP algorithms. Algorithm is fed training data so that it can become more and more accurate. Some are not required to be trained with specific data inputs but can learn on their own over time. Like how a teenager evolves to ignore you when you say "No" or sulk their way to room, the algorithm starts to understand or process human behavior.

We initially saw these used in our PhD labs and by scientists. Lot of these logic has been around for more than 30-40 decades even.So if you are trying to start on NLP, we have tons of algorithm code out there. The trick is to understand what they are for, how to use them, tweak them and customize them.

Also, most of the NLP solutions /algorithms out there had been used for literature and for news industry primarily. It was one of the most common use-cases in the world. Then when medicine evolved, we found that algorithms were customized and tweaked to understand the medical terms or say "lingo". This is important - the word shot in medical term means an injection while in normal usage it means different things - Tequila shot , or a gun shot etc.,

How do we learn new words? We use a dictionary? Yes. So for NLP to work, we need dictionaries to be built or used if already available. General language dictionaries you would get as a result of all those scholars and scientists. Medical dictionaries are also available due to the adoption by medical community and also that the medical terminologies need to be universal across.

In NLP, these are also called "Ontologies", they do not just have words and meanings but also will help us understand what are the related words. I will not go more in to it. Just understand it is all just big bag of words. We want our program to understand the meaning of the word so that the response is correct.

Coming to our command "I need a coffee".

NLP does simple things we take for granted with our human brain

• It breaks it into each word, if required letters. Let us stick to words for our purpose. So, command becomes "I", "need", "a", "coffee".
• Now, it tries to see what is "I" in terms of rules of the language - that is our most dreaded grammar. "I" is identified as "subject and also a noun", "need" - could be a verb or noun, "a" - article, "Coffee" is again a noun maybe.
• Now program thinks like our own brain does - what is the word "need" stand for? It looks at words near by - "I" clearly a subject, so it can still be a noun ???? and it sees "coffee" - figures out that in this command - subject wants a coffee. so need is a verb.
• Now, the rest of the processing of whole command is similar to what we saw in our bot discussions. Intent is "need a coffee", so the program maps it to the action "get near coffee shops" or "get recipe of coffee" then forms a response back to speaker that is You.

So, what happens can now be given some algorithm names to sound sophisticated

Tokenize - break it into words ? or letters?

Parts - of - speech Tagging - Which is a verb, which is a noun...

Stop words - actually a nice term for those words or letters which you really have no use of. You can use this for your benefit when you go deeper. You can ignore a sentence or paragraph based on these. How our brains have got trained to ignore millionth call for water post 11 pm? or "mom... mommmmmmmm....daad.........daddddddddddd". These are stop words in our brain!!!

and so on... If you are lazy to read a book for your class essay, you can just get e-version of the book, feed it to NLP and ask it to write it for you. If you can do this you will be working for one of those MIB organizations if you know what I mean!! It seems easy when you hear it, but to teach the program to do it is another story. But when it actually does it, you will so fall in love with it. You will be smirking at smart phones for a day atleast. It is like thinking your bike or car understands you... (as of today - No! it does not.. I am sorry to break the bubble!).

Every step of the processing - Split, Identify, Tag, Patternize, Analyze, Decipher, Merge, Infer, Summarize, Learn, Remember, Predict, Test, Repeat. I used my own words but different terms for different steps exist as per the purpose. Each have a backbone of mathematical models, logical derivations etc., It is good to learn from scratch but NLP on the go requires both "learning by reverse engineering" and also "starting from basics".

I am taking one more small step forward - I mentioned 'summarize'. Summarizers are used predominantly in news world where you have feeds from various sources around the world coming in. There should be a program, which identifies which news is on which 'topic', what are they 'salient points' and what is so 'frequently mentioned' in the feed. Once it gets all these, it does what we do to summarize - read through the whole thing, pick up only the gist of the matter, the start, end and crucial twist in the story. Limit the whole to say 300 words as given as a requirement. That is exactly what these news summarizers do.

Some parts of it is also done by our beloved Google Search, it goes fetches information, ranks it based on relevance and gives it.

So... for this part, we know what NLP and NLU stand for, what goes on at a high level, what is a summarizer. I will let you explore the world till next article calls for me on the topic.

Parting thought - no machine program as of now has reached the accuracy in summarization or output that a human brain could give on processing information. So - give yourself a high-five.

(Disclaimer - if you are an NLP expert, excuse the layman lingo and simplicity of this without the calculus, statistics and data science. I get nightmares when I go there!!)