How do you use natural language processing (NLP) to generate insights from unstructured text data?

In defining the goal to generate insights from unstructured text data it's important to determine the intended audience. Insights from the same medical record could be very useful to a doctor, but "foreign" to a patient, depending on how technical the language of the insight is.

Begin by cleaning the text—removing punctuation, converting to lowercase, and eliminating stop words (common words like 'the' and 'is'). Next, tokenize the text, breaking it into individual words or phrases. Use lemmatization to reduce words to their base form (e.g., 'running' becomes 'run'). For example, in analyzing customer reviews, these steps clarify the sentiment by focusing on essential words. Finally, convert text to numerical data using methods like TF-IDF (Term Frequency-Inverse Document Frequency), which highlights important words. Preprocessing text makes it ready for further analysis and insight extraction.

Tokenization and normalization are critical in text preprocessing. For example, while longer sentences may be considered more important in most documents, in medical records, shorter sentences such as lines in test results, may carry more significance. Sentence and word embeddings are also a critical part of NLP algorithms. While they are not technically part of NLP preprocessing, they are often used in conjunction with preprocessing to facilitate NLP summarization. Sentence and word embeddings involve representing the text in a mathematical / vector format that can be processed by NLP algorithms.

Modern NLP models, such as ChatGPT, generate valuable insights from an unstructured text via extractive summarization through the clustering of output embeddings from pre-trained deep learning models. While GPT-3, GPT-4, PaLM, BERT, T5 and other transformer-based models, took the world by storm, it's important to remember that there are other, less complex NLP models, that may do an excellent job of unstructured text processing, with much less computing power. For example, Frequency-Based Sentence Scoring methods use information theory to assign each sentence of the input with a score that is based on relative frequencies. A high value for a sentence means that its content is likely to be informative.

Visualization can be a powerful tool in generating insights from unstructured text because it can help to communicate the results of the summarization process to users clearly and concisely. By using visualization techniques, NLP practitioners can help to make the results of their algorithms more accessible and useful to a wider audience. For example, WellAI's Medical Interactive Concept Cloud tool has been helping ER physicians and other medical professionals summarize patients' medical records, which is the ultimate unstructured text. Not only the tool allows understanding of a patient's medical history in seconds, but it also provides a link of the patient's medical conditions to an ICD, CPT or UMLS medical code for billing.

It's important to note that most modern NLP algorithms and packages are implementing most, if not all, of the steps of processing an unstructured text, such as preprocessing, sentence and word embedding, and summarization. Python has a variety of text summarization algorithms, from simple and efficient such as Gensim and Sumy, to more powerful but more computationally expensive such as T5, GPT-3 and GPT-4.