The Rise of Reasoning in Generative AI
OpenAI’s New Strawberry Models: Are LLMs Finally Reasoning? Implications for Private Information, Personal Data and Trade Secrets.
Recently, I had the opportunity to deliver a talk titled 'Why Graph Technology for AI Reasoning' (linkedin.com/pulse/why-graph-technology-ai-reasoning-diogo-braga-5uaif/) at the Data Science Conference DACH—shout out to the organizers for such an amazing event!
My presentation explored how graphs can enhance large language models (LLMs), particularly by improving contextualization, reducing bias, and maintaining content quality when dealing with private and sensitive data—a critical concern for any project handling proprietary information. I view knowledge graphs as a 'Digital Twin of Business Expertise'—a structure that digitally mirrors both the data landscape and the institutional knowledge held by human experts, ensuring its preservation for future innovation.
Following my presentation, I received two particularly thought-provoking questions that I believe highlight the current challenges and opportunities in leveraging LLMs for private and sensitive data. Let’s explore these questions in detail.
Why Graphs Are Worth the Investment
After my talk, an audience member raised a question:
'Are graphs too expensive to justify their use with Generative AI, given their significant development effort?'.
This sparked further exploration on my part, leading me to the conclusion that while graphs may require higher initial investment, their value in a RAG (Retrieval-Augmented Generation) framework is undeniable. Graphs enable reasoning within LLM-based systems, offering benefits that simpler approaches cannot match.
There is ongoing debate about whether LLMs, such as OpenAI’s latest Strawberry models (openai.com/o1/), have reached true reasoning capabilities. These models are solving increasingly complex problems—problems that many humans struggle to comprehend—covering fields like physics, mathematics, and science. As a result, they will undoubtedly stand out as one of the most impactful libraries globally, providing access to a wealth of knowledge through a unified platform. This commitment to public open-source not only democratizes information but also empowers researchers and innovators to discover more effectively. While these models may still stumble on seemingly simple tasks (e.g., distinguishing whether 9.11 is bigger than 9.8), the trajectory is clear: they will get there. Yet here is the key challenge: this form of reasoning is built on vast amounts of public, open-source data.
So, 'How do we leverage such models when dealing with sensitive, proprietary, or personal information?'.
This is where data management and architecture take precedence. Before LLMs even enter the equation, organizations must structure their data carefully, capturing nuanced concepts that machine learning algorithms can then apply effectively. Graphs excel in this area.
While vector-only RAG approaches are useful for leveraging LLMs in private projects, ensuring that confidential information stays secure, they have limitations. Graphs, while potentially requiring more effort to develop, go beyond vectors by capturing intricate, interconnected relationships within data. This ability to model complex relationships makes graphs a long-term investment for organizations aiming to continuously innovate and protect their intellectual property.
It is important to recognize that both vector-based and graph-based RAG approaches were designed to facilitate the use of LLM technology with private data—whether personal information, trade secrets, or intellectual property. Fine-tuning LLMs with internal data is an option, but it is not yet widespread—and it does not address the 'black-box' issue, where models provide answers without transparent reasoning. RAG pipelines, on the other hand, allow us to leverage LLM capabilities without exposing sensitive data, and when enhanced with graphs, they offer a more explainable and insightful path forward through their structured representation of data.
Let’s go back to: 'Are graphs too expensive?'.
It depends on your organization’s strategy. For institutions that prioritize innovation and seek to maintain human-domain expertise as a competitive advantage, graphs represent a smart, long-term investment. They ensure continuous digital advancement while safeguarding proprietary knowledge.
Graphs as the Digital Twin of Business Expertise
During the Q&A, another audience member proposed a similar concept but focused solely on vectors in the RAG framework, leaving graphs out of the equation. This vector-only approach is effective when retrieving answers from unstructured data, like text, and providing direct references to source documents. However, I see a significant limitation in this strategy: it does not account for the reasoning that comes from human domain expertise and the structured data within an organization ecosystem.
Human expertise often is not documented—it lives in the minds of specialists and in the collective experience of an organization. When relying solely on vector-only RAG, we risk generating 'hallucinations'—answers that sound plausible but are factually incorrect. This happens because models, when used without graphs, lack the ability to interpret the deeper, conceptual relationships between data points. They capture patterns but miss the context and interconnectedness that human knowledge brings.
So, how do we address this?
'How do we prevent hallucinations and automate the verification and contextualization of human expertise?'.
The answer lies in knowledge graphs. By consolidating a wide variety of data types and relationships, graphs preserve the complexity and nuance of relevant information. They excel at representing multi-scale effects, heterogeneous data, and intricate interactions between concepts—all critical for reasoning. Explore further on: Let Your Graph Do the Talking: Encoding Structured Data for LLMs (arxiv.org/abs/2402.05862).
Here is the key point: graphs can encode not only structured data but also human expertise. They capture the “why” and “how” behind decisions, forming an evolving repository of knowledge within an organization. By leveraging graphs, we build AI systems that do not just retrieve information—they reason through it, understanding complex relationships in ways that go beyond surface-level patterns.
While the initial effort of implementing graphs might seem high, the long-term value of preserving and leveraging domain expertise is immense. Graphs ensure that an organization’s critical knowledge is not lost but continuously refined and expanded. In short, they are the backbone for AI systems that aim to reason, not just respond.
Recommended by LinkedIn
LLMs Will Augment, Not Replace, Databases
One crucial point to emphasize is that while LLMs are powerful tools, they are still just that—language models, not databases. LLMs will augment how we access, interpret, and generate insights from text, but structured data will always remain foundational to how organizations operate. These models excel at processing text—unstructured data, yet structured data is critical for reasoning and decision-making. Graphs are key in ensuring that we can draw meaningful connections between both types of data.
During a special session on graph technology at the Vienna Data Science Group (meetup.com/vienna-data-science-group-meetup/events/302179767/), I had the pleasure of having an enlightening discussion with Andreas Kollegger , one of the foremost experts in this field. From this exchange, I drew several key takeaways.
The new OpenAI Strawberry models are beginning to integrate reasoning capabilities, a step closer to what we aim for when building AI systems using a graph-based knowledge base in a RAG framework. However, it is important to recognize that LLMs are not a substitute for databases. LLMs are designed to interpret and reason over data we provide, but they lack the intrinsic data management capabilities of databases, such as state tracking, historical context, efficient query handling, etc—especially with proprietary or sensitive data.
In a database, the state of the data is preserved, allowing us to retrieve, update, and manage it dynamically. Databases offer capabilities beyond what LLMs can handle, and these are critical when working with structured, often private, data that underpins many of our business processes. Graph databases, in particular, allow us to capture intricate relationships and contextual dependencies, making them a valuable tool for reasoning, something LLMs can then augment.
It is easy to confuse the capabilities of LLMs with the role of databases. LLMs can reason over the data we give them, but that reasoning only works when the underlying data is well-structured and connected. This is where graph databases excel, enabling AI systems to navigate complex, interconnected data environments. LLMs might optimize processes, but the structure and logic behind those processes remain grounded in how we manage our data.
Therefore, data preparation remains one of the most critical challenges in digital transformation, particularly when integrating LLMs. Though LLMs are adept at handling unstructured data like text, structured data continues to play a significant role in determining the accuracy and reliability of the solutions we create. Combining the power of LLMs with the reasoning capabilities of graph databases offers a promising path forward for organizations aiming to leverage both structured and unstructured data.
Proven Applications of Graph-based RAG
Financial Information Extraction
In collaboration between NVIDIA and BlackRock, a HybridRAG system—integrating vector-based retrieval with knowledge graphs—was applied to financial documents such as earnings call transcripts. This combination improved both retrieval accuracy and context relevance by capturing the domain-specific language and relationships within financial data. The result was more accurate and contextually relevant answers, outperforming both traditional VectorRAG and GraphRAG individually.
Key metrics that were highlighted include:
Source: HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction (arxiv.org/abs/2408.04948)
Customer Service
At LinkedIn, a GraphRAG system was deployed for customer service question-answering, significantly improving retrieval accuracy by incorporating the structure of past issue tickets into a knowledge graph. By retaining intra- and inter-issue relationships, this method improved both retrieval performance and the quality of the generated answers. The system reduced median resolution time for customer issues by 28.6%, highlighting the effectiveness of graph-based retrieval in operational contexts.
Source: Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering (arxiv.org/abs/2404.17723)
Broader Applications and Future Directions
A broader survey of GraphRAG applications highlights its utility across various industries such as healthcare, finance, and customer service. GraphRAG’s ability to leverage relational knowledge from graphs allows for better contextual understanding and reduced 'hallucination' in LLM outputs.
GraphRAG’s framework, consisting of Graph-Based Indexing, Graph-Guided Retrieval, and Graph-Enhanced Generation, offers structured and relational data retrieval, making it suitable for domains requiring high accuracy and domain-specific knowledge extraction. As industries adopt more complex AI-driven solutions, the use of GraphRAG can address limitations in purely text-based retrieval, offering deeper insights through its structured, relational data handling capabilities.
Source: Graph Retrieval-Augmented Generation: A Survey (arxiv.org/abs/2408.08921)
Final Thoughts: Explainability, Reasoning, and Responsible AI
In closing, while vector-only approaches in RAG pipelines are highly effective for unstructured data, graphs offer a layer of reasoning, especially when human expertise and complex relationships of our structured data come into play.
Beyond reasoning, explainability and interpretability are essential for ensuring responsible AI use, especially where critical decisions are made. In fields like healthcare, drug discovery, and law, AI must not only deliver accurate results but also provide transparent reasoning, avoiding the pitfalls of 'black-box' models. For instance, in healthcare, clear explanations are vital in medical diagnostics to maintain trust and ensure patient safety. In drug discovery, understanding the relationships between biological entities can lead to breakthroughs, but only if the system's decision-making process is interpretable. Similarly, in legal systems, transparency in AI-driven case analysis is necessary to safeguard justice and ethical standards.
To unlock the potential of Generative AI within private and sensitive environments, organizations must look beyond the initial effort and focus on the long-term benefits. Graphs allow us to represent, store, and reason with human knowledge, making them indispensable in the evolving landscape of AI reasoning.