Building EnterpriseGPT

Applying lessons from cloud computing to AI

Artificial Intelligence (AI) is no longer a future prospect but a present-day force fundamentally reshaping enterprise operations. As organizations navigate this transformation, developing effective, secure, and tailored AI solutions requires a deep understanding of its core components—from data strategy and management to model selection and deployment—and strategic decisions about infrastructure control.

The current trajectory of AI adoption mirrors the evolution of cloud computing. Initially, enterprises approached the cloud with caution, wary of vendor lock-in, security vulnerabilities, and compliance challenges. This rational skepticism stemmed from the inherent risks of ceding control over critical assets.

Driven by competitive pressures and innovation, cloud migration accelerated, only to reveal new complexities—particularly around cost management. Unpredictable expenses led many organizations to re-evaluate, resulting in cloud repatriation efforts aimed at regaining control over spending, performance, and security.

AI is now traversing a similar path. Enterprises are eagerly integrating AI capabilities, often leveraging third-party platforms and public APIs. While this accelerates adoption, it simultaneously surfaces significant concerns regarding data sovereignty, security, and long-term financial viability.

Learning from the cloud experience, forward-thinking organizations recognize the strategic value of building internal, controlled AI infrastructure—an "Enterprise GPT." Such an approach aligns AI capabilities with specific business needs, integrates seamlessly with proprietary data, and prioritizes governance and compliance.

The Foundation: Data as the Engine of AI

Building such a controlled infrastructure starts with mastering the fundamental fuel of AI: data. Modern AI, particularly Large Language Models (LLMs), relies on massive datasets to learn intricate patterns. The challenge lies in harnessing the petabytes of data generated daily—much of it unstructured and underutilized. Generative AI provides powerful tools to extract insights, elevating data's strategic value. This underscores the critical importance of diverse, high-quality data sources for training robust and differentiated AI models.

Preparing the Fuel: Data Processing and Labeling

As my friend Aaron Fulkerson, CEO of Opaque, and my co-host for the AI Confidential Podcast notes, “Data in the age of AI isn’t an advantage—your data will be your only advantage.” Protecting data sovereignty and keeping it private for your use is a core need in AI implementations. But before you can do that, you need to make your data usable.

Raw data requires meticulous preparation before it can power AI models. This involves cleaning, structuring, and contextualization, often through data labeling, which provides the necessary ground truth for effective learning. Effective data utilization hinges on robust processing and labeling workflows, supported by various tools:

Tools for Processing and Labeling Data:

Storing the Data: The Role of Vector Databases

Traditional databases struggle with the high-dimensional data representations central to AI. Storing and querying the resulting vector embeddings requires specialized databases. Vector databases are designed to index and retrieve these vectors, which capture semantic meaning, enabling essential AI tasks like similarity search, recommendation, and retrieval-augmented generation (RAG) across unstructured data.

Vector Database Options:

Learning Paradigms: ML, Deep Learning, and Model Training

Understanding the different approaches to machine learning is key to selecting and applying the right techniques:

• Machine Learning (ML): The foundational field where algorithms learn from data to make predictions or decisions.
• Deep Learning: A subset of ML using multi-layered neural networks to model complex patterns effectively.
• Learning Approaches:

Generative AI: Creating New Possibilities

Generative AI models represent a significant leap, capable of creating novel content. You’ll need to understand the core vocabulary to have meaningful conversations. Here are key operational concepts to understand (I highly recommend you ask your favorite model to help you get a more thorough understanding, or review the presentation I referenced):

• Training: Optimizing model parameters using data, often involving broad pretraining followed by specific fine-tuning.
• Distillation: Creating smaller, efficient models that mimic larger ones.
• Inference: Deploying trained models to generate outputs from new inputs.
• Tokens & Context Window: Understanding how models process information (tokens) and their capacity (context window) is crucial for application design.
• Feedback Loops: Implementing mechanisms for continuous model improvement based on performance and user interaction.

The Strategy: Open Source AI and Infrastructure Control

While proprietary models offer ease of access, the open source AI ecosystem provides compelling advantages for enterprises prioritizing control, customization, and long-term strategy. Concerns regarding data privacy, model transparency, and cost predictability drive significant interest in open source alternatives and frameworks (PyTorch, TensorFlow, Hugging Face).

Notable Open Source & Enterprise LLMs for Consideration:

These models, combined with the broader ecosystem, form the building blocks for bespoke Enterprise GPT solutions. Tools like Ollama further simplify running many of these open source models locally for development and experimentation. Emerging AI Agent frameworks also enhance the ability to orchestrate complex tasks using these components.

Here’s a discussion of the models from our AIE Network AI CIO author, John M. Willis on IBM’s Mixture of Experts podcast. He provides excellent insight into how these models differ in focus and application.

AI Agent Frameworks for Orchestration:

Bringing these elements together, a conceptual architecture for an "Enterprise GPT Stack" might resemble the following configuration:

• User Interface: Chatbots (like ChatGPT, internal tools, application integrations).
• Orchestration: Frameworks like LangChain, LlamaIndex, or agent frameworks (CrewAI, AutoGen) to manage workflows, prompt engineering, and tool usage.
• Models: Selection of foundation models (proprietary APIs or hosted open source like those above), potentially fine-tuned on enterprise data. Includes embedding models for RAG.
• Vector Database: Chosen from options like Milvus, Weaviate, Qdrant, Chroma, etc., for storing and retrieving relevant data embeddings.
• Data Sources: Internal knowledge bases, documents, databases, and real-time data feeds.
• Infrastructure: Cloud platforms (AWS, Azure, GCP) or on-premises hardware, potentially utilizing container orchestration (Kubernetes) and ML Ops tooling.

Ultimately, controlling and deeply understanding the AI infrastructure—from data pipelines to model deployment—is not merely operational but a strategic imperative. It empowers enterprises to build unique, defensible competitive advantages by tailoring AI to proprietary data and core processes.

This control fosters agility, enabling faster innovation cycles than reliance on external providers allows. While hybrid approaches leveraging both proprietary and open source elements may be practical, building internal capabilities and maintaining infrastructure control facilitates optimized costs, enhances security, ensures compliance, and enables the creation of valuable intellectual property.

Strategically leveraging open source tools alongside robust data governance allows enterprises to construct powerful, customized AI capabilities, securing greater control over their technological future and competitive positioning.