How to Build a Private AI Onboarding Assistant

Starting a new job can be overwhelming, with numerous policies, procedures, and acronyms to learn. In this article, I will guide you through setting up a chatbot application that can help with your onboarding process. The approach uses tools that run locally on your computer, so your conversations will be free and private. How does that sound? You'll need a somewhat powerful machine. I did the work related to this article on a MacBook Pro with an M1 chip and 16GB of memory. We will use Retrieval-Augmented Generation (RAG) to enable the chatbot to answer company-related questions that weren’t part of the training data for the large language model (LLM) that powers it.. With the help of Claude, I created an employee handbook for the fictional company Omni Consumer Products (OCP) from the 1987 film, RoboCop. I made sure to include content in this document that wasn’t available during the LLM’s training, demonstrating that the application relies on uploaded documents.

In order to implement this application yourself, you will have to install some components on your computer including:

• Ollama
• AnythingLLM Desktop
• llama3.2

What Are These Tools?

Ollama is an open-source tool designed to run large language models (LLMs) directly on a local machine, providing users with full control over their data and enhancing privacy and security. You can download it from this page.

AnythingLLM is an open-source, all-in-one AI application developed by Mintplex Labs designed to simplify the use of large language models (LLMs) and AI agents locally. You can install AnythingLLM Desktop from this page.

The model I used for this project was llama3.2 from Meta. After you have installed Ollama, you can download this model onto your computer by following these steps:

• Start the Ollama server with this command:

ollama serve        

• From another command prompt, download the model with the command:

ollama pull llama3.2:latest        

Launch the AnythingLLM application and ensure it is running correctly.

Click the action button labeled "New Workspace". In the dialog that appears, type the name "OCP Onboarding" and save it. You will see your new workspace in the left hand panel.

Click the settings (gear) icon to the right of the workspace name and navigate to the Chatbot Settings tab on the page that appears.

In the Workspace LLM Provider drop down, change the value from the system default to: Ollama.

In the Workspace Chat model drop down, select llama3.2:latest (which may be selected automatically if it's the only model you have pulled down).

Scroll to the bottom of the page and click the "Update workspace" button. Close this page and return to the home page when the update is complete.

Click the upload icon to the right of the workspace name. The page that appears will allow you to add the OCP Employee Handbook to the project.

• Download the file from this link and upload it to the project using the "Click to upload or drag and drop" action.
• Select the uploaded file and click the Move to Workspace action. Close the window when the embedding of the document is complete.

Now you're ready to start chatting. Make sure you have the OCP Onboarding workspace selected and type your questions.

Once you've had the chance to use the app in this mode, there are some options you can experiment with...

You could try different LLMs. Search the Ollama site for models and use the "ollama pull (model name)".

You may want to experiment with a couple of values on the Chat Settings page:

• Prompt: This text box provides guidance to the model as to how it should handle prompts.
• LLM Temperature: This value drives the creativity of the model's responses. Lower values are less creative and will typically drive more consistent responses.

Conclusion

By using AnythingLLM Desktop and Ollama, you can create an RAG-enhanced chatbot capable of answering questions related to your own content. As you can imagine, this same approach could be applied to use cases outside of what is demonstrated here. For example, your uploaded documents could include information about new products, etc.

Have fun with it and good luck with the new gig!