Building a Local Code Assistant with Open Source LLMs: A Step-by-Step Guide

In the rapidly evolving world of software development, leveraging large language models (LLMs) like Meta's Llama3 or the code-specific Starcoder2 can greatly enhance coding efficiency and capability. This guide provides a detailed walkthrough on setting up a local development environment using Ollama, an open-source platform for managing and running LLMs on your hardware. By the end of this tutorial, you'll have a functional coding assistant integrated within your VSCode or JetBrains IDE.

Install Ollama

Ollama is an open-source application designed for running, creating, and sharing large language models (LLMs) locally on your own hardware. It started by supporting the Llama2 model and has since expanded its library to include other models like Mistral,Phi-2, Llama3 etc. Ollama allows developers to interact with these models directly via a command-line interface, offering a more secure alternative to cloud APIs by not sending data to third-party services. Additionally, Ollama supports a variety of models, which can be easily customized and managed through its simple API.

Download LLMs Using Ollama

Once Ollama is installed, you can download the desired models using the command line. For general purpose and coding tasks, you can use Llama3, and for more focused coding assistance, also you can use starcoder2.

StarCoder2 is the next generation of transparently trained open code LLMs that comes in three sizes: 3B, 7B and 15B parameters.

to download llama3 :

ollama pull llama3        

Llama3 is an exceptionally versatile model capable of handling a wide array of inquiries, whether they pertain to Python programming or other subjects.

To focus its capabilities on your objective of creating a Python Coding Assistant, it's advisable to configure Llama3 with a specific system message. This will direct the model to prioritize tasks associated with Python code completion and assistance.

Create a 𝗠𝗼𝗱𝗲𝗹𝗳𝗶𝗹𝗲 with your adjustments

FROM llama3

PARAMETER temperature 0.2

SYSTEM """You are Python coding assistant. Help me autocomplete my Python code."""

Create a new model from this Modelfile.

ollama create python-helper -f ./Modelfile

If you want to use starcoder2, then just run

ollama pull starcoder2:3b        

Integrating with Your IDE

For Visual Studio Code users:

Install the Continue extension from the VSCode extension.

Navigate to the bottom of the interface and click on the "+" sign to add a new model.

Choose 'Ollama' as your provider.

Select 'Autodetect' which will automatically populate the list with all the models you have downloaded through Ollama.

If you face any issue regarding autocompletion -

Configure the extension to connect with your local model by modifying the config file as follows:

"tabAutocompleteModel": {

"title": "Tab Autocomplete Model",

"provider": "ollama",

"model": "starcoder2:3b",

"apiBase": "http://localhost:11434"

}

Experience the Magic of Tab-Autocomplete

As you begin coding, your Python Coding Assistant will continually generate suggestions, enhancing your coding efficiency. Moreover, you can highlight any piece of code, press Command + L, and pose questions directly to your assistant, making it even easier to overcome obstacles and streamline your development process.