From the course: Responsible Generative AI and Local LLMs

Unlock this course with a free trial

Join today to access over 24,800 courses taught by industry experts.

Deploying LLMs with Lorax and SkyPilot

Deploying LLMs with Lorax and SkyPilot

From the course: Responsible Generative AI and Local LLMs

Start my 1-month free trial Buy for my team

Deploying LLMs with Lorax and SkyPilot

“

- [Instructor] One of the big problems that you face when you're working with large language models is how do you deploy those? And one of the solutions here is LoRAX. And if we take a look at the documentation here from Predibase, it's actually surprisingly simple. You can see that you can run a container with a base large language model. In this case we're going to do Mistral and then you could choose the GPUs. And then that's it. You're really ready to go. And you can see here there's some examples of how to prompt it. So this is pretty compelling. And what's also nice about this is it's actually built in Rust and it's got great inference performance. Now, I think one of the easier ways to deploy this is to actually use a cloud-based GPU. So if we go to SkyPilot here, right, and we click on it, all we have to do is do a PIP install and then make sure that it's ready to go. And then it's actually surprisingly simple because all you need to do is actually paste in a YAML file. This…

Contents