Ever wanted to run a powerful language model locally on your own PC? Good news — with open-source tools and Local AI models, it’s totally possible. In this easy tutorial, we’ll walk through how to run Gemma 3, an open source LLM, on a Windows computer using Kolosal AI. Kolosal AI is a lightweight app (only ~20 MB) that lets you run Local LLM models with no internet required. That means no cloud needed (no subscriptions or server fees) — everything runs offline on your device, keeping your data private and secure. Let’s get started and have your own local AI up and running in just four steps!
1. Download Kolosal AI
First, grab the Kolosal AI application. Head to https://kolosal.ai and find the download section. There’s a big “Download for Windows” button — click that to download the Kolosal AI installer (an
.exe file) for Windows. The download is small (around 20 MB) and Kolosal AI supports both CPU and GPU out of the box, so it should work on most Windows PCs.Once the installer file (
KolosalAI_Installer.exe) is downloaded, you're ready for the next step.2. Install Kolosal AI on Windows
Now, run the installer you just downloaded. Double-click the
KolosalAI_Installer.exe file and proceed through the installation wizard. If Windows shows a SmartScreen warning (because Kolosal is a new app), click “More info” then “Run anyway” – the installer is safe to run (Kolosal AI is open-source and trusted). Follow the prompts to install Kolosal AI like any other program (choose your install location, then click Next/Finish). It’s a quick install.Once installation is complete, you’ll have Kolosal AI installed on your system.
3. Open Kolosal AI
Time to launch the app! Find Kolosal AI in your Start menu (or on your Desktop if you created a shortcut) and open it up. On first launch, you’ll be greeted by a clean, developer-friendly interface. The window is divided into sections: a sidebar for chat sessions, a main area for the chat conversation, and a right-hand panel with model settings and prompts. Initially, no model is loaded yet — the chat area will be empty, waiting for a model to be loaded.
Notice the interface is ready for chat, but we need to load a language model to start using it. Kolosal AI makes this easy through its built-in Model Manager. We’ll use it to download the Gemma 3 model in the next step.
Now for the most important part — getting the Gemma 3 model itself. Kolosal AI includes a convenient Model Manager that lets you download open-source LLMs with one click. Here’s how to add Gemma 3:
- Open the Model Manager: In the Kolosal AI app, look for a Select Model button or menu option. (In the bottom, within the text input area) Click that, and a window or panel will show a list of available models.
- Find Gemma 3: In the list of models, locate Gemma 3. (Models might be grouped by provider; Gemma is an open model by Google, so check under any Google category.) You might see multiple Gemma variants — for example, Gemma 3 1B (a 1-billion-parameter model) or larger ones. We recommend starting with the Gemma 3 1B model for local use.
- Download the model: Click the Download button next to Gemma 3 (1B). Kolosal AI will begin downloading the model files. This may take a few minutes depending on your internet speed, as the model can be a few gigabytes or less if it’s a quantized version. You’ll see a progress indicator during the download.
- Load the model: Once the download finishes, Kolosal AI will automatically load Gemma 3 into memory. After a few seconds, you should see an indication in the app that the Gemma 3 model is loaded and ready to use. For example, the model’s name might appear at the bottom status bar (e.g., “Gemma 3 1B loaded”). If it doesn’t load automatically, simply select Gemma 3 from the model list to activate it.
Great — you now have the Gemma 3 LLM running locally on your machine! 🎉 At this point, you can start chatting with it in the Kolosal AI interface just like you would with an online AI chatbot, except this one is running on your PC (no internet required).
Start chatting with Your Local LLM
Now that Gemma 3 is up and running, let’s try a quick demo to see it in action. In the Kolosal AI chat interface, you can type a prompt (question or task) and hit Enter. The model will generate a response right before your eyes, entirely offline.
Example Prompt: “ Suggest a unique pasta recipe using mushrooms and lemon.” 🍝
Imagine we ask Gemma 3 this question in the chat. After a moment, the model will respond with a creative idea. For instance, Gemma 3 might output something like:
Gemma 3 :“Sure! How about a zesty Mushroom Lemon Herb Pasta? Start by sautéing a mix of mushrooms (cremini and shiitake work great) in olive oil with garlic. In a separate pot, cook your favorite pasta (like linguine) until al dente. For the sauce, combine fresh lemon juice, a bit of lemon zest, and a splash of cream or coconut milk for richness. Toss the pasta with the sautéed mushrooms, the lemony sauce, and a handful of chopped fresh basil and parsley for an herbal kick. Finish it off with grated parmesan and a pinch of black pepper. You’ll have a bright, tangy mushroom-lemon pasta that’s truly unique!”
As you can see, the Local AI model comes up with a pretty cool recipe idea! You can now experiment with any prompt you like — ask questions, request ideas, or have a casual conversation. Gemma 3 will do its best to assist, all without using the internet.
Congratulations, you’ve successfully set up an open source LLM locally on your Windows PC in just four steps. 🎉 You now have your very own AI model (Gemma 3) running with Kolosal AI. Not only do you get the convenience of an AI assistant on your desktop, but you also maintain full control and privacy since everything is processed right on your machine.
Feel free to explore more models in Kolosal AI’s Model Manager. Gemma 3 is just one example — Kolosal supports many popular open-source LLMs like Mistral, LLaMA, Qwen, and others. With a few clicks, you can download and try out different models to see what each is capable of. The platform is very flexible, allowing you to switch models or even run an LLM server on your PC for other apps to use, all 100% local.
We hope this tutorial was helpful. Now go ahead and have fun with your Local LLM! Ask it questions, get creative ideas, and enjoy the power of Local AI at your fingertips. Happy experimenting with Kolosal AI and open-source models! 🚀