Run a Local AI Chatbot on Raspberry Pi 5 with Ollama

Small open models with a few billion parameters run on an 8 GB Pi 5. They are slower than cloud AI, but private and great for learning how LLMs work.

Video tutorial

▶ Ian WoottenWatch on YouTube ↗

Video by Ian Wootten, embedded from YouTube. The written steps below are our own and match the parts list.

Mission steps

1. Use a Pi 5 with 8 GB or more

   Add active cooling; generating text keeps all cores busy.

2. Install Ollama

   Run the official install script for Linux.

3. Pull a small model

   Start with a 1–3 billion parameter model. Larger models may not fit in RAM or will be very slow.

4. Chat or use the API

   Type in the terminal, or send requests to the local API on port 11434 from Python.

5. Connect it

   Use it to summarize sensor logs, answer questions about your docs, or power a voice assistant.

Code

curl -fsSL https://ollama.com/install.sh | sh
ollama run llama3.2:3b      # or another small model from ollama.com/library

import requests
r = requests.post("http://localhost:11434/api/generate",
                  json={"model": "llama3.2:3b", "prompt": "Explain PWM in one sentence.", "stream": False})
print(r.json()["response"])

Tip: test every script on the bench before installing it in a moving machine.

Questions