How to Use Ollama

Article

What is Ollama?

Ollama is an open-source runtime that downloads, packages, and serves large language models on your own machine. It's the easiest way to go from "I want to try Llama" to a running model with an HTTP API in under five minutes.

Under the hood it wraps llama.cpp with smart defaults: automatic quantization picks, GPU offload detection, model caching, and an OpenAI-compatible endpoint so existing SDKs just work.

Install Ollama

macOS / Linux:

``` curl -fsSL https://ollama.com/install.sh | sh ```

Windows: Download the installer from ollama.com/download.

Docker:

``` docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama ```

Verify with ollama --version. The daemon runs in the background at http://localhost:11434.

Pull your first model

``` ollama pull llama3.3 ollama run llama3.3 "Explain HNSW indexes in two sentences." ```

First run downloads the weights (5-40 GB depending on the model). Subsequent runs hit local cache and start in seconds.

Choose the right model

| Use case | Model | Size | RAM | |---|---|---|---| | General chat | llama3.3 | 8B | 16 GB | | Coding | deepseek-coder-v3 | 33B | 32 GB | | Reasoning | deepseek-r2:32b | 32B | 32 GB | | Multilingual | qwen3 | 14B | 20 GB | | Tiny / edge | qwen3:4b | 4B | 8 GB | | Vision | llama3.3-vision | 11B | 20 GB | | Embeddings | nomic-embed-text | 137M | 2 GB |

Hardware & RAM guide

- Apple Silicon (M2/M3/M4): unified memory means RAM = VRAM. M3 Pro 36GB runs 30B models comfortably. - NVIDIA: RTX 4090 (24GB) handles 32B quantized. Two 4090s via tensor parallelism cover 70B. - CPU only: works but expect 2-10 tok/sec on 8B. Fine for batch jobs, painful for chat.

Rule of thumb: params (B) x 0.7 = GB needed for Q4 quantization. A 13B model needs ~9 GB.

The OpenAI-compatible API

Ollama exposes an OpenAI-style endpoint, which means every SDK that speaks OpenAI also speaks Ollama:

``` curl http://localhost:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "llama3.3", "messages": [{"role":"user","content":"Hello"}] }' ```

Code examples

Python (OpenAI SDK):

```python from openai import OpenAI client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama") resp = client.chat.completions.create( model="llama3.3", messages=[{"role":"user","content":"Summarize HNSW in one tweet."}] ) print(resp.choices[0].message.content) ```

TypeScript:

```typescript import OpenAI from "openai"; const ai = new OpenAI({ baseURL: "http://localhost:11434/v1", apiKey: "ollama" }); const r = await ai.chat.completions.create({ model: "llama3.3", messages: [{ role: "user", content: "Write a haiku about caches." }], }); console.log(r.choices[0].message.content); ```

Customize with Modelfiles

A Modelfile is a Dockerfile-style recipe that bakes a system prompt, parameters, or LoRA into a named model:

``` Modelfile FROM llama3.3 PARAMETER temperature 0.2 SYSTEM "You are a terse senior staff engineer. Answer in code-first style." ```

``` ollama create staff-eng -f Modelfile ollama run staff-eng "How do I debounce a React handler?" ```

Build local RAG

Pair Ollama with nomic-embed-text for embeddings and a local vector store (Chroma, LanceDB, or pgvector) for fully offline retrieval-augmented generation:

1. Embed your docs with nomic-embed-text via /v1/embeddings 2. Store vectors in chroma 3. On each query: embed, top-k, stuff into llama3.3 prompt

Production tips

- Pin model versions — pull by digest, not tag, so deploys are reproducible. - Set OLLAMA_KEEP_ALIVE=24h to keep hot models in VRAM. - Cap concurrent requests with OLLAMA_NUM_PARALLEL; default is 1 per model. - Front with nginx or Caddy for TLS, auth, and rate limiting. - Use vLLM or TGI instead of Ollama once you need >50 concurrent users — they batch better.

Troubleshooting

- "out of memory" -> pull a smaller quant: ollama pull llama3.3:8b-q4_K_M. - Slow on Mac -> make sure you're on Apple Silicon native build, not Rosetta. - GPU not detected (Linux) -> install NVIDIA Container Toolkit if using Docker; check nvidia-smi works. - Hangs on first response -> it's loading weights into VRAM. Set OLLAMA_KEEP_ALIVE to avoid the cold start.

What to build next

Now that Ollama runs locally, plug it into your editor (see best AI coding assistants), wire it into an agent loop, or self-host an internal chatbot. Privacy, zero rate limits, and zero per-token cost — that's the Ollama dividend.