How to run your own local AI coding agent with Ollama and OpenCode

A step-by-step guide to running a private AI coding assistant on your own GPU. Install Ollama, choose an open model such as Qwen, tune VRAM, context length and KV cache, connect the OpenCode terminal agent, and fix the common errors.

Can you have an AI coding assistant that reads your files, edits them and runs commands, without sending a single line of code to the cloud? Yes. You need a graphics card, a free model server called Ollama, an open language model, and a free terminal agent such as OpenCode. This guide shows how to put the three together on your own computer, and how to tune the settings so the whole model fits in GPU memory, which is the difference between a fast agent and a painfully slow one.

Quick answer. Install Ollama, pull a model that fits your GPU, raise the context length and shrink the KV cache with four environment variables, then point OpenCode at http://localhost:11434/v1. Check that size_vram / size equals 1.0. That is the whole recipe. The rest of this post explains each step.

An analogy. The model is a very well-read employee, Ollama is the office where that employee works, and OpenCode is the assistant who carries your questions in, hands the employee the right documents and does the paperwork. The office must be big enough for the employee and the documents, or work slows to a crawl.

OpenCode (terminal agent)  →  HTTP API  →  Ollama (model server)  →  GPU  →  answer back to OpenCode

What you need

Item Minimum Comfortable
Operating system Linux, macOS or Windows Linux with systemd (the tuning commands below assume it)
GPU memory (VRAM) 8 GB 24 GB
System RAM 16 GB 32 GB or more
Disk space 20 GB 100 GB if you want several models
Cost Free software A GPU you already own

Your VRAM decides which models are realistic. A 24 GB card such as an RTX 4090 comfortably runs a 27B to 32B model in 4-bit form. A smaller card works too, with a smaller model and lower expectations.

1. How the three parts fit together

Part What it does Where it runs
Model A file of learned weights, for example a Qwen model Wherever Ollama runs
Ollama Loads the model into GPU memory, computes answers, exposes an OpenAI-compatible API on port 11434 Your PC or a server
OpenCode A coding agent in the terminal. Builds prompts, reads and edits files, runs shell commands, sends text to the API Your PC

Three facts are worth remembering:

  • The model runs where Ollama runs. OpenCode only sends text and receives text.
  • File edits and shell commands always happen on the machine running OpenCode, even if the model runs on another computer.
  • Your prompts and file contents travel to whichever endpoint you choose. With localhost they never leave your computer.

2. Install Ollama

On Linux:

Shell
curl -fsSL https://ollama.com/install.sh | sh
ollama --version

The installer creates and starts a background service named ollama. On macOS and Windows, download the installer from the Ollama website instead.

Check that you have a working NVIDIA driver:

Shell
nvidia-smi

If this command fails, install the proprietary NVIDIA driver first.

3. Choose and download a model

How to choose

Rule Why
The model plus its working memory must fit in VRAM A full GPU load is the biggest speed factor
The model must support tool calling Agents read files and run commands through tools. Models without reliable tool calling fail at multi-step work
Bigger models follow instructions better Very small models (7B to 8B) often lose track in multi-step agent tasks
Q4_K_M is the usual balance Q8_0 is closer to full quality but about twice the size
Reasoning models think before they answer More accurate on hard tasks, slower and heavier on tokens

What fits on which GPU

These are rough guidelines. Check real tags and sizes on the Ollama model library.

Model class Fits on 24 GB? Agent suitability
27B to 32B at Q4 (about 16 to 20 GB) Yes, with a reduced context window Good. The best quality that fits
14B at Q4 to Q8 Yes, with a large context Moderate. Fine for simpler tasks
7B to 8B Yes, with a very large context Weak for agents. Fine for chat and quick edits
70B and larger at Q4 (40 GB and up) No, part runs on the CPU Too slow on a single 24 GB card
100B and larger No Server-class hardware only

Download it

Shell
ollama pull qwen3.8:27b-mtp-q4_K_M
ollama list

This example is a Qwen 27B model in 4-bit form, about 16 GB to download. If the tag is not found, open the Ollama library and copy a current tag that matches your VRAM. Then test it:

Shell
curl -s http://localhost:11434/v1/models
ollama run qwen3.8:27b-mtp-q4_K_M "Answer in 3 words."

Reasoning models print a “Thinking…” block before the answer. That is normal.

4. Tune Ollama so the model fits in VRAM

This is the step most guides skip, and it matters most. With default settings, a 27B model on a 24 GB card ended up only 88.8% inside GPU memory. The rest sat in system RAM, and generation was slow. Four settings fixed it.

What the four settings do

Variable Meaning Effect
OLLAMA_CONTEXT_LENGTH How many tokens a request can hold: prompt, history and answer together Memory grows with it. Too small breaks agents, because the system prompt and tool list are large. Ollama’s built-in default is small, so raise it
OLLAMA_FLASH_ATTENTION=1 A memory-efficient attention method Less VRAM at long context, usually faster. Needed for a quantized KV cache
OLLAMA_KV_CACHE_TYPE=q8_0 Stores the conversation memory (the KV cache) in 8 bits instead of 16 Roughly halves that memory with a small quality loss. q4_0 quarters it, with a more visible loss
OLLAMA_NUM_PARALLEL=1 How many requests run at once Memory is reserved per slot. One slot is right for a single user

Apply them on Linux

Shell
sudo systemctl edit ollama

An editor opens. Paste these lines above the line that says “Edits below this comment will be discarded”:

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=24576"
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"
Environment="OLLAMA_NUM_PARALLEL=1"

Save (in nano: Ctrl+O, Enter, Ctrl+X), then restart:

Shell
sudo systemctl restart ollama
systemctl show ollama | grep -i Environment

Important: these lines are not shell commands. Pasting them into a terminal gives [Service]: command not found. They belong in the editor that systemctl edit opens.

On macOS and Windows, set the same variables the way your system sets environment variables, then restart Ollama.

Verify that the model is fully on the GPU

Shell
ollama run qwen3.8:27b-mtp-q4_K_M "Hi" > /dev/null
curl -s http://localhost:11434/api/ps | python3 -c "import sys,json; m=json.load(sys.stdin)['models'][0]; print(m['size_vram']/m['size'], m['context_length'])"
nvidia-smi --query-gpu=memory.used,memory.total --format=csv

The target is 1.0 24576. Anything below 1.0 means some layers run on the CPU and everything slows down.

Here is how the ratio improved on a 24 GB RTX 4090 with a 27B Q4 model:

Step Settings size_vram / size
Default install Context 32768, 16-bit KV cache 0.888
Tuning, round 1 Context 32768, flash attention, q8_0 KV cache 0.937
Tuning, round 2 Context 24576, flash attention, q8_0 KV cache, one slot 1.0

If the ratio is still below 1.0

Work down this list, one change at a time:

  1. Turn on flash attention and the q8_0 KV cache.
  2. Set OLLAMA_NUM_PARALLEL=1.
  3. Lower the context length: 32768, then 24576, then 16384.
  4. Try the q4_0 KV cache.
  5. Free GPU memory. Close browsers and video apps, or stop the graphical session for a test.
  6. Use a smaller model or a smaller quantization.

The reason is simple. VRAM must hold the model weights, the KV cache, some compute buffers, and anything else using the GPU, such as your desktop. When the total is too big, Ollama pushes layers to system RAM. The settings above shrink the KV cache until everything fits.

5. Install OpenCode and connect it

Install

Shell
curl -fsSL https://opencode.ai/install | bash
source ~/.bashrc
opencode --version

If opencode: command not found remains, open a new terminal or run ~/.opencode/bin/opencode.

Write the config

The file is ~/.config/opencode/opencode.json. Add a local provider:

json
{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "local-ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Local Ollama",
      "options": {
        "baseURL": "http://localhost:11434/v1",
        "apiKey": "ollama"
      },
      "models": {
        "qwen3.8:27b-mtp-q4_K_M": {
          "name": "Qwen 27B (local)",
          "limit": { "context": 24576, "output": 8192 },
          "tool_call": true
        }
      }
    }
  }
}

What each field means

Field Meaning
npm The library OpenCode uses to talk to the API. @ai-sdk/openai-compatible works with Ollama
baseURL The API root, including /v1
apiKey A placeholder. Ollama ignores it, but the library needs some value
Model key Must match the Ollama tag exactly. Copy it from ollama list
limit.context Must be no larger than OLLAMA_CONTEXT_LENGTH
limit.output The maximum tokens per answer
tool_call Declares tool support. Required for agent work

Three rules prevent most config errors:

  • Every provider must sit inside the "provider" object. A provider at the top level is silently ignored.
  • Do not add variants or reasoning_effort. A wrong variants value makes OpenCode drop the whole provider, and Ollama has no reasoning_effort setting.
  • Keep one config file. If both opencode.json and opencode.jsonc exist, back up and keep just one.

Edit the file safely with a script

Hand-editing JSON is easy to get wrong. This script adds the provider for you, after a backup:

Shell
cp ~/.config/opencode/opencode.json ~/.config/opencode/opencode.json.bak
python3 - <<'PY'
import json, os
p = os.path.expanduser('~/.config/opencode/opencode.json')
d = json.load(open(p))
d.setdefault('provider', {})['local-ollama'] = {
    "npm": "@ai-sdk/openai-compatible",
    "name": "Local Ollama",
    "options": {"baseURL": "http://localhost:11434/v1", "apiKey": "ollama"},
    "models": {"qwen3.8:27b-mtp-q4_K_M": {
        "name": "Qwen 27B (local)",
        "limit": {"context": 24576, "output": 8192},
        "tool_call": True}}
}
json.dump(d, open(p, 'w'), indent=2)
PY

Then check the result:

Shell
python3 -m json.tool ~/.config/opencode/opencode.json > /dev/null && echo OK
opencode models | grep -i local

6. Use your agent

Shell
cd /path/to/your/project
opencode

Inside the terminal interface:

Command Purpose
/models Choose the model. Check the provider label, because the same model name can exist on several providers
/compact Shrink the conversation when the context fills up
/exit Quit

Restart OpenCode after every config change.

Start with a small, safe task, such as “explain what this project does” or “add a docstring to this function”. Then move to edits across several files.

7. Optional: share one GPU machine with several computers

If a stronger computer is on your network, you can run Ollama there and connect OpenCode from your laptop. Only two things change.

On the GPU machine, make Ollama listen on the network, using the same override file as before:

Environment="OLLAMA_HOST=0.0.0.0:11434"

On your laptop, point the provider at that machine instead of localhost:

json
"baseURL": "http://GPU-MACHINE-ADDRESS:11434/v1"

Test it with curl -s http://GPU-MACHINE-ADDRESS:11434/v1/models.

Security warning. Ollama has no login. Anyone who can reach the port can use the model. Expose it only inside a trusted network or a VPN, and block the port from the internet with a firewall. Remember also that your prompts and file contents now cross that network.

Very large models need very large hardware. A 125B model in 8-bit form is close to 190 GB, which is server territory. Expect a minute or so for the first request, because loading such a model takes time.

8. Which setup should you choose?

Your situation Choice
Private code, offline work, everyday tasks A 27B to 32B model on your own 24 GB GPU
Hard tasks across many files, best accuracy The largest model you can run, on a shared server
GPU below 16 GB A 14B model. Expect lower reliability
No GPU A small model works but is slow. Use it to learn, not for real work

Things to keep in mind

  • Cold start. The first request loads the model into GPU memory. Agents may time out. Wait and resend. Switching models causes another cold start.
  • Context fills up. Long sessions run out of room. Use /compact or start a new session.
  • Models unload after 5 minutes by default. Set OLLAMA_KEEP_ALIVE=30m to keep them loaded longer, or -1 to keep them loaded forever.
  • Other GPU programs steal VRAM. Browsers with hardware acceleration, video and other CUDA jobs can push layers back to the CPU. Recheck the ratio after starting them.
  • Agents act on your machine. OpenCode edits files and runs shell commands. Read each tool call, especially with a model you do not know yet, and work in a project that is under Git, so every change can be undone.
  • Thinking models answer more slowly, because they reason first.

Common problems and fixes

Problem Fix
opencode: command not found after installing Run source ~/.bashrc, open a new terminal, or use ~/.opencode/bin/opencode
snap install opencode is refused Snap needs classic confinement, which removes the sandbox. Use the official installer instead
[Service]: command not found The override lines were pasted into the shell. Put them into sudo systemctl edit ollama
The local provider does not appear in /models The block is outside "provider", the file was not saved, the JSON is invalid, or OpenCode was not restarted
Only some providers appear Check with grep -c local-ollama ~/.config/opencode/opencode.json and add the block with the script above
size_vram / size is below 1.0 Follow the tuning list in section 4
The first request is slow and the agent gives up It is a cold start. Wait, then resend
Connection refused to a remote machine Wrong address, the machine is not reachable from your network or VPN, or the port is blocked
Model name not found A typo in the tag. Copy the exact name from ollama list. The : and _ count
Answers cut off, or errors in long chats The context is full or limit.context is larger than the server setting. Use /compact and make the two values match
Generation is slow on your own GPU Part of the model runs on the CPU, or another program uses the GPU. Check /api/ps and nvidia-smi

Command cheat sheet

Command Purpose
ollama pull <model> Download a model
ollama list List installed models
ollama run <model> "prompt" Run a model for one question or interactively
ollama ps Show loaded models and the GPU/CPU split
ollama rm <model> Delete a model
sudo systemctl restart ollama Apply new settings
journalctl -u ollama -f Follow the server log
watch -n 1 nvidia-smi Watch VRAM live during a request
opencode models List the models OpenCode knows from your config

Frequently asked questions

Is a local AI coding agent really private? With localhost, prompts and files stay on your computer. If you point the agent at a remote server or a cloud service, the text you send goes there, so check the provider name before you start.

How much VRAM do I need to run a local LLM? Plan for the model file size plus several gigabytes for the KV cache and your desktop. A 27B model in Q4 form needs about 19 GB for the weights alone, so a 24 GB card is comfortable and a 16 GB card needs a smaller model.

What is the KV cache? It is the model’s short-term memory of the conversation. It grows with the context length, and it can take several gigabytes. Quantizing it to 8 bits halves that memory.

Why is my local model slow? The most common cause is that part of the model does not fit in VRAM and runs on the CPU. Check size_vram / size. It should be 1.0.

Is a local model as good as a cloud model? Not at the very top end. A good open 27B to 32B model handles everyday coding tasks well, and the largest cloud models are stronger on the hardest ones. Local models trade some accuracy for privacy, no per-token cost and offline use.

Which is better, Ollama or OpenCode? They do different jobs. Ollama serves the model, and OpenCode is the agent that uses it. You need both, or an alternative for each.

Further reading

Try it yourself

  1. Install Ollama, pull a model that fits your GPU, and ask it one question with ollama run.
  2. Run the verification command and write down your size_vram / size value before any tuning.
  3. Apply the four settings, restart Ollama and compare the new ratio.
  4. Connect OpenCode to your local model and ask it to explain a small project.
  5. Lower the context length by one step and see how the memory use in nvidia-smi changes.

Basheer Al-Tawil is a PhD researcher in Robotics and AI at Otto von Guericke University. Found a mistake or have a question? Write to me.

Browse by topic