How to run your own local AI coding agent with Ollama and OpenCode
A step-by-step guide to running a private AI coding assistant on your own GPU. Install Ollama, choose an open model such as Qwen, tune VRAM, context length and KV cache, connect the OpenCode terminal agent, and fix the common errors.
Can you have an AI coding assistant that reads your files, edits them and runs commands, without sending a single line of code to the cloud? Yes. You need a graphics card, a free model server called Ollama, an open language model, and a free terminal agent such as OpenCode. This guide shows how to put the three together on your own computer, and how to tune the settings so the whole model fits in GPU memory, which is the difference between a fast agent and a painfully slow one.
Quick answer. Install Ollama, pull a model that fits your GPU, raise the context length and shrink the KV cache with four environment variables, then point OpenCode at http://localhost:11434/v1. Check that size_vram / size equals 1.0. That is the whole recipe. The rest of this post explains each step.
An analogy. The model is a very well-read employee, Ollama is the office where that employee works, and OpenCode is the assistant who carries your questions in, hands the employee the right documents and does the paperwork. The office must be big enough for the employee and the documents, or work slows to a crawl.
OpenCode (terminal agent) → HTTP API → Ollama (model server) → GPU → answer back to OpenCodeWhat you need
| Item | Minimum | Comfortable |
|---|---|---|
| Operating system | Linux, macOS or Windows | Linux with systemd (the tuning commands below assume it) |
| GPU memory (VRAM) | 8 GB | 24 GB |
| System RAM | 16 GB | 32 GB or more |
| Disk space | 20 GB | 100 GB if you want several models |
| Cost | Free software | A GPU you already own |
Your VRAM decides which models are realistic. A 24 GB card such as an RTX 4090 comfortably runs a 27B to 32B model in 4-bit form. A smaller card works too, with a smaller model and lower expectations.
1. How the three parts fit together
| Part | What it does | Where it runs |
|---|---|---|
| Model | A file of learned weights, for example a Qwen model | Wherever Ollama runs |
| Ollama | Loads the model into GPU memory, computes answers, exposes an OpenAI-compatible API on port 11434 |
Your PC or a server |
| OpenCode | A coding agent in the terminal. Builds prompts, reads and edits files, runs shell commands, sends text to the API | Your PC |
Three facts are worth remembering:
- The model runs where Ollama runs. OpenCode only sends text and receives text.
- File edits and shell commands always happen on the machine running OpenCode, even if the model runs on another computer.
- Your prompts and file contents travel to whichever endpoint you choose. With
localhostthey never leave your computer.
2. Install Ollama
On Linux:
curl -fsSL https://ollama.com/install.sh | sh
ollama --versionThe installer creates and starts a background service named ollama. On macOS and Windows, download the installer from the Ollama website instead.
Check that you have a working NVIDIA driver:
nvidia-smiIf this command fails, install the proprietary NVIDIA driver first.
3. Choose and download a model
How to choose
| Rule | Why |
|---|---|
| The model plus its working memory must fit in VRAM | A full GPU load is the biggest speed factor |
| The model must support tool calling | Agents read files and run commands through tools. Models without reliable tool calling fail at multi-step work |
| Bigger models follow instructions better | Very small models (7B to 8B) often lose track in multi-step agent tasks |
| Q4_K_M is the usual balance | Q8_0 is closer to full quality but about twice the size |
| Reasoning models think before they answer | More accurate on hard tasks, slower and heavier on tokens |
What fits on which GPU
These are rough guidelines. Check real tags and sizes on the Ollama model library.
| Model class | Fits on 24 GB? | Agent suitability |
|---|---|---|
| 27B to 32B at Q4 (about 16 to 20 GB) | Yes, with a reduced context window | Good. The best quality that fits |
| 14B at Q4 to Q8 | Yes, with a large context | Moderate. Fine for simpler tasks |
| 7B to 8B | Yes, with a very large context | Weak for agents. Fine for chat and quick edits |
| 70B and larger at Q4 (40 GB and up) | No, part runs on the CPU | Too slow on a single 24 GB card |
| 100B and larger | No | Server-class hardware only |
Download it
ollama pull qwen3.8:27b-mtp-q4_K_M
ollama listThis example is a Qwen 27B model in 4-bit form, about 16 GB to download. If the tag is not found, open the Ollama library and copy a current tag that matches your VRAM. Then test it:
curl -s http://localhost:11434/v1/models
ollama run qwen3.8:27b-mtp-q4_K_M "Answer in 3 words."Reasoning models print a “Thinking…” block before the answer. That is normal.
4. Tune Ollama so the model fits in VRAM
This is the step most guides skip, and it matters most. With default settings, a 27B model on a 24 GB card ended up only 88.8% inside GPU memory. The rest sat in system RAM, and generation was slow. Four settings fixed it.
What the four settings do
| Variable | Meaning | Effect |
|---|---|---|
OLLAMA_CONTEXT_LENGTH |
How many tokens a request can hold: prompt, history and answer together | Memory grows with it. Too small breaks agents, because the system prompt and tool list are large. Ollama’s built-in default is small, so raise it |
OLLAMA_FLASH_ATTENTION=1 |
A memory-efficient attention method | Less VRAM at long context, usually faster. Needed for a quantized KV cache |
OLLAMA_KV_CACHE_TYPE=q8_0 |
Stores the conversation memory (the KV cache) in 8 bits instead of 16 | Roughly halves that memory with a small quality loss. q4_0 quarters it, with a more visible loss |
OLLAMA_NUM_PARALLEL=1 |
How many requests run at once | Memory is reserved per slot. One slot is right for a single user |
Apply them on Linux
sudo systemctl edit ollamaAn editor opens. Paste these lines above the line that says “Edits below this comment will be discarded”:
[Service]
Environment="OLLAMA_CONTEXT_LENGTH=24576"
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"
Environment="OLLAMA_NUM_PARALLEL=1"Save (in nano: Ctrl+O, Enter, Ctrl+X), then restart:
sudo systemctl restart ollama
systemctl show ollama | grep -i EnvironmentImportant: these lines are not shell commands. Pasting them into a terminal gives [Service]: command not found. They belong in the editor that systemctl edit opens.
On macOS and Windows, set the same variables the way your system sets environment variables, then restart Ollama.
Verify that the model is fully on the GPU
ollama run qwen3.8:27b-mtp-q4_K_M "Hi" > /dev/null
curl -s http://localhost:11434/api/ps | python3 -c "import sys,json; m=json.load(sys.stdin)['models'][0]; print(m['size_vram']/m['size'], m['context_length'])"
nvidia-smi --query-gpu=memory.used,memory.total --format=csvThe target is 1.0 24576. Anything below 1.0 means some layers run on the CPU and everything slows down.
Here is how the ratio improved on a 24 GB RTX 4090 with a 27B Q4 model:
| Step | Settings | size_vram / size |
|---|---|---|
| Default install | Context 32768, 16-bit KV cache | 0.888 |
| Tuning, round 1 | Context 32768, flash attention, q8_0 KV cache |
0.937 |
| Tuning, round 2 | Context 24576, flash attention, q8_0 KV cache, one slot |
1.0 |
If the ratio is still below 1.0
Work down this list, one change at a time:
- Turn on flash attention and the
q8_0KV cache. - Set
OLLAMA_NUM_PARALLEL=1. - Lower the context length: 32768, then 24576, then 16384.
- Try the
q4_0KV cache. - Free GPU memory. Close browsers and video apps, or stop the graphical session for a test.
- Use a smaller model or a smaller quantization.
The reason is simple. VRAM must hold the model weights, the KV cache, some compute buffers, and anything else using the GPU, such as your desktop. When the total is too big, Ollama pushes layers to system RAM. The settings above shrink the KV cache until everything fits.
5. Install OpenCode and connect it
Install
curl -fsSL https://opencode.ai/install | bash
source ~/.bashrc
opencode --versionIf opencode: command not found remains, open a new terminal or run ~/.opencode/bin/opencode.
Write the config
The file is ~/.config/opencode/opencode.json. Add a local provider:
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"local-ollama": {
"npm": "@ai-sdk/openai-compatible",
"name": "Local Ollama",
"options": {
"baseURL": "http://localhost:11434/v1",
"apiKey": "ollama"
},
"models": {
"qwen3.8:27b-mtp-q4_K_M": {
"name": "Qwen 27B (local)",
"limit": { "context": 24576, "output": 8192 },
"tool_call": true
}
}
}
}
}What each field means
| Field | Meaning |
|---|---|
npm |
The library OpenCode uses to talk to the API. @ai-sdk/openai-compatible works with Ollama |
baseURL |
The API root, including /v1 |
apiKey |
A placeholder. Ollama ignores it, but the library needs some value |
| Model key | Must match the Ollama tag exactly. Copy it from ollama list |
limit.context |
Must be no larger than OLLAMA_CONTEXT_LENGTH |
limit.output |
The maximum tokens per answer |
tool_call |
Declares tool support. Required for agent work |
Three rules prevent most config errors:
- Every provider must sit inside the
"provider"object. A provider at the top level is silently ignored. - Do not add
variantsorreasoning_effort. A wrongvariantsvalue makes OpenCode drop the whole provider, and Ollama has noreasoning_effortsetting. - Keep one config file. If both
opencode.jsonandopencode.jsoncexist, back up and keep just one.
Edit the file safely with a script
Hand-editing JSON is easy to get wrong. This script adds the provider for you, after a backup:
cp ~/.config/opencode/opencode.json ~/.config/opencode/opencode.json.bak
python3 - <<'PY'
import json, os
p = os.path.expanduser('~/.config/opencode/opencode.json')
d = json.load(open(p))
d.setdefault('provider', {})['local-ollama'] = {
"npm": "@ai-sdk/openai-compatible",
"name": "Local Ollama",
"options": {"baseURL": "http://localhost:11434/v1", "apiKey": "ollama"},
"models": {"qwen3.8:27b-mtp-q4_K_M": {
"name": "Qwen 27B (local)",
"limit": {"context": 24576, "output": 8192},
"tool_call": True}}
}
json.dump(d, open(p, 'w'), indent=2)
PYThen check the result:
python3 -m json.tool ~/.config/opencode/opencode.json > /dev/null && echo OK
opencode models | grep -i local6. Use your agent
cd /path/to/your/project
opencodeInside the terminal interface:
| Command | Purpose |
|---|---|
/models |
Choose the model. Check the provider label, because the same model name can exist on several providers |
/compact |
Shrink the conversation when the context fills up |
/exit |
Quit |
Restart OpenCode after every config change.
Start with a small, safe task, such as “explain what this project does” or “add a docstring to this function”. Then move to edits across several files.
7. Optional: share one GPU machine with several computers
If a stronger computer is on your network, you can run Ollama there and connect OpenCode from your laptop. Only two things change.
On the GPU machine, make Ollama listen on the network, using the same override file as before:
Environment="OLLAMA_HOST=0.0.0.0:11434"On your laptop, point the provider at that machine instead of localhost:
"baseURL": "http://GPU-MACHINE-ADDRESS:11434/v1"Test it with curl -s http://GPU-MACHINE-ADDRESS:11434/v1/models.
Security warning. Ollama has no login. Anyone who can reach the port can use the model. Expose it only inside a trusted network or a VPN, and block the port from the internet with a firewall. Remember also that your prompts and file contents now cross that network.
Very large models need very large hardware. A 125B model in 8-bit form is close to 190 GB, which is server territory. Expect a minute or so for the first request, because loading such a model takes time.
8. Which setup should you choose?
| Your situation | Choice |
|---|---|
| Private code, offline work, everyday tasks | A 27B to 32B model on your own 24 GB GPU |
| Hard tasks across many files, best accuracy | The largest model you can run, on a shared server |
| GPU below 16 GB | A 14B model. Expect lower reliability |
| No GPU | A small model works but is slow. Use it to learn, not for real work |
Things to keep in mind
- Cold start. The first request loads the model into GPU memory. Agents may time out. Wait and resend. Switching models causes another cold start.
- Context fills up. Long sessions run out of room. Use
/compactor start a new session. - Models unload after 5 minutes by default. Set
OLLAMA_KEEP_ALIVE=30mto keep them loaded longer, or-1to keep them loaded forever. - Other GPU programs steal VRAM. Browsers with hardware acceleration, video and other CUDA jobs can push layers back to the CPU. Recheck the ratio after starting them.
- Agents act on your machine. OpenCode edits files and runs shell commands. Read each tool call, especially with a model you do not know yet, and work in a project that is under Git, so every change can be undone.
- Thinking models answer more slowly, because they reason first.
Common problems and fixes
| Problem | Fix |
|---|---|
opencode: command not found after installing |
Run source ~/.bashrc, open a new terminal, or use ~/.opencode/bin/opencode |
snap install opencode is refused |
Snap needs classic confinement, which removes the sandbox. Use the official installer instead |
[Service]: command not found |
The override lines were pasted into the shell. Put them into sudo systemctl edit ollama |
The local provider does not appear in /models |
The block is outside "provider", the file was not saved, the JSON is invalid, or OpenCode was not restarted |
| Only some providers appear | Check with grep -c local-ollama ~/.config/opencode/opencode.json and add the block with the script above |
size_vram / size is below 1.0 |
Follow the tuning list in section 4 |
| The first request is slow and the agent gives up | It is a cold start. Wait, then resend |
Connection refused to a remote machine |
Wrong address, the machine is not reachable from your network or VPN, or the port is blocked |
| Model name not found | A typo in the tag. Copy the exact name from ollama list. The : and _ count |
| Answers cut off, or errors in long chats | The context is full or limit.context is larger than the server setting. Use /compact and make the two values match |
| Generation is slow on your own GPU | Part of the model runs on the CPU, or another program uses the GPU. Check /api/ps and nvidia-smi |
Command cheat sheet
| Command | Purpose |
|---|---|
ollama pull <model> |
Download a model |
ollama list |
List installed models |
ollama run <model> "prompt" |
Run a model for one question or interactively |
ollama ps |
Show loaded models and the GPU/CPU split |
ollama rm <model> |
Delete a model |
sudo systemctl restart ollama |
Apply new settings |
journalctl -u ollama -f |
Follow the server log |
watch -n 1 nvidia-smi |
Watch VRAM live during a request |
opencode models |
List the models OpenCode knows from your config |
Frequently asked questions
Is a local AI coding agent really private?
With localhost, prompts and files stay on your computer. If you point the agent at a remote server or a cloud service, the text you send goes there, so check the provider name before you start.
How much VRAM do I need to run a local LLM? Plan for the model file size plus several gigabytes for the KV cache and your desktop. A 27B model in Q4 form needs about 19 GB for the weights alone, so a 24 GB card is comfortable and a 16 GB card needs a smaller model.
What is the KV cache? It is the model’s short-term memory of the conversation. It grows with the context length, and it can take several gigabytes. Quantizing it to 8 bits halves that memory.
Why is my local model slow?
The most common cause is that part of the model does not fit in VRAM and runs on the CPU. Check size_vram / size. It should be 1.0.
Is a local model as good as a cloud model? Not at the very top end. A good open 27B to 32B model handles everyday coding tasks well, and the largest cloud models are stronger on the hardest ones. Local models trade some accuracy for privacy, no per-token cost and offline use.
Which is better, Ollama or OpenCode? They do different jobs. Ollama serves the model, and OpenCode is the agent that uses it. You need both, or an alternative for each.
Further reading
- Ollama documentation
- Ollama model library
- OpenCode documentation
- NVIDIA System Management Interface (nvidia-smi)
Try it yourself
- Install Ollama, pull a model that fits your GPU, and ask it one question with
ollama run. - Run the verification command and write down your
size_vram / sizevalue before any tuning. - Apply the four settings, restart Ollama and compare the new ratio.
- Connect OpenCode to your local model and ask it to explain a small project.
- Lower the context length by one step and see how the memory use in
nvidia-smichanges.