Contents

Making a model that runs on your own PC write code. That much has been possible for years, but what was possible was completion: filling in the rest of a line you were already typing. The agentic pattern, where the model reads the repository, edits several files and runs the tests, was too heavy to run locally.

That started to shift in late 2025 and through 2026. This article lays out how far you can actually get today and where you get stuck, using primary sources only. The conclusion up front: it works, but there is one gate, and if you do not change one setting it breaks on you silently.

📌 About the numbers in this article: every benchmark and model specification comes from the developer's own announcement (Mistral AI official, the Qwen official model card, the Ollama official docs, the Cline official docs, Anthropic official). Nothing is taken second-hand from roundup posts, because while researching this I ran into several roundups that had attributed the scores of one model size to a different one.

1. What changed: open models reached the agentic tier

The clearest sign of the shift is that model developers themselves now market their models for agentic coding.

The official Qwen model card says the model supports most platforms such as Qwen Code and CLINE, and features a specially designed function call format, so the editor extensions are named outright (Qwen3-Coder-30B-A3B-Instruct model card). In other words, the model is built with a particular coding agent in mind.

Mistral AI is heading the same way. Devstral 2, announced on December 9, 2025, is a dedicated model explicitly aimed at agentic coding, and the smaller Devstral Small 2 (24B) was released under Apache 2.0 (Introducing: Devstral 2 and Mistral Vibe CLI).

When it was completion only

Finish the line you are typing. A small model is enough, and the context can stay short. It ran comfortably on local hardware

Once it becomes agentic

You need tool calls emitted accurately and a long context held reliably. Open models that clear that bar have arrived

2. Two kinds of tool: Continue and Cline are not the same thing

Start out confusing the two and your expectations will not match the results. Both are VS Code extensions and both connect to Ollama, but their design philosophies are completely different.

　 Continue Cline
Character A bundle of completion, chat and edit An autonomous coding agent
How roles are split A separate model per role: chat / edit / apply / rerank / autocomplete One model plans and executes the whole way through
Hardware demands Low. A few GB of model is enough for completion High. It needs a long context and accurate tool calls
What it suits Making your own hands faster as you write Handing over a whole task

Continue's one-model-per-role design fits local hardware well, because you can give completion a small fast model and chat a bigger one. The official Ollama guide names lightweight models outright for completion, such as qwen2.5-coder:1.5b and starcoder2:3b (Continue — Ollama guide).

⚠️ That said, the models recommended in official docs can be out of date. The guide above suggests llama3.1:8b and deepseek-r1:32b for chat, and those are a good few generations behind what you can pick today. The procedure in the docs still works, but do not take the model names at face value; re-pick them using section 4, "Picking a model", below.

3. The first pitfall: context length is decided by your VRAM

This is the most important section in the article. Miss it and you land in the state where setup succeeded, yet the agent starts behaving incoherently halfway through. And no error is raised.

The cause is Ollama's default context length. It is not a fixed value; it is chosen automatically from how much VRAM you have (Ollama — Context length).

VRAM Default context length
Under 24 GiB 4k
24 to 48 GiB 32k
48 GiB and above 256k

A typical gaming PC (8 to 16 GB of VRAM) lands on the top row. That means 4k. An agent blows past that easily once you add the system prompt, file contents and the back-and-forth of tool calls, so the start of the conversation is quietly cut away. It forgets instructions, repeats the same action, loses track of the goal: the cause is not that the model is stupid, it is that the context is being discarded at the door.

For exactly this use case, the Ollama docs state that you should set the context to at least 64000 tokens for demanding work such as web search and coding tools. You configure it with an environment variable when starting the server.

OLLAMA_CONTEXT_LENGTH=64000 ollama serve

If you use the app version, the settings screen has a slider for the same thing.

⚠️ But raising it is not the end of the story. The docs warn that increasing the context length increases the amount of memory the model needs to run.

Once it no longer fits in VRAM, part of the model is pushed onto the CPU and things get dramatically slower. Check whether the setting is really taking effect with ollama ps, which is what the docs tell you to do. That is where you see whether the model fits entirely on the GPU.

Cline has its own answer to the same problem. The official docs recommend enabling Use Compact Prompt and keeping tasks narrow, since a smaller context responds faster (Cline — Running models locally). The agent's own system prompt is long, so there is a setting specifically to shrink it.

4. Picking a model: what actually works, by VRAM

Do not rely on the recommendations in roundup posts. Within what I checked, some listed mid-2025 models as "the pick for 2026", and one had put a 480B model's score in the column for a 30B model. What follows is only what the developers themselves published.

Model Size Context License SWE-bench Verified
Qwen3.6-35B-A3B 35B (3B active) 262,144 (up to 1,010,000) Apache 2.0 73.4
Devstral Small 2 24B 256K Apache 2.0 68.0%
Devstral 2 (for reference, too big) 123B 256K Modified MIT 72.2%

Sources: Qwen3.6-35B-A3B model card (Terminal-Bench 2.0 is 51.5 and QwenClawBench is 52.6) and the Mistral AI announcement (December 9, 2025). These scores were measured by each vendor on its own models, not by a third party running them side by side.

The number to notice is the "3B active" on Qwen3.6-35B-A3B. Of the 35B, only 3B worth is actually engaged on each pass, and that is where MoE (Mixture of Experts) earns its keep. It is a structure that aims for the intelligence of a large model and the speed of a small one at once, which makes it a good fit for local use.

Download size and required specs

In the Ollama library, qwen3.6 offers 27b (18GB) and 35b (23GB) side by side. There are also -mlx tags for Mac.

Cline's official memory guidance goes like this: 16 to 32GB for small models, 32 to 64GB for mid-size coding models, and 64GB or more for large models with a larger context.

VRAM 8 to 12GB

Stick to completion with Continue. Either give up on the agentic route or pair it with the cloud

VRAM 16 to 24GB

Devstral Small 2 (24B) is within reach. Quantization is mandatory because you need headroom to raise the context

VRAM 24GB+ / unified memory 32GB+

Qwen3.6 at 27b or 35b becomes realistic. The default context also rises to 32k

Mistral writes that Devstral Small 2 runs on consumer GPUs and even on CPU-only setups. It is worth keeping in mind, though, that "runs" and "runs fast enough to be practical as an agent" are two different things.

5. Setup: the shortest path

1. Install Ollama and set the context

The order matters. Decide the context length before you pull a model.

OLLAMA_CONTEXT_LENGTH=64000 ollama serve

The installation steps themselves are covered in our complete guide to Ollama, so refer to that.

2. Pull a model

ollama pull qwen3.6:27b

If VRAM is tight, pick Devstral Small 2 instead. What separates both of these from a general-purpose chat model is that they were built with agentic use in mind.

3. Confirm it is on the GPU

ollama ps

Do not skip this check. If part of the model has been pushed onto the CPU, the experience is a completely different thing. Most of the "local LLMs are too slow to be usable" verdicts trace back to exactly this.

4. Connect the extension

In both Cline and Continue, choose Ollama as the provider and point it at http://localhost:11434. Cline's official caveat is the plain one, make sure Ollama is running before you send a prompt, and no special configuration is needed.

If you are using Cline, enable Use Compact Prompt at the same time.

6. The gap with the cloud has become hard to measure

Here is where I want to be candid. A clean comparison of the form "local models have reached X% of the cloud" cannot be produced right now. The reason is that the benchmarks no longer line up.

The open-model side publishes SWE-bench Verified: 73.4 for Qwen3.6-35B-A3B, 68.0% for Devstral Small 2. The frontier side, meanwhile, is drifting away from that metric.

In fact, Anthropic's Claude Opus 5 announcement carries no SWE-bench Verified figure at all. What it lists is Frontier-Bench v0.1, CursorBench 3.2, the AA Coding Agent Index and FrontierCode 1.1, and much of it is phrased in relative rather than absolute terms (more than double the performance of Opus 4.8, within 0.5% of Fable 5's peak score, and so on).

⚠️ So when you see a claim that local is at some percentage of Claude, be suspicious. The odds are good that the two sides were not measured on the same metric, or that the comparison is against a Claude from several generations ago. These days SWE-bench Verified lines up little more than open models against each other.

What can still be said about the gap

Even without matching numbers, the places where the difference shows up structurally are clear enough.

Where local wins

Your code never leaves the machine, no metered billing (run it as often as you like), it works offline, and there are no rate limits

Where the cloud wins

Reasoning that spans several files, stability over long autonomous runs, no upfront investment, and the model gets newer without you doing anything

There is a reason the gap shows up most in "reasoning that spans several files". That work consumes context heavily and stacks dozens of judgments on top of each other. A difference in the accuracy of a single judgment compounds with every round trip. A gap you would not notice while touching up one file becomes something you can feel at repository scale.

7. Is local really free?

It is true that no API bill arrives, but it is not free. The shape of the cost has simply changed.

Item Local Cloud
Upfront cost A GPU with lots of VRAM, or plenty of unified memory None
Cost that grows with use Electricity only Token billing or a subscription
Cost that is easy to miss The effort of setup and upkeep, plus keeping up with model releases None (the provider carries it)

Which is cheaper therefore flips depending on what you compare. If you already own a 24GB-class GPU, local runs at close to zero extra cost. If you do not, the price of that GPU buys you years of subscription. It is more accurate to treat the answer as depending on two things: whether you run a lot every day, and whether you already own the hardware.

8. When to use which

Good reasons to choose local

The code cannot leave the building (contracts, internal policy), you already have the hardware, you want to iterate without counting requests, or you work offline

Bad reasons

"Because it is free" (the hardware has not been counted) and "because it looks fast" (the cloud is usually faster)

The most realistic setup is to use both. Hand Continue's completion to a small local model and let it run all the time, then pass substantial pieces of work to a cloud agent. Completion is frequent and light per call, which suits local; an agent is infrequent and heavy per call, which suits the cloud. The load profiles are exactly inverted.

Summary

  • Open models have reached agentic coding. Both Qwen and Mistral now ship dedicated models that name editor extensions outright
  • The biggest gate is Ollama's default context length. Under 24GiB of VRAM it is 4k, and the agent breaks silently. The official recommendation for coding work is 64000 or more
  • Once you raise it, check with ollama ps that the model is on the GPU. Pushed onto the CPU, it becomes a different kind of slow
  • Continue and Cline are different tools. The first assigns a model per role for completion; the second is an autonomous agent. Their hardware demands differ
  • "What percentage of the cloud" comparisons barely hold up any more. The frontier side is drifting away from publishing SWE-bench Verified
  • It is not "free", the cost has a different shape. The answer flips on whether you already own the hardware

FAQ

Q1. Can I run the agentic setup on a GPU with 8GB of VRAM?

It is a stretch, because even if the model itself fits, there is no headroom left to raise the context. Ollama's default drops to 4k below 24GiB of VRAM, and pushing that to 64000 increases the memory required, which is hard to reconcile at 8GB. The realistic options are to stick to completion with Continue, or to run completion locally and put the agent in the cloud.

Q2. Cline or Continue, which should I start with?

If this is your first local LLM, start with Continue, because its hardware demands are lower and it is easier to isolate problems when things do not work. Confirm that completion runs comfortably before moving on to the agentic route, and you will be able to tell whether a problem is the model or the configuration.

Q3. Does it work on a Mac?

Yes. Unified memory doubles as VRAM, so large models are actually easier to load. The Ollama library also offers -mlx tags of qwen3.6 for Apple Silicon. Note, though, that the same rule of deriving the default context length from memory size applies, so you still need to check.

Q4. Which model is "the smartest"?

By published scores alone, the highest among those listed here is Qwen3.6-35B-A3B with 73.4 on SWE-bench Verified. But these are each vendor's own measurements, not a third-party side-by-side. In practice, the license (Devstral Small 2 and Qwen3.6 are both Apache 2.0) and whether the model fits in the VRAM you have matter more.

Q5. Why does it "break without raising an error"?

Because anything beyond the context length is handled as truncation, not as an error. The model responds perfectly well to the range it was handed. The result shows up as forgotten instructions, repeated actions and a lost goal, and it looks like the model is simply not capable enough. Knowing that this symptom should make you suspect the configuration changes how fast you narrow it down.

Q6. Which quantization should I pick?

If you are unsure, starting around Q4_K_M is the safe move. The differences between formats (GGUF / GPTQ / AWQ) and how to choose are collected in our complete guide to quantization formats. For coding work, "a slightly smaller model with lighter quantization" tends to be more stable than "a bigger model quantized harder", because the tool-call format has to be followed exactly.

Q7. Is it safe to use on company code?

As long as everything stays local, the code never leaves the machine, and that is the single biggest advantage of a local LLM. Do check the extension's own settings, though: even with the provider pointed at Ollama, telemetry or some other feature may still talk to the outside. And whether your policy permits it is a separate question from the technology, so check your internal rules first.

Related articles