How to Run a Local LLM: AI on Your Own PC — Specs, Tools, and the Best Models for Beginners

You probably assume a large language model (LLM) has to run in the cloud. In fact, running AI entirely inside your own PC — a "local LLM" — is now a realistic option. Your data never leaves the machine, there are no API fees, and it works without internet. Small models are only a few gigabytes, so even an ordinary laptop can run them.

This article lays out, for beginners, what a local LLM is, its upsides and downsides, the specs you need, how to get started (from installing Ollama with the official steps to confirming it works), how to choose a model, and when to use it versus the cloud.

1. What Is a Local LLM?

A local LLM means running an AI model like ChatGPT or Claude directly on your own PC (or phone) instead of in the cloud. The AI you normally use sends your input to a far-off server, processes it there, and returns the answer — but a local LLM does all that processing on the machine in front of you.

Think of it like "music streaming vs. downloading." Cloud AI is like streaming over the network every time you want to listen; a local LLM is like downloading the track to your device so you can play it offline, anytime. Once you put a model — a file of a few gigabytes — onto your PC, you can run it with no internet and no fees.

2. Why Local? Upsides and Downsides

Local LLMs have three big draws: privacy, cost, and offline use. Because input never leaves your machine, confidential data is easier to handle, and there are no API fees. Since there's no per-request charge, the gap grows the more you run the same job thousands of times.

There are downsides too. Let's be honest about them.

In short: "a step behind the cutting edge on smarts, but a clear winner on privacy, cost, and offline." For uses where you can accept that trade-off, a local LLM is a powerful option.

3. PC Specs You Need, and Quantization

"Will it run on my PC?" is probably your biggest question. The keys are memory (RAM, or VRAM on a GPU) and quantization.

Quantization is a compression technique that lowers the numeric precision of a model in exchange for a much smaller size. The standard format for local use is GGUF, and Q4_K_M is the go-to balance point. You can check how much it shrinks on Ollama's tag list: Llama 3.1 8B is 16 GB at 16-bit (fp16) but 4.9 GB at Q4_K_M — roughly 30% of the size. The Q4_K_M build is also what you get with ollama run llama3.1:8b (llama3.1 tag list, as of September 30, 2026). Hugging Face lists 207,667 models in GGUF format (Hugging Face's GGUF filter, same date).

So how big are they in practice? Here are the download sizes from Ollama's model library, smallest first.

*Source: the tag pages in Ollama's model library (as of September 30, 2026). Size ranges are shown as the library lists them. Audio: the table on the gemma4 page lists it for E2B and E4B only (the tag list itself shows Text, Image). The muse-glimmer row comes from the muse-glimmer tag list (as of October 5, 2026).

Two tips for reading the table:

• You need more memory than the file size: the table shows file sizes. While running, memory for the conversation history (context) comes on top. Ollama's default context is 4,096 tokens, and the longer you set it, the more memory it uses (Ollama FAQ).
• Check whether it fits on the GPU with ollama ps: if the PROCESSOR column says "100% GPU", the whole model is on the GPU; "100% CPU" means it is running in system memory. A split such as "48%/52% CPU/GPU" means part of it spilled over (same FAQ). The part that runs on the CPU tends to be slower, so that is your cue to step down one tag size.

On Apple Silicon Macs, the CPU and GPU share the same memory (unified memory), so a Mac with plenty of memory can run larger tags on the GPU more easily. The OS and hardware requirements, per each tool's official pages, are as follows.

Sources: Ollama's macOS and Windows docs, and LM Studio's system requirements (all as of September 30, 2026).

The NPU (a dedicated AI component) in recent laptops is not yet used by Ollama. For what an NPU does, see What Is an NPU, and Does Your PC Need One?

For a first try, the tags near the top of the table (around 2–5 GB) are plenty.

By contrast, with tools like Claude Code and Codex that run the model on a server, PC performance only affects how fast local commands run and how much memory is free. The time the AI spends thinking does not change (Claude Code and Codex: PC specs you need).

4. How to Start — Picking a Tool and Your First Run

If you want to go deeper with Ollama, see our complete Ollama guide (commands, API, customization).

Pick one of two go-to tools depending on your style.

Both are free and run on Windows, Mac, and Linux. There are others too — Jan, Open WebUI, llama.cpp. If you prefer clicking to typing, install LM Studio and download one small model — that's the fastest way in. If the command line doesn't scare you, the Ollama steps below get you running in minutes.

Running your first model with Ollama

Here we follow the steps in Ollama's official README and docs, from installation to checking that it works (as of September 30, 2026). For the full command list and finer details, see the commands section of our Ollama guide; here we stick to the four things you need the first time.

1. Install. On macOS and Linux, run this line in a terminal:

curl -fsSL https://ollama.com/install.sh | sh

On Windows, run this line in PowerShell:

irm https://ollama.com/install.ps1 | iex

If you'd rather not pipe a script from the internet straight into your shell, the official manual downloads work too (Ollama.dmg for Mac, OllamaSetup.exe for Windows, and the manual install steps for Linux). For Docker, there's the official ollama/ollama image (Ollama README).

2. Run a model. Like the first example in the README, let's run Gemma 4. If the model isn't on your machine yet, it's downloaded first, then the chat starts.

ollama run gemma4

The default gemma4 tag is 6.6–9.5 GB. If memory is tight, name a smaller tag instead, such as ollama run gemma4:e2b or ollama run llama3.2 (2.0 GB).

3. Check that it's running. In another terminal, run the command below to see the loaded model and its PROCESSOR column (GPU or CPU). As noted in section 3, "100% GPU" is what you want.

ollama ps

4. Clean up. ollama stop gemma4 unloads it from memory, and ollama rm gemma4 deletes it when you no longer need it. Models live in ~/.ollama/models on macOS, /usr/share/ollama/.ollama/models on Linux, and C:\Users\%username%\.ollama\models on Windows; you can change this with the OLLAMA_MODELS environment variable (Ollama FAQ).

5. How to Choose a Model (Examples as of September 2026)

Models turn over quickly, so it pays to learn the criteria before the names: (1) does it fit in your memory (the table in section 3), (2) do you need image or audio input, (3) is it strong at what you need, such as your language or code, and (4) is it still being updated? You can check names, sizes, and update dates in Ollama's model library. Representative examples as of September 30, 2026:

The trick is to go step by step: try one smaller tag, confirm with ollama ps that it's on the GPU, and move up one size only if it falls short. Just as with our cloud AI comparison, the only way to know if a model fits your use is to try it.

6. Local vs. Cloud: When to Use Which

For a deeper look at the differences, performance gap, and how to choose between local and cloud, see our article on the local LLM vs cloud LLM comparison.

Local and cloud aren't an either/or — the smart move is to split them by role.

• Local is a fit for: handling confidential data, high-volume or repetitive tasks, offline environments, and routine work where you want to keep costs down.
• Cloud is a fit for: hard problems that need top accuracy, research that needs up-to-date information, and heavy coding — where the top-tier models like ChatGPT, Claude, and Gemini shine.

For example, "do drafts, classification, and summaries locally for free, and send only the final polish or the hard parts to the cloud" works well. Keep in mind that "free" refers to API fees: you still pay for electricity and hardware. If your volume is small, the cloud can come out cheaper — which one wins flips with how much you use it.

Summary

Three takeaways on local LLMs.

• What it is: running an AI model on your own PC. Data never leaves, with no API fees and no offline limits.
• How to start: LM Studio for beginners, Ollama for developers (install with the official script, run ollama run, and check with ollama ps that it's on the GPU). Begin with a small tag of around 2–5 GB. Quantization (Q4_K_M) shrinks a model to about 30% of its fp16 size.
• When to use: the cloud is smarter. Use local for confidential, high-volume, and offline work, and the cloud for hard problems — combining both is best.

Start by installing LM Studio and running one small model. The experience of "AI talking entirely inside your own PC" is more refreshing than you'd expect. If you want to understand the mechanics, see how LLMs work too.

FAQ