Local AI is just a file and a program. Here is how to run it yourself

Local AI is just a file and a program. Here is how to run it yourself

Everywhere you look, someone is telling you to run AI on your own hardware. Almost none of them stop to explain what that actually means. The jargon piles up fast — weights, quantization, VRAM, GGUF — and the whole thing starts to look like it demands a data-science degree and a workstation that costs more than a used car. That impression is wrong, and it is exactly why most people give up and keep paying for a cloud subscription. So let us strip it down to one honest sentence, then unpack the pieces underneath.

Local AI is simply a model file sitting on your computer and a program that runs it. That is it. No cloud, no API keys, no internet, no subscription. Everything else is a small detail.

[video: Local AI Demystified — Mohammed Kmail]

What local AI actually is (and how it differs from ChatGPT)

When you open ChatGPT, Claude, or Gemini, your prompt is not processed on your desk. It is shipped over the network to a data center, where a machine you have never seen runs a very large model, and the reply is streamed back to you. Your own computer contributes almost nothing — it is effectively a remote terminal into someone else's hardware.

Running locally inverts that arrangement. The model — the file that holds all of the learned behavior — is copied onto your machine, and your own CPU or GPU performs the computation. Nothing has to leave your desk. That shift buys three concrete advantages:

• Private. Your data never travels anywhere; it stays on your computer.
• Free. No subscription, no per-token cost.
• Offline. A plane, a coffee shop with terrible Wi-Fi — it does not matter.

The honest tradeoff: the models you can run at home are much smaller than the frontier models behind Claude or OpenAI. But they have gotten shockingly good over the last couple of years, and for a huge amount of everyday tasks — even coding — they are more than enough.

The five building blocks

Every local-AI tool, no matter how polished, is built from the same five pieces. Understand these and you understand local models better than most people posting about them.

Building block 1: the model file

A model is literally just a file. A really big file — could be hundreds of gigabytes — full of numbers called weights. Billions of numbers baked in when the model was trained. That file does not think and does not run; it sits on your disk like any other file. Companies like Meta, Google, Alibaba, and Mistral release these files for free — those are your open models: Llama, Gemma, Qwen, DeepSeek. You can literally just download them.

Building block 2: model sizes (parameters)

Model names carry numbers like 4 billion, 8 billion, 70 billion. The B stands for billions of parameters — effectively how many numbers are inside the file. The rule is simple: more parameters generally means a smarter model, but also a bigger file that needs more memory and compute. An 8-billion model is a few gigabytes; a 70-billion model is so large most laptops cannot even load it, and some cannot even store it.

Building block 3: quantization

This is the biggest trick in local AI, and it sounds scarier than it is. Quantization is the same idea as compressing a photo: you store those billions of numbers with less precision, and the file gets dramatically smaller with barely any quality loss. A model that normally needs 16 GB of memory might need only 5–7 GB after quantization. The GGML / GGUF format you keep hearing about is just the standard file format for these compressed models. Quantization is the entire reason normal computers can run AI models at all.

Building block 4: the inference engine

This is the part almost nobody explains. A model is a file full of numbers, and a file cannot run itself. You need a program that loads those numbers into memory and does the math to predict the next token. That program is the inference engine. The most famous one is llama.cpp. And here is the secret: almost every tool you will use — LM Studio, Ollama, Docker Model Runner — is just a wrapper around an engine like this. The engine does all the work; the tool makes it nice to use.

Building block 5: your hardware

There is really only one question that decides what you can run: how much memory do you have, and how fast is it? On a PC with a dedicated graphics card, that number is your VRAM. On a modern M-series Mac, it is your unified memory (Apple shares memory between CPU and GPU). The rule of thumb: the model file needs to fit inside that memory with a little room to spare.

• 8 GB of VRAM/unified memory → 3–4 billion parameter models, even unquantized.
• 16 GB → up to 7–8 billion parameters.
• 32 GB → the 14–30 billion range, where local models start to feel genuinely smart.

Two things matter, and they trade off. Memory capacity dictates the size of the model you can run; memory speed dictates tokens per second. A dedicated GPU is often two to three times faster at inference but has less capacity; a Mac with 128 GB of unified memory can run far bigger models, but slowly. The 14–35 billion range is the sweet spot — decent performance without feeling like you are missing out. And remember the context window: longer conversations and bigger documents also fill your memory, because all that context has to fit in the same space.

The four ways to run a model

There are a lot of ways to run a local model, and like anything in software it comes down to how much control you want. Whatever you pick, you are making the same three decisions: pick a model, pick a size and quantization that fits your memory, and decide how you want to talk to it (a chat window or code). Keep that in mind and every tool makes sense.

1. LM Studio — the clickable app

LM Studio is a regular desktop app. You click, you download, you touch everything, and you never have to open a terminal. Search the model view for a model that matches your memory, pick a quantization level (Q4, Q6, Q8 — lower is more compressed), check the capabilities you need (vision, tool use, reasoning), download, load, and chat. It is one of the easiest ways to download models and has the most features. It also exposes a local server and API if you are a developer who wants to talk to it with curl.

2. Ollama — the developer favorite

Ollama is a little less visual and gives you a bit less control, but it is the most popular way for developers to download and run local models. Install it from ollama.com and it becomes a terminal command:

ollama list          # models you already have
ollama pull <model>   # download a model from the hub
ollama run <model>    # load it into memory and chat

While Ollama is running, it serves all its models on a local port (11434) with an OpenAI-compatible API — so you can send requests in the same format you would to ChatGPT or Anthropic, and it auto-loads any model you ask for. That makes it the natural choice when your scripts and apps need to talk to a model on your own machine.

3. Docker Model Runner — models as containers

Docker Model Runner is an experimental feature in Docker Desktop that treats models like containers. You can write Dockerfiles and compose files and ship models directly with your applications as dependencies. It works very well on Linux with Nvidia hardware (CPU works but is extremely slow), and it exposes a REST API like the others. If you already live inside a Docker stack and want models sitting right next to your apps, this is the way to deploy them.

4. Full code — run the engine yourself

The last tier is running a model in pure code, bringing in your own inference engine (llama.cpp) instead of relying on a wrapper. The surprise is that llama.cpp is the engine all the tools above already use — you are just invoking it directly. Load the model file, create a response, run it:

# pseudo-code: load a GGUF model file and generate
model = load_model("qwen2.5-7b-instruct-q4.gguf")
response = model.chat("Write a haiku about running AI locally")
print(response)

You can also point your code at a running Ollama, LM Studio, or Docker Model Runner server and do the exact same thing. Most developers end up managing models through something like Ollama and invoking them in code — the manual method is for when you want to understand every piece and run in the most efficient way.

Which one should you use?

• Just want to chat, never see a terminal? Use LM Studio. Easiest downloads, most features.
• Developer, want a model your scripts and apps can talk to? Use Ollama. It is what most people reach for daily.
• Already living in Docker, deploying things? Use Docker Model Runner. Great in production.
• Want to understand everything and run most efficiently? Use your own code with llama.cpp.

The bottom line

All of these tools use the same building blocks: a model, which is literally a bunch of numbers in a file, and an inference engine, with fancy features on top. If you understand that, you understand local models. You do not need a monster computer — even a phone can run small models. Pick the tool that matches how much control you want, pick a model that fits your memory, and you are running AI on your own desk, free, private, and offline.

Go deeper: the concepts, explained in detail

This article is the 10,000-foot view. If you want to actually understand the machinery — what a weight is, how quantization really works, why a GPU is fast, what the GGUF format contains — the resources below go deep on each concept. Every link is a primary or authoritative source, verified 2026-09-08.

What a model actually is (weights, parameters, tokens)

• Hugging Face — "What are LLMs?" A clear, beginner-friendly breakdown of what a large language model is, the three transformer types, and how tokenization works. https://huggingface.co/learn/agents-course/en/unit1/what-are-llms
• Jay Alammar — "The Illustrated Transformer" The classic visual explanation of the transformer architecture that powers every modern LLM. If you want to understand what is actually inside the model file, start here. https://jalammar.github.io/illustrated-transformer/
• Databricks — "What are Large Language Models?" A solid overview of what LLMs are, how they are trained, and what they can do. https://www.databricks.com/blog/what-are-large-language-models

Quantization and the GGUF format

• Maarten Grootendorst — "A Visual Guide to Quantization" The single best deep dive on quantization, with 60+ illustrations. Explains the intuition, the methods, and the trade-offs. https://www.maartengrootendorst.com/blog/quantization/
• GGUF file format specification The official spec for the GGUF format — the binary layout, the metadata keys, and every quantization type (Q4_K_M, Q8_0, etc.). https://github.com/ggml-org/ggml/blob/master/docs/gguf.md
• Hugging Face Hub — GGUF docs How GGUF models are stored and served on the Hub, with a full table of quantization types and their bit-widths. https://huggingface.co/docs/hub/en/gguf
• Paul Ilvez — "Demystifying LLM Quantization Suffixes" What Q4_K_M, Q8_0, and Q6_K really mean, with a perplexity table showing the quality loss per quantization level. https://medium.com/@paul.ilvez/demystifying-llm-quantization-suffixes-what-q4-k-m-q8-0-and-q6-k-really-mean-0ec2770f17d3
• SitePoint — "Q4 vs Q6 vs Q8: The Quantization Decision Framework" A practical decision framework for choosing a quantization level, with a checklist. https://www.sitepoint.com/q4-vs-q6-vs-q8-quantization-local-llms/

Hardware: VRAM, GPUs, and memory

• Hyperstack — "VRAM Requirements for LLMs" How much VRAM you actually need for different model sizes, and why it matters. https://www.hyperstack.cloud/blog/case-study/how-much-vram-do-you-need-for-llms
• LM Po — "A Guide to Estimating VRAM for LLMs" A practical walkthrough of estimating VRAM for a given model. https://medium.com/@lmpo/a-guide-to-estimating-vram-for-llms-637a7568d0ea
• DataCamp — "Understanding TPUs vs GPUs in AI" Why GPUs are fast at the parallel math that LLMs need, and how they differ from CPUs and TPUs. https://www.datacamp.com/blog/tpu-vs-gpu-ai
• JAX Scaling Book — "How to Think About GPUs" A deeper, more technical look at GPU architecture — streaming multiprocessors, HBM memory, and why memory bandwidth sets inference speed. https://jax-ml.github.io/scaling-book/gpus/

The tools, in depth

• llama.cpp (GitHub) The inference engine behind most local-AI tools. The README explains its goal and the build docs cover GPU backends (CUDA, Vulkan, Metal). https://github.com/ggml-org/llama.cpp
• llama.cpp HTTP server docs How to run llama.cpp as a local server with a REST API — the same pattern Ollama and LM Studio wrap. https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
• Ollama docs — importing a GGUF model How to bring your own GGUF file into Ollama with a Modelfile. https://docs.ollama.com/import
• Hugging Face — "Use Ollama with any GGUF Model" Run GGUF models from the Hub directly through Ollama. https://huggingface.co/docs/hub/en/ollama
• n8n — "How to Run a Local LLM" A complete setup guide covering Ollama, LM Studio, Jan, and other runtimes, with model recommendations. https://blog.n8n.io/local-llm/
• Xybrid — "Run LLMs Locally: A Complete Guide" Which models fit on which hardware, the major runtimes, and what performance to expect. https://www.xybrid.ai/blog/run-llm-locally-complete-guide

Sources

• Tech With Tim, "Local AI Explained: How to Run AI Models on Your Computer" — https://www.youtube.com/watch?v=edIHPoWgIKU (transcript fetched 2026-09-08)
• llama.cpp (the inference engine behind most local-AI tools) — https://github.com/ggml-org/llama.cpp
• Ollama — https://ollama.com
• LM Studio — https://lmstudio.ai
• Docker Model Runner (Docker Desktop experimental feature) — https://docs.docker.com/desktop/features/model-runner/