active
Local LLM workbench
A repeatable way to load, quantise, and compare open-weight models on one 12 GB GPU — and to find where a smaller model stops being worth the memory it saves.
- status
- Active — measurement passes ongoing
- hardware
- NVIDIA Titan X (Maxwell), 12 GB VRAM, 32 GB system RAM
- stack
- llama.cpp, GGUF quantisations, PowerShell harness
- models
- Qwen-family baselines, with others swapped in for comparison
The question
A 12 GB card sits in an awkward spot. It is enough to run useful models locally, and not enough to stop thinking about memory. Every choice — parameter count, quantisation level, context length, how many layers get offloaded — trades against the others, and the guidance online is mostly written for hardware two generations newer.
So: on this card, what is the largest model worth running, and at what quantisation does the output stop being trustworthy?
Setup
llama.cpp is the runtime, chosen because it exposes the knobs that matter and runs the same way on both Windows and WSL. Models are pulled as GGUF builds so quantisation levels can be swapped without re-converting anything.
A PowerShell harness wraps the whole thing: it loads a model at a given quantisation, runs a fixed prompt set, records throughput and peak memory, and writes one row per run to CSV. Same prompts, same seed, same order, every time. Without that discipline the numbers drift enough to be meaningless.
What gets measured
- Tokens per second, prompt evaluation and generation reported separately.
- Peak VRAM and the layer-offload split that produced it.
- Time to first token at a realistic context length, not an empty one.
- Output quality on a fixed prompt set: instruction following, code that compiles, and whether the model invents facts when it should say it does not know.
Measurement pass in progress. Results tables land here as each model and quantisation level finishes its run.
Notes so far
- Peak memory during a run matters more than the file size on disk. A model that loads comfortably can still fall over once context fills.
- The Maxwell architecture is missing a lot of what modern kernels assume. Comparisons against newer cards are not apples to apples, and that is fine — the point is the floor, not the ceiling.
- Quality degradation is not gradual. There is usually a step where a model goes from slightly worse to confidently wrong, and finding that step is most of the value.
Reproducing it
Everything needed is public: llama.cpp, a GGUF build from the model's own repository, and the harness. The only local dependency is the hardware, which is listed above so the numbers can be read in context.
Related work in the lab
The same rig feeds the agent experiments — a local model with tools attached is a very different threat model from a local model answering questions.