>>328756
>download llama.cpp and install
>if you're AMD or Intel GPU use Vulkan, if you're Nvidia use CUDA or Vulkan
>HIP is slower than Vulkan and Intel dGPU compute is a meme
In regards to models, there's multiple things to know.
First is whether the model is a dense model or an MoE model:
A dense model is fully loaded into VRAM and accesses all of its parameters during interference, offloading to RAM makes these slow as shit and/or may cause crashes.
A MoE (Mixture-of-Experts) model instead only loads part of its parameters into VRAM (the "active" parameters) and keeps the rest in RAM.
On Huggingface this is typically denoted by the model parameter size designation and naming scheme, a dense model usually follows
>[Model Name]-<parameter number>B(illion)
whereas a MoE
>[Model Name]-A<parameter number>B-<parameter number>B
Two common examples:
>Qwen3.6-27B
and
>Qwen3.6-A3B-35B
with the 27B being dense and the A3B-35B being MoE with 3B active parameters and 35B total parameters.
Dense models reference all parameters during inference, while MoE models have routing layers that selectively load parameters into VRAM depending on the prompt.
MoE are at a disadvantage here as the routing layers may miss useful parameters during inference, but they can run at a fraction of VRAM use with only a minor performance penalty and run faster than dense models when entirely loaded in VRAM.
Check the model description and architecture in any case, some MoE aren't labeled correctly as this isn't a hard rule.
Next thing to consider is quantization, this is basically lossy image compression but for artificial neural networks so they can fit into poorfag computers and still run albeit a bit more retarded.
The models I mentioned above are 50-70GB in their raw BF16 full-precision baseline which is obviously retardedly huge and won't fit into an RTX 5090 so outside the smallest models you'll probably end up running a quant.
Quantization uses the following naming scheme:
<Legacy
>QX_Y (Y is either 0 or 1)
With X being the baseline bits-per-weight and Y whether or not it uses a straight number or does some additional floating point math to increase it slightly.
Example:
>Q4_1
Is a quant with a 4-bit baseline actually 4.25 for some math autism reason with Y denoting it pushed to 4.5 bits per weight uniformly across the model.
Legacy quants are obsolete outside of the near-lossless Q8_0 for quantizing models but the underlying algorithms are still used for quantizing context windows which llama.cpp supports.
<K
>QX_K_Y
X being baseline bits and Y being the secondary size encoding.
K-quants selectively quantize select parameters of a model at higher bpw than the baseline using 7D interdimensional backgammon logic I don't understand enough to put into this post, the secondary encoding size determines the tendency of the encoder to go beyond the baseline.
In rough terms S equals an average 4.25bpw, M 4.5, XL 4.7 and so on, quantization is a highly autistic field with a number of custom schemes developed by HF users (like unsloth's UD quants) but the basic number designations are usually unchanged.
Q4_K_M is the huggingface client's and everyone else's default quant as it typically retains 95% of the performance compared to the F16 or BF16 baseline, higher quants increase this with Q8_0 being at 99% and the latest UD-Q8_K_XL being at 99.1488% or something.
Below Q4 is where models get increasingly retarded and schizophrenic outside of carefully preset and perhaps post-trained "quants" or models trained at a low bpw to begin with.
<I
>IQX-YY
X being baseline bits and Y being either XS or NL apparently rather than a bpw designation this is some matrix storage scheme which performs nearly identical in benchmarks but NL is supposed to be easier on the CPU or something idk they run the same on my end.
Quasi-successor to K-quants, these don't use an algorithm using a generic heuristic applied across the whole model to determine bpw allocation but instead do this according to an importance matrix, this imatrix being generated by asking the model to be quantized a bunch of questions and observing which parameters light up more.
They can exceed the performance of K-quants in certain aspects while also regressing in others, I-quants are often the only real option for semi-viable Q3 or Q2 quants but quality varies strongly according to the imatrix file (and the preceding training question dataset) and as such haven't replaced K-quants.
Ok so after all that you can now choose your pokemodel.
It can be any model that is in GGUF and officially supported by llama.cpp, if it isn't supported there's probably a meme fork out there.
In terms of models the currently most relevant for basic local AI faggotry are:
>Qwen3.6
Generally numba wan in most aspects at its size, comes in three flavors depending your hardware and/or needs. Do not ask about unusual incidents in 1989.
>Gemma 4
Goolag's answer to Qwen. Not as good as Qwen3.6 but also doesn't think itself to death nearly as much, but be aware it refuses to use any search engine that isn't Google if you're running it as part of an Agent harness.
>Bonsai
Qwen3.6 architecture but trained from scratch at 1-bit which actually works even if it's dumber than regular Qwen.
>MiniCPM
Small Agent-focused models, not sure if they're any good or just look good on benchmarks but there's community finetunes aplenty so ehhh.
>Qwen2.5-3
Legacy, use if you can't run the others or if you're a developer and want to try out experimental finetuning/quantization/etc. schemes.
>DeepSeek
The R1 model almost broke the western economy in 2025 but is obsolete by now, the smaller models used to popular but have been obsoleted by Qwen and later DeepSeek versions whose parameter counts require a datacenter to run. Install this if you're a boomer journalist who has been tasked with writing agitprop about how China is winning but also inferior to glorious American Empire.
>GPT-OSS
Saltman's open source GPT release. Was one of the first sort of viable local vibeslopping models when it came out but is now obsolete.
>Llama-3.X
Zucc's entry that laid much of the groundwork for local AI yet is now severely outdated and responds poorly to quantization, but fags still make finetuned ERP models with its architecture to this day.
Pick your model on HF, copy the model name to clipboard and run llama-server with something akin to
llama-server -hf some/model -ngl 99 -fa on -c 4096 -parallel 1 or if it's a MoE
llama-server -hf some/model -ngl 99 -n-cpu-moe 99 -fa on -c 4096 -parallel 1
llama.cpp should start downloading the model and once the download finishes boot up a server instance which can then be accessed at 127.0.0.1:8080.
Check the memory consumption and talk to the model to see if it runs properly, then adjust settings to your liking (such as the tiny 4k context window in my example).
If you want to download a specific quant in the model repo, put a :[quant designation] at the end of the name, like
llama-server -hf some/model:Q4_K_XL -ngl 99 -fa on -c 4096 -parallel 1