{"id":2953,"date":"2026-11-03T19:27:04","date_gmt":"2026-11-03T19:27:04","guid":{"rendered":"https:\/\/www.balajibandi.com\/?p=2953"},"modified":"2026-11-03T19:32:49","modified_gmt":"2026-11-03T19:32:49","slug":"open-source-models","status":"publish","type":"post","link":"https:\/\/www.balajibandi.com\/?p=2953","title":{"rendered":"Open-Source Models"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">I have been working lot these days to test Local LLM that fits and work as smooth as possible on my Legacy hardware<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I am running llama.cpp with gguff models as below for testing :<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Llama-3.2-3B-Instruct-Q4_K_M.gguf<br>Qwen3-8B-Q4_K_M.gguf<br>Qwen3.5-2b-Kimi-and-Opus-Distillation.Q8_0.gguf<br>TwIL-LM3-Pro-Q4_K_M.gguf<br>gemma-3-1b-it-f16.gguf<br>gemma-4-E2B-it-heretic-Q4_K_M.gguf<br>gemma-4-E2B-it-heretic-Q5_K_M.gguf<br>gemma-4-E2B-it-heretic-mmproj-F16.gguf<br>gemma4-coding-Q4_K_M.gguf<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So the question come to me what the meaning after the model we see K, M and some numbers what that means to me<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Lets Look below and understand that.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The numbers (3, 4, 5, 6, 8) stand for the number of bits used per weight (quantization level),<br>while the suffixes (KS, KM, KL) describe the size configuration (Small, Medium, Large) of the quantization method.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In local LLMs, a standard model is compressed (quantized) from 16-bit down to smaller sizes so it can fit into your computer&#8217;s RAM or GPU VRAM.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The Number (Bits per Weight)<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The lower the number, the smaller the file size and the faster it runs, but you lose a bit of the model&#8217;s intelligence.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Q3 (3-bit):<\/strong> Ultra-compressed. The file size is tiny and incredibly fast, but the model loses significant accuracy and might write broken sentences or gibberish. (Not recommended unless you have very low RAM).<\/li>\n\n\n\n<li><strong>Q4 (4-bit):<\/strong> <strong>The Industry Sweet Spot.<\/strong> It shrinks the model file size by roughly 70% while retaining about 95%+ of the original model&#8217;s intelligence.<\/li>\n\n\n\n<li><strong>Q5 (5-bit):<\/strong> <strong>The Best Balance.<\/strong> Slightly larger file size than Q4, but recovers almost all precision loss. Highly recommended if your system has the extra room.<\/li>\n\n\n\n<li><strong>Q6 (6-bit):<\/strong> Near-perfect quality. It is mathematically identical to the unquantized model to the human eye, but the file size begins to get heavy.<\/li>\n\n\n\n<li><strong>Q8 (8-bit):<\/strong> Maximum quality quantization. Very heavy file sizes, but zero noticeable drop in intelligence.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">The Suffixes (K-Quant Sizes)<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">When using the modern &#8220;K-quant&#8221; method (like <code>Q4_K_M<\/code>), different parts of the model are compressed at different strengths. Critical &#8220;brain&#8221; layers get less compression, while less important layers get compressed heavily.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>S (Small \/ KS):<\/strong> Compresses almost all layers down to the target bit size. It results in the smallest file size possible for that number but sacrifices a tiny bit of intelligence.<\/li>\n\n\n\n<li><strong>M (Medium \/ KM):<\/strong> <strong>The Recommended Default.<\/strong> It keeps the most critical layers at a higher precision while compressing the rest. It provides the best performance-to-file-size ratio.<\/li>\n\n\n\n<li><strong>L (Large \/ KL):<\/strong> Compresses very few layers aggressively. It gives the absolute highest precision for that bit-level, but the file size is the largest.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"619\" height=\"426\" src=\"https:\/\/www.balajibandi.com\/wp-content\/uploads\/2026\/11\/image.png\" alt=\"\" class=\"wp-image-2954\" srcset=\"https:\/\/www.balajibandi.com\/wp-content\/uploads\/2026\/11\/image.png 619w, https:\/\/www.balajibandi.com\/wp-content\/uploads\/2026\/11\/image-300x206.png 300w\" sizes=\"auto, (max-width: 619px) 100vw, 619px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><code>--mmproj<\/code> flag<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">gemma-4-E2B-it-heretic<mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-red-color\">-mmproj<\/mark>.gguf<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Images:<\/strong> A paperclip\/image icon will unlock next to your chat box in the default browser GUI allowing you to drag-and-drop or upload photos.<\/li>\n\n\n\n<li><strong>Audio:<\/strong> If your frontend wrapper supports microphone inputs, it will now successfully forward audio binaries or base64 sound chunks over the server&#8217;s API endpoints without crashing out.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">&#8211;mmproj ~\/models\/gemma-4-E2B-it-heretic-mmproj.gguf<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You can use on GUI image and Audio for the Local LLM to review and deliver the outcome<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"734\" height=\"357\" src=\"https:\/\/www.balajibandi.com\/wp-content\/uploads\/2026\/11\/image-2.png\" alt=\"\" class=\"wp-image-2958\" srcset=\"https:\/\/www.balajibandi.com\/wp-content\/uploads\/2026\/11\/image-2.png 734w, https:\/\/www.balajibandi.com\/wp-content\/uploads\/2026\/11\/image-2-300x146.png 300w, https:\/\/www.balajibandi.com\/wp-content\/uploads\/2026\/11\/image-2-703x342.png 703w\" sizes=\"auto, (max-width: 734px) 100vw, 734px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Example i use like below to get best outcome<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">.\/build\/bin\/llama-server<br>-m ~\/models\/gemma-4-E2B-it-heretic-Q5_K_M.gguf<br>&#8211;mmproj ~\/models\/gemma-4-E2B-it-heretic-mmproj.gguf<br>-a &#8220;BB5&#8221;<br>-c 131000<br>-ngl 80<br>&#8211;parallel 1<br>-t 16<br>-tb 32<br>-b 2048<br>-ub 1024<br>&#8211;flash-attn auto<br>&#8211;cache-type-k q4_0<br>&#8211;cache-type-v q4_0<br>&#8211;no-mmap<br>&#8211;host 0.0.0.0<br>&#8211;port 8080<br>&#8211;log-file \/var\/log\/llama_server.log<br>&#8211;log-colors on<br>&#8211;metrics<br>&#8211;context-shift<br>&#8211;fit on<br>&#8211;fit-ctx 16384<br>&#8211;keep 1<br>-n -1<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">i get below token process speed.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"787\" height=\"56\" src=\"https:\/\/www.balajibandi.com\/wp-content\/uploads\/2026\/11\/image-1.png\" alt=\"\" class=\"wp-image-2957\" srcset=\"https:\/\/www.balajibandi.com\/wp-content\/uploads\/2026\/11\/image-1.png 787w, https:\/\/www.balajibandi.com\/wp-content\/uploads\/2026\/11\/image-1-300x21.png 300w, https:\/\/www.balajibandi.com\/wp-content\/uploads\/2026\/11\/image-1-759x54.png 759w, https:\/\/www.balajibandi.com\/wp-content\/uploads\/2026\/11\/image-1-703x50.png 703w\" sizes=\"auto, (max-width: 787px) 100vw, 787px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Happy Labbing&#8230;&#8230;&#8230;&#8230;&#8230;..!<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>I have been working lot these days to test Local LLM that fits and work as smooth as possible on my Legacy hardware I am running llama.cpp with gguff models as below for testing : Llama-3.2-3B-Instruct-Q4_K_M.ggufQwen3-8B-Q4_K_M.ggufQwen3.5-2b-Kimi-and-Opus-Distillation.Q8_0.ggufTwIL-LM3-Pro-Q4_K_M.ggufgemma-3-1b-it-f16.ggufgemma-4-E2B-it-heretic-Q4_K_M.ggufgemma-4-E2B-it-heretic-Q5_K_M.ggufgemma-4-E2B-it-heretic-mmproj-F16.ggufgemma4-coding-Q4_K_M.gguf So the question come to me what the meaning after the model we see K, M and some numbers [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[46],"tags":[],"class_list":["post-2953","post","type-post","status-publish","format-standard","hentry","category-ai"],"_links":{"self":[{"href":"https:\/\/www.balajibandi.com\/index.php?rest_route=\/wp\/v2\/posts\/2953","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.balajibandi.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.balajibandi.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.balajibandi.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.balajibandi.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2953"}],"version-history":[{"count":3,"href":"https:\/\/www.balajibandi.com\/index.php?rest_route=\/wp\/v2\/posts\/2953\/revisions"}],"predecessor-version":[{"id":2959,"href":"https:\/\/www.balajibandi.com\/index.php?rest_route=\/wp\/v2\/posts\/2953\/revisions\/2959"}],"wp:attachment":[{"href":"https:\/\/www.balajibandi.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2953"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.balajibandi.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2953"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.balajibandi.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2953"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}