<p><strong>Load → Quantize → Fine-tune → Export</strong> — All in One Line</p>
<p>We’re excited to announce QuantLLM v2.0 — a complete redesign focused on simplicity, performance, and developer experience. This release transforms LLM quantization from a complex multi-step process into a single, intuitive workflow.</p>
<pre><code class="language-python"><span class="hljs-comment"># Recommended</span>
pip install git+https://github.com/codewithdark-git/QuantLLM.git

<span class="hljs-keyword">from</span> quantllm <span class="hljs-keyword">import</span> turbo

model = turbo(<span class="hljs-string">&quot;mistralai/Mistral-7B&quot;</span>)
model.generate(<span class="hljs-string">&quot;Hello!&quot;</span>)
model.finetune(data, epochs=<span class="hljs-number">3</span>)
<span class="hljs-comment"># model.export(&quot;gguf&quot;, quantization=&quot;Q4_K_M&quot;)</span>
model.push(<span class="hljs-string">&quot;QuantLLM/Mistral-7B-4bit-gguf&quot;</span>, <span class="hljs-built_in">format</span>=<span class="hljs-string">&quot;gguf&quot;</span>) <span class="hljs-comment"># format &#x27;gguf&#x27;, &#x27;onnx&#x27;, &#x27;mlx&#x27;, &#x27;safetensors&#x27;</span>
</code></pre>
<h3>🧠 45+ Model Architectures</h3>
<p>Llama 2/3, Mistral, Mixtral, Qwen 1/2, Phi 1/2/3, Gemma, Falcon, DeepSeek, Yi, StarCoder, ChatGLM, InternLM, Baichuan, StableLM, BLOOM, OPT, MPT, GPT-NeoX…</p>
<h3>📦 Multi-Format Export</h3>
<table>
<thead>
<tr>
<th>Format</th>
<th>Use Case</th>
<th>Command</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>GGUF</strong></td>
<td>llama.cpp, Ollama, LM Studio</td>
<td><code>model.export(&quot;gguf&quot;)</code></td>
</tr>
<tr>
<td><strong>ONNX</strong></td>
<td>ONNX Runtime, TensorRT</td>
<td><code>model.export(&quot;onnx&quot;)</code></td>
</tr>
<tr>
<td><strong>MLX</strong></td>
<td>Apple Silicon (M1/M2/M3/M4)</td>
<td><code>model.export(&quot;mlx&quot;)</code></td>
</tr>
<tr>
<td><strong>SafeTensors</strong></td>
<td>HuggingFace</td>
<td><code>model.export(&quot;safetensors&quot;)</code></td>
</tr>
</tbody>
</table>
<h3>GGUF Quantization Types</h3>
<table>
<thead>
<tr>
<th>Type</th>
<th>Bits</th>
<th>Quality</th>
<th>Use Case</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>Q2_K</code></td>
<td>2-bit</td>
<td>🔴 Low</td>
<td>Minimum size</td>
</tr>
<tr>
<td><code>Q3_K_M</code></td>
<td>3-bit</td>
<td>🟠 Fair</td>
<td>Very constrained</td>
</tr>
<tr>
<td><code>Q4_K_M</code></td>
<td>4-bit</td>
<td>🟢 Good</td>
<td><strong>Recommended</strong> ⭐</td>
</tr>
<tr>
<td><code>Q5_K_M</code></td>
<td>5-bit</td>
<td>🟢 High</td>
<td>Quality-focused</td>
</tr>
<tr>
<td><code>Q6_K</code></td>
<td>6-bit</td>
<td>🔵 Very High</td>
<td>Near-original</td>
</tr>
<tr>
<td><code>Q8_0</code></td>
<td>8-bit</td>
<td>🔵 Excellent</td>
<td>Best quality</td>
</tr>
</tbody>
</table>
<h3>Load GGUF Models</h3>
<pre><code class="language-python"><span class="hljs-keyword">from</span> quantllm <span class="hljs-keyword">import</span> TurboModel

model = TurboModel.from_gguf(
    <span class="hljs-string">&quot;TheBloke/Llama-2-7B-Chat-GGUF&quot;</span>, 
    filename=<span class="hljs-string">&quot;llama-2-7b-chat.Q4_K_M.gguf&quot;</span>
)
<span class="hljs-built_in">print</span>(model.generate(<span class="hljs-string">&quot;Hello!&quot;</span>))
</code></pre>
<h3>Made with 🧡 by <a href="https://github.com/codewithdark-git" target="_blank" rel="noopener nofollow">Dark Coder</a></h3>
<p><a href="https://github.com/codewithdark-git/QuantLLM" target="_blank" rel="noopener nofollow">⭐ Star on GitHub</a> •
<a href="https://github.com/codewithdark-git/QuantLLM/issues" target="_blank" rel="noopener nofollow">🐛 Report Bug</a> •
<a href="https://github.com/sponsors/codewithdark-git" target="_blank" rel="noopener nofollow">💖 Sponsor</a></p>
<p><strong>Happy Quantizing! 🚀</strong></p>

Get 1% better daily. Development resource sharing made easy. Post, comment, or upvote to show your contribution and support other developers! 🙌

⚠️ Please do not post any promotional resources.

Dev Squad

QuantLLM v2.0 is a Python library that simplifies LLM quantization, fine-tuning, and export into a single-line workflow. It supports 45+ model architectures including Llama, Mistral, and Qwen, and can export to multiple formats (GGUF, ONNX, MLX, SafeTensors) with various quantization levels from 2-bit to 8-bit. The library provides a unified API for loading models, fine-tuning them, and exporting to different deployment targets like llama.cpp, ONNX Runtime, and Apple Silicon.

The Ultra-Fast LLM Quantization & Export Library