TurboQuant - Extreme KV Cache Quantization · ggml-org/llama.cpp · Discussion #20969

github.com/ggml-org/llama.cpp/discussions/20969

Story Details

Last Updated
Total Articles	1

Main Topics

Cache (computing)

#cache-computing

computing component that transparently stores data so that future requests for that data can be served faster

Llama (language model)

#llama-language-model

large language model by Meta AI

PPL Corporation

#ppl-corporation

American energy company

Similar to This

github.com Aug 27

feat: import qwen4exp (Qwen3.8-Flash-Next) support from upstream PR #27742 by giveen · Pull Request #324 · TheTom/llama-cpp-turboquant

github.com/TheTom/llama-cpp-turboquant/pull/324

1 similar article

github.com Aug 15

[FR] Make GGML_CUDA_FA_ALL_QUANTS the default or add runtime warning for missing quantized FA kernels · Issue #24485 · ggml-org/llama.cpp

github.com/ggml-org/llama.cpp/issues/24485

0 similar articles

github.com Jul 24

GitHub - fewtarius/CachyLLama: LLM inference in C/C++: LLM inference in C/C++. Contribute to fewtarius/CachyLLama development by creating an account on GitHub.

github.com/fewtarius/CachyLLama

0 similar articles

github.com May 17

GitHub - RedToasty/llama.cpp_qts: Fixing --split-mode tensor, with different KV cache quantization types.

github.com/RedToasty/llama.cpp_qts

0 similar articles

github.com Apr 07

kv-cache : support attention rotation for heterogeneous iSWA by ggerganov · Pull Request #21513 · ggml-org/llama.cpp

github.com/ggml-org/llama.cpp/pull/21513

0 similar articles

github.com Apr 06

ggml: add Q1_0 1-bit quantization support (CPU) by khosravipasha · Pull Request #21273 · ggml-org/llama.cpp

github.com/ggml-org/llama.cpp/pull/21273

0 similar articles

github.com Apr 03

GitHub - TheTom/llama-cpp-turboquant: LLM inference in C/C++: LLM inference in C/C++. Contribute to TheTom/llama-cpp-turboquant development by creating an ...

github.com/TheTom/llama-cpp-turboquant

0 similar articles

github.com Apr 03

(revert) kv-cache : do not quantize SWA KV cache by ggerganov · Pull Request #21332 · ggml-org/llama.cpp

github.com/ggml-org/llama.cpp/pull/21332

0 similar articles

github.com Apr 01

llama : rotate activations for better quantization by ggerganov · Pull Request #21038 · ggml-org/llama.cpp

github.com/ggml-org/llama.cpp/pull/21038

0 similar articles

github.com Apr 01

GitHub - jimliddle/turboquant-amd-vulkan: A TurboQuant implementation with Llama.cpp for AMD with Vulkan runtime

github.com/jimliddle/turboquant-amd-vulkan

0 similar articles

github.com Apr 01

GitHub - joshuagamboa/turboquant-apple-silicon: High-performance Rust integration for aggressive KV cache quantization on Apple Silicon GPUs (Metal) ...

github.com/joshuagamboa/turboquant-apple-silicon

0 similar articles

github.com Mar 31

GitHub - MartinCrespoC/QuantumLeap---Llama.cpp-TurboQuant: 🚀 Run any LLM on any hardware. 130% faster MoE inference with ExpertFlow + TurboQuant KV compression ...

github.com/MartinCrespoC/QuantumLeap---Llama.cpp-TurboQuant

0 similar articles

github.com Mar 31

GitHub - quantumaikr/TurboQuant.cpp: turboquant.cpp: turboquant.cpp. Contribute to quantumaikr/TurboQuant.cpp development by creating an account on GitHub.

github.com/quantumaikr/TurboQuant.cpp

0 similar articles

ZDNET

Credibility A−

Mar 31

What Google's TurboQuant can and can't do for AI's spiraling cost: Google's real-time quantization could be important for running local AI. Here's why.

zdnet.com/article/what-googles-turboquant-can-and-cant-do-for-ais-spiraling-cost/

0 similar articles

github.com Mar 27

GitHub - cksac/turboquant-model: Contribute to cksac/turboquant-model development by creating an account on GitHub.

github.com/cksac/turboquant-model

0 similar articles

github.com Mar 16

How do i specify which gpu to use for kv cache? How to offload expert tensors to specific gpu? · ggml-org/llama.cpp · Discussion #20642

github.com/ggml-org/llama.cpp/discussions/20642

0 similar articles

github.com Mar 08

What's the difference between batch-size and ubatch-size? · ggml-org/llama.cpp · Discussion #6328

github.com/ggml-org/llama.cpp/discussions/6328

0 similar articles

github.com Mar 04

ggml : add NVFP4 quantization type support by richarddd · Pull Request #19769 · ggml-org/llama.cpp

github.com/ggml-org/llama.cpp/pull/19769

0 similar articles