A primer on key AI LLM concepts — KV cache, weights, quantization, and how inference actually works
tomshardware.com/tech-industry/artificial-intelligence/a-primer-on-key-ai-llm-concepts-kv-cache-weights-quantization-and-how-inference-actually-works
Paid articlePaywallStory Details
| Last Updated | |
| Total Articles | 2 |
Main Topics
individual artificial neural network considered intelligent
computing component that transparently stores data so that future requests for that data can be served faster
Similar to This
Previous
How flash storage can cut GPU waste from KV-cache recomputation: LLM inference can waste up to 60% of GPU compute recomputing context. WEKA explains how KV-cache ...
venturebeat.com/ai/up-to-60-of-gpu-compute-goes-to-recomputing-context-flash-storage-can-put-that-capacity-back-to-work
github.com May 17
GitHub - RedToasty/llama.cpp_qts: Fixing --split-mode tensor, with different KV cache quantization types.
github.com/RedToasty/llama.cpp_qts
github.com Apr 07
TurboQuant - Extreme KV Cache Quantization · ggml-org/llama.cpp · Discussion #20969
github.com/ggml-org/llama.cpp/discussions/20969
github.com Apr 02
GitHub - jhammant/Turbo1bit: Turbo1Bit: Combining 1-bit LLM weights (Bonsai) with TurboQuant KV cache compression for maximum inference efficiency. 4 ...
github.com/jhammant/Turbo1bit
github.com Apr 01
GitHub - joshuagamboa/turboquant-apple-silicon: High-performance Rust integration for aggressive KV cache quantization on Apple Silicon GPUs (Metal) ...
github.com/joshuagamboa/turboquant-apple-silicon
github.com Mar 31
GitHub - quantumaikr/TurboQuant.cpp: turboquant.cpp: turboquant.cpp. Contribute to quantumaikr/TurboQuant.cpp development by creating an account on GitHub.
github.com/quantumaikr/TurboQuant.cpp
What Google's TurboQuant can and can't do for AI's spiraling cost: Google's real-time quantization could be important for running local AI. Here's why.
zdnet.com/article/what-googles-turboquant-can-and-cant-do-for-ais-spiraling-cost/
github.com Mar 31
GitHub - Memoriant/dgx-spark-kv-cache-benchmark: KV cache quantization benchmarks on NVIDIA DGX Spark GB10 — three novel findings including the dequantization ...
github.com/Memoriant/dgx-spark-kv-cache-benchmark
github.com Mar 24
GitHub - cenconq25/delta-compress-llm: Exploiting temporal coherence in LLM inference-- delta encoding for KV cache compression and weight-skip prediction ...
github.com/cenconq25/delta-compress-llm
github.com Mar 24
GitHub - samfurr/foveated_kv: Importance-adaptive mixed-precision KV cache compression for LLM inference on Apple Silicon.
github.com/samfurr/foveated_kv
Next