GitHub - samfurr/foveated_kv: Importance-adaptive mixed-precision KV cache compression for LLM inference on Apple Silicon.

Story Details

Last Updated
Total Articles	1

Main Topics

X (social network)

#x-social-network

American social networking service founded in 2006

Cache (computing)

#cache-computing

computing component that transparently stores data so that future requests for that data can be served faster

Apple silicon

#apple-silicon

series of SoC and SiP processors from Apple

Similar to This

tomshardware.com

Credibility C

A primer on key AI LLM concepts — KV cache, weights, quantization, and how inference actually works

tomshardware.com/tech-industry/artificial-intelligence/a-primer-on-key-ai-llm-concepts-kv-cache-weights-quantization-and-how-inference-actually-works

Paid articlePaywall

0 similar articles

0 4

VentureBeat

Oct 16

How flash storage can cut GPU waste from KV-cache recomputation: LLM inference can waste up to 60% of GPU compute recomputing context. WEKA explains how KV-cache ...

venturebeat.com/ai/up-to-60-of-gpu-compute-goes-to-recomputing-context-flash-storage-can-put-that-capacity-back-to-work

0 similar articles

github.com Aug 15

[FR] Make GGML_CUDA_FA_ALL_QUANTS the default or add runtime warning for missing quantized FA kernels · Issue #24485 · ggml-org/llama.cpp

github.com/ggml-org/llama.cpp/issues/24485

0 similar articles

github.com May 22

Asymmetric KV q8/q4 cache to preserve precision while reducing memory usage · ggml-org/llama.cpp · Discussion #23470

github.com/ggml-org/llama.cpp/discussions/23470

0 similar articles

github.com Apr 11

GitHub - ahb-sjsu/turboquant-pro: Embedding & KV cache compression for LLMs and vector databases. PCA-Matryoshka + TurboQuant: 27x compression at 99 ...

github.com/ahb-sjsu/turboquant-pro

0 similar articles

github.com Apr 05

GitHub - AlphaWaveSystems/tqai: TurboQuant KV cache compression for local LLM inference — 80% memory savings, near-zero quality loss on 8B+ models ...

github.com/AlphaWaveSystems/tqai

0 similar articles

github.com Apr 02

GitHub - jhammant/Turbo1bit: Turbo1Bit: Combining 1-bit LLM weights (Bonsai) with TurboQuant KV cache compression for maximum inference efficiency. 4 ...

github.com/jhammant/Turbo1bit

0 similar articles

github.com Apr 02

GitHub - matt-k-wong/turboquant-mlx-full: Contribute to matt-k-wong/turboquant-mlx-full development by creating an account on GitHub.

github.com/matt-k-wong/turboquant-mlx-full

0 similar articles

github.com Apr 01

GitHub - joshuagamboa/turboquant-apple-silicon: High-performance Rust integration for aggressive KV cache quantization on Apple Silicon GPUs (Metal) ...

github.com/joshuagamboa/turboquant-apple-silicon

0 similar articles

github.com Mar 31

GitHub - quantumaikr/TurboQuant.cpp: turboquant.cpp: turboquant.cpp. Contribute to quantumaikr/TurboQuant.cpp development by creating an account on GitHub.

github.com/quantumaikr/TurboQuant.cpp

0 similar articles

github.com Apr 01

GitHub - SharpAI/SwiftLM: ⚡ Native Swift LLM inference server for Apple Silicon. OpenAI-compatible API, SSD streaming for 100B+ MoE models ...

github.com/SharpAI/SwiftLM

0 similar articles

github.com Mar 27

GitHub - e1n00r/tinyserve: Run 20B-400B+ MoE models on 8GB consumer GPUs. 3-tier expert caching (SSD→RAM→GPU), native MXFP4/GGUF Q4_K compute ...

github.com/e1n00r/tinyserve

0 similar articles

github.com Mar 26

GitHub - gs-ai/mlm-memory: A functionally operational, mathematically unhinged system for achieving 10× effective memory amplification on Apple Silicon using ...

github.com/gs-ai/mlm-memory

0 similar articles

github.com Mar 24

GitHub - cenconq25/delta-compress-llm: Exploiting temporal coherence in LLM inference-- delta encoding for KV cache compression and weight-skip prediction ...

github.com/cenconq25/delta-compress-llm

0 similar articles