GitHub - RedToasty/llama.cpp_qts: Fixing --split-mode tensor, with different KV cache quantization types.

Story Details

Last Updated
Total Articles	1

Main Topics

Llama (language model)

#llama-language-model

large language model by Meta AI

Cache (computing)

#cache-computing

computing component that transparently stores data so that future requests for that data can be served faster

Large language model

#large-language-model

language model built with very large amounts of texts

Similar to This

github.com Oct 17

CUDA: faster MMVQ for low-bit types by am17an · Pull Request #30517 · ggml-org/llama.cpp

github.com/ggml-org/llama.cpp/pull/30517

0 similar articles

github.com Oct 08

ggml-cuda: assign four GDN state columns per warp by SongXiaoXi · Pull Request #30087 · ggml-org/llama.cpp

github.com/ggml-org/llama.cpp/pull/30087

0 similar articles

github.com Oct 06

RPC: add `-sm tensor` by am17an · Pull Request #26610 · ggml-org/llama.cpp: LLM inference in C/C++. Contribute to ggml-org/llama.cpp development by creating an ...

github.com/ggml-org/llama.cpp/pull/26610

0 similar articles

github.com Sep 03

GitHub - mihailescu2m/llama.cpp: LLM inference in C/C++: LLM inference in C/C++. Contribute to mihailescu2m/llama.cpp development by creating an account on ...

github.com/mihailescu2m/llama.cpp

0 similar articles

github.com Aug 27

llama: model_loader: add TENSOR_READ_LAZY by ngxson · Pull Request #27794 · ggml-org/llama.cpp

github.com/ggml-org/llama.cpp/pull/27794

0 similar articles

github.com Aug 23

GitHub - oussemah/llama.cpp at moe-cache-ousemma: LLM inference in C/C++. Contribute to oussemah/llama.cpp development by creating an account on GitHub.

github.com/oussemah/llama.cpp/tree/moe-cache-ousemma

0 similar articles

github.com Aug 21

llama: lfm2/lfm2moe: enable tensor split by cdanis · Pull Request #26993 · ggml-org/llama.cpp

github.com/ggml-org/llama.cpp/pull/26993

0 similar articles

github.com Aug 15

[FR] Make GGML_CUDA_FA_ALL_QUANTS the default or add runtime warning for missing quantized FA kernels · Issue #24485 · ggml-org/llama.cpp

github.com/ggml-org/llama.cpp/issues/24485

0 similar articles

github.com Jul 24

GitHub - fewtarius/CachyLLama: LLM inference in C/C++: LLM inference in C/C++. Contribute to fewtarius/CachyLLama development by creating an account on GitHub.

github.com/fewtarius/CachyLLama

0 similar articles

github.com May 15

What is wrong with llama.cpp release page?

github.com/ggml-org/llama.cpp/releases

0 similar articles

github.com Apr 09

ggml: backend-agnostic tensor parallelism (experimental) by JohannesGaessler · Pull Request #19378 · ggml-org/llama.cpp

github.com/ggml-org/llama.cpp/pull/19378

0 similar articles

github.com Apr 07

kv-cache : support attention rotation for heterogeneous iSWA by ggerganov · Pull Request #21513 · ggml-org/llama.cpp

github.com/ggml-org/llama.cpp/pull/21513

0 similar articles

github.com Apr 07

TurboQuant - Extreme KV Cache Quantization · ggml-org/llama.cpp · Discussion #20969

github.com/ggml-org/llama.cpp/discussions/20969

0 similar articles

github.com Apr 06

ggml: add Q1_0 1-bit quantization support (CPU) by khosravipasha · Pull Request #21273 · ggml-org/llama.cpp

github.com/ggml-org/llama.cpp/pull/21273

0 similar articles

github.com Apr 03

(revert) kv-cache : do not quantize SWA KV cache by ggerganov · Pull Request #21332 · ggml-org/llama.cpp

github.com/ggml-org/llama.cpp/pull/21332

0 similar articles

github.com Apr 01

llama : rotate activations for better quantization by ggerganov · Pull Request #21038 · ggml-org/llama.cpp

github.com/ggml-org/llama.cpp/pull/21038

0 similar articles

github.com Apr 01

GitHub - jimliddle/turboquant-amd-vulkan: A TurboQuant implementation with Llama.cpp for AMD with Vulkan runtime

github.com/jimliddle/turboquant-amd-vulkan

0 similar articles

github.com Mar 24

GitHub - cenconq25/delta-compress-llm: Exploiting temporal coherence in LLM inference-- delta encoding for KV cache compression and weight-skip prediction ...

github.com/cenconq25/delta-compress-llm

0 similar articles

github.com Mar 16

How do i specify which gpu to use for kv cache? How to offload expert tensors to specific gpu? · ggml-org/llama.cpp · Discussion #20642

github.com/ggml-org/llama.cpp/discussions/20642

0 similar articles

github.com Mar 04

ggml : add NVFP4 quantization type support by richarddd · Pull Request #19769 · ggml-org/llama.cpp

github.com/ggml-org/llama.cpp/pull/19769

0 similar articles