GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Deduplicated dequantization code by JohannesGaessler · Pull Request #1453 · ggml-org/llama.cpp

Conversation

@JohannesGaessler

@JohannesGaessler JohannesGaessler commented May 14, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

For my GPU acceleration PR #1412 I used a template to decouple the code for matrix vector multiplication and dequantization. This PR applies the same principle to the dequantization during prompt processing: the same dequantization kernels can be reused with a different template. This allows for the deduplication of CUDA code to ensure consistency. As a side effect the new kernels are also slightly faster on my hardware.

Performance numbers for perplexity calculation on the first 100 lines of wikitext:

GPU Model ms/t master ms/t PR
RTX 3090 7b q4_0 3.61 3.60
RTX 3090 7b q4_1 3.82 3.74
RTX 3090 7b q5_0 3.75 3.64
RTX 3090 7b q5_1 3.75 3.67
RTX 3090 7b q8_0 4.29 4.05
RTX 3090 7b f16 4.91 4.86
GTX 1070 7b q4_0 9.78 7.39
GTX 1070 7b q4_1 9.86 7.67
GTX 1070 7b q5_0 10.01 7.63
GTX 1070 7b q5_1 10.12 7.79
GTX 1070 7b q8_0 11.88 8.28
GTX 1070 7b f16 10.62 10.69

I will also add GTX 1070 numbers once I have them. Done.

The goal of this PR is not to optimize performance. The goal is to simplify the code base to allow for easier development. If the new kernels don't cause a performance regression I consider that good enough.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
ice0, VitorNaidek, and sevenreasons reacted with thumbs up emoji
All reactions
  • 👍 3 reactions
@JohannesGaessler
Deduplicated dequantization code
0e8ca77
@JohannesGaessler JohannesGaessler added the refactoring Refactoring label May 14, 2023
@ggerganov

ggerganov commented May 14, 2023

Copy link
Copy Markdown
Member

I don't see performance degradation on RTX 4080
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ggerganov
ggerganov requested a review from slaren May 14, 2023 18:17
slaren
slaren approved these changes May 14, 2023
View reviewed changes
@ggerganov
ggerganov merged commit eb36362 into ggml-org:master May 14, 2023
@JohannesGaessler JohannesGaessler mentioned this pull request May 14, 2023
OpenCL dequant_mul_mat #1459
Merged
@Bearsaerker Bearsaerker mentioned this pull request Mar 12, 2025
Eval bug: Gemma 3 extremly slow prompt processing when using quantized kv cache. #12352
Closed
Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026
@JohannesGaessler
cuda : deduplicated dequantization code (ggml-org#1453)
1c1f5d6
phuongncn pushed a commit to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4 that referenced this pull request Apr 28, 2026
@JohannesGaessler
cuda : deduplicated dequantization code (ggml-org#1453)
c7ef8d3
ljubomirj pushed a commit to ljubomirj/llama.cpp that referenced this pull request May 6, 2026
@JohannesGaessler
cuda : deduplicated dequantization code (ggml-org#1453)
2d54144
AlexiAlp pushed a commit to minghaop/llama.cpp that referenced this pull request Jun 2, 2026
@JohannesGaessler
cuda : deduplicated dequantization code (ggml-org#1453)
5409e4c
AlexiAlp pushed a commit to minghaop/llama.cpp that referenced this pull request Jun 2, 2026
@JohannesGaessler
cuda : deduplicated dequantization code (ggml-org#1453)
3ba27b7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@slaren slaren slaren approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

refactoring Refactoring

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@JohannesGaessler @ggerganov @slaren
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.