GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — ggml : remove bit shuffling by ggerganov · Pull Request #1405 · ggml-org/llama.cpp

Conversation

@ggerganov

@ggerganov ggerganov commented May 11, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Close #1241

  • Drop Q4_2 support
  • Changed bit-order for Q4 and Q5 (breaking change)
  • Preplexity is perplexing as usual

New timings:

Model Measure F16 Q4_0 Q4_1 Q5_0 Q5_1 Q8_0
7B ms/tok @ 4th 128 50 54 75 83 75
7B ms/tok @ 8th 123 44 52 53 58 72
13B ms/tok @ 4th 332* 93 101 150 164 141
13B ms/tok @ 8th 308* 81 96 96 104 136
  • these numbers vary a lot since the model is on the 32GB limit of my MacBook

Old timings:

Model Measure F16 Q4_0 Q4_1 Q5_0 Q5_1 Q8_0
7B ms/tok @ 4th 128 56 61 91 95 75
7B ms/tok @ 8th 128 47 55 53 59 75
13B ms/tok @ 4th 239 104 113 176 185 141
13B ms/tok @ 8th 240 85 99 108 117 147

overall, all these numbers seem to have about +/- 10% variablility from run to run. not ideal benchmark, but not sure what else to do

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
digiwombat, Green-Sky, m0sh1x2, lin72h, mirek190, Crimsonfart, and lolxdmainkaisemaanlu reacted with thumbs up emoji
All reactions
  • 👍 7 reactions
@ggerganov
ggerganov force-pushed the remove-vzip-2 branch from fe7b8df to f9bbbe3 Compare May 11, 2023 17:06
@ggerganov
ggerganov marked this pull request as ready for review May 11, 2023 18:36
@ggerganov
ggerganov requested a review from sw May 11, 2023 18:36
@ggerganov ggerganov mentioned this pull request May 11, 2023
ggml : remove bit shuffling #1305
Closed
18 tasks done
ggerganov and others added 26 commits May 11, 2023 21:37
@ggerganov
ggml : remove Q4_0 bit shufling (ARM NEON)
5fa47bf
@ggerganov
ggml : remove Q4_1 bit shuffling (ARM NEON + reference)
844d2af
@ggerganov
ggml : nibbles_from_floats() + bytes_from_nibbles() (ARM NEON)
fd2a137
@ggerganov
ggml : remove Q4_2 bit shuffling (WIP, BROKEN)
9f3285f
@ggerganov
ggml : remove Q5_0 bit shuffling (ARM NEON)
aa78dfe
@ggerganov
ggml : 2x faster scalar implementations
b37a08f
@ggerganov
ggml : remove Q5_1 bit shuffling (ARM NEON + scalar)
292a778
@ggerganov
ggml : simplify scalar dot
caaacd5
@ggerganov
ggml : remove WASM SIMD bit shuffling + remove vzip for ARM 32-bit
0add640
@ggerganov
ggml : fix Q4_1 quantization
9472d0e
@ggerganov
ggml : update cuBLAS + normalize variable names
cdc9607
@ggerganov
ggml : remove Q4_2 mode
4bf1c8a
@ggerganov
ggml : minor formatting
b08c39b
@ggerganov
ggml : fix Q5_0 quantization
8367455
@ggerganov
scripts : add script for measuring the time per token
928d2f3
@sw @ggerganov
AVX implementations (#1370)
9e49d20
@ggerganov
ggml : uniform 5th bit extraction
489bd13
@ggerganov
llama : produce error upon loading old model files
d52172a
@ggerganov
llama : fix model magic/version write
09032e0
@ggerganov
ggml : speed-up Q5_0 + Q5_1 at 4 threads
b7ad385
@ggerganov
ggml : preserve old Q4 and Q5 formats
695f396
@ggerganov
ggml : simplify Q8_1 - no need for low / high sums anymore
582a39f
@ggerganov
ggml : fix Q8_0 and Q8_1 rounding
6680244
@ggerganov
Revert "AVX implementations (#1370)"
bd5e373 
This reverts commit 948d124.
@ggerganov
ggml : fix AVX2 implementation
5bc286a
@ggerganov
sha : update hashes for 7B and 13B
e038e01
@philpax

philpax commented May 12, 2023

Copy link
Copy Markdown

Cool to see this is merged, i'm slightly confused though, seems like the converter is still writing the old file version (1) ?
image

I think that's intentional - the converter converts the f16 model to a GGJT v1 f16 model, which is then quantised to a GGJT v2 qX_Y model. (The saver always writes version 2, from what I can see.)
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@LostRuins

LostRuins commented May 12, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Collaborator

Doesn't seem like it though, I don't see references where File version 2 is set during quantization either.

Edit: i'm wrong. It's set in write_magic()
philpax reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@TheBloke

TheBloke commented May 12, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Could someone please share the way of requantizing? <img alt="image" width="594" src="https://user-example https://huggingface.co/TheBloke/WizardLM-7B-uncensored-GGML/blob/main/README.md What should be done with WizardLM-7B-uncensored.q5_1.bin or which tool need to by used for https://huggingface.co/ehartford/WizardLM-13B-Uncensored which is the source model.

Check my repos again. I've re-quantised all my GGMLs using the latest code, in q4_0, q5_0, q5_1 and q8_0 variants. So no need to do it yourself unless you want to.
mirek190, mix1009, teristam, lin72h, Green-Sky, jeddyhhh, regstuff, christianwengert, Vuizur, Priestru, and 2 more reacted with thumbs up emoji zellski, zenyr, LostRuins, Crimsonfart, sep332, sweenzor, josemarcosrf, olinorwell, IDDT, clemdesr, and 3 more reacted with heart emoji
All reactions
  • 👍 12 reactions
  • ❤️ 13 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@wzgrx wzgrx mentioned this pull request May 13, 2023
llama.cpp: loading model from models\Chinese-LLaMA-7B\ggml-model-q4_0.bin error loading model: unknown (magic, version) combination: 67676a74, 00000002; is this really a GGML file? llama_init_from_file: failed to load model oobabooga/textgen#2020
Closed
1 task done
@ekryski ekryski mentioned this pull request May 13, 2023
Implement Together Computer's Red Pajama 3B Base/Chat model #1337
Closed
@philpax philpax mentioned this pull request May 13, 2023
Support the bit-shuffling changes from llama.cpp rustformers/llm#198
Closed
@mudler mudler mentioned this pull request May 13, 2023
⬆️ Update go-skynet/go-llama.cpp mudler/LocalAI#245
Merged
@sw sw mentioned this pull request May 13, 2023
ggml : add AVX support based on AVX2 code #1430
Merged
@nikhil-xb nikhil-xb mentioned this pull request May 17, 2023
can't use mmap because of ggml? zylon-ai/private-gpt#190
Closed
@valdesguefa valdesguefa mentioned this pull request May 17, 2023
Failed to execute script 'koboldcpp' due to unhandled exception! LostRuins/koboldcpp#180
Closed
@awinml awinml mentioned this pull request May 17, 2023
Not Compatible with Models quantized with updated llama.cpp q4 and q5 quantization. abetlen/llama-cpp-python#227
Closed
@ChaoticByte ChaoticByte mentioned this pull request May 18, 2023
Bump llama-cpp-python[server] from 0.1.48 to 0.1.50 ChaoticByte/Eucalyptus-Chat#2
Merged
@sw sw mentioned this pull request May 18, 2023
make issue on sbc odroid #482
Closed
@absadiki absadiki mentioned this pull request May 18, 2023
support new version of llamacpp absadiki/pyllamacpp#9
Open
@mconsidine mconsidine mentioned this pull request May 22, 2023
runtime error in example/server #1557
Closed
@absadiki absadiki mentioned this pull request May 22, 2023
Can't import vicuna models : (bad f16 value 5) absadiki/pyllamacpp#7
Closed
@matthiasgeihs matthiasgeihs mentioned this pull request Jun 2, 2023
llama.cpp: loading model ......terminate called after throwing an instance of 'std::runtime_error' abetlen/llama-cpp-python#303
Closed
@Entretoize Entretoize mentioned this pull request Jun 15, 2023
train-text-from-scratch.exe stop after "begin training" (tensor->src0 is null) #1869
Closed
@SkibaSAY SkibaSAY mentioned this pull request Jul 7, 2023
[User] train-text-from-scratch.exe stop before"begin training" #2131
Closed
@Green-Sky Green-Sky mentioned this pull request Aug 3, 2023
GGUF file format specification ggml-org/ggml#302
Merged
@ikawrakow ikawrakow mentioned this pull request Aug 20, 2023
More efficient HellaSwag implementation #2677
Merged
@xinchun-wang xinchun-wang mentioned this pull request Sep 1, 2023
Add support for AMX instructions (bf16 and/or int8) #2555
Closed
4 tasks done
BobbyKay1
BobbyKay1 approved these changes Dec 10, 2023
View reviewed changes
@Bearsaerker Bearsaerker mentioned this pull request Mar 12, 2025
Eval bug: Gemma 3 extremly slow prompt processing when using quantized kv cache. #12352
Closed
Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026
@ggerganov @sw
ggml : remove bit shuffling (ggml-org#1405)
703b216 
* ggml : remove Q4_0 bit shufling (ARM NEON)

* ggml : remove Q4_1 bit shuffling (ARM NEON + reference)

* ggml : nibbles_from_floats() + bytes_from_nibbles() (ARM NEON)

* ggml : remove Q4_2 bit shuffling (WIP, BROKEN)

* ggml : remove Q5_0 bit shuffling (ARM NEON)

* ggml : 2x faster scalar implementations

* ggml : remove Q5_1 bit shuffling (ARM NEON + scalar)

* ggml : simplify scalar dot

* ggml : remove WASM SIMD bit shuffling + remove vzip for ARM 32-bit

* ggml : fix Q4_1 quantization

* ggml : update cuBLAS + normalize variable names

* ggml : remove Q4_2 mode

* ggml : minor formatting

* ggml : fix Q5_0 quantization

* scripts : add script for measuring the time per token

* AVX implementations (ggml-org#1370)

* ggml : uniform 5th bit extraction

* llama : produce error upon loading old model files

* llama : fix model magic/version write

* ggml : speed-up Q5_0 + Q5_1 at 4 threads

* ggml : preserve old Q4 and Q5 formats

* ggml : simplify Q8_1 - no need for low / high sums anymore

* ggml : fix Q8_0 and Q8_1 rounding

* Revert "AVX implementations (ggml-org#1370)"

This reverts commit 948d124.

* ggml : fix AVX2 implementation

* sha : update hashes for 7B and 13B

* readme : update timings + remove warning banner

* llama : update v2 PR number to 1405

* ggml : fix WASM comments

* ggml : back to original bit order

* readme : add note that Q4 and Q5 have been changed

* llama : fix return for unknown version

---------

Co-authored-by: Stephan Walter <[email protected]>
phuongncn pushed a commit to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4 that referenced this pull request Apr 28, 2026
@ggerganov @sw
ggml : remove bit shuffling (ggml-org#1405)
65f22db 
* ggml : remove Q4_0 bit shufling (ARM NEON)

* ggml : remove Q4_1 bit shuffling (ARM NEON + reference)

* ggml : nibbles_from_floats() + bytes_from_nibbles() (ARM NEON)

* ggml : remove Q4_2 bit shuffling (WIP, BROKEN)

* ggml : remove Q5_0 bit shuffling (ARM NEON)

* ggml : 2x faster scalar implementations

* ggml : remove Q5_1 bit shuffling (ARM NEON + scalar)

* ggml : simplify scalar dot

* ggml : remove WASM SIMD bit shuffling + remove vzip for ARM 32-bit

* ggml : fix Q4_1 quantization

* ggml : update cuBLAS + normalize variable names

* ggml : remove Q4_2 mode

* ggml : minor formatting

* ggml : fix Q5_0 quantization

* scripts : add script for measuring the time per token

* AVX implementations (ggml-org#1370)

* ggml : uniform 5th bit extraction

* llama : produce error upon loading old model files

* llama : fix model magic/version write

* ggml : speed-up Q5_0 + Q5_1 at 4 threads

* ggml : preserve old Q4 and Q5 formats

* ggml : simplify Q8_1 - no need for low / high sums anymore

* ggml : fix Q8_0 and Q8_1 rounding

* Revert "AVX implementations (ggml-org#1370)"

This reverts commit 948d124.

* ggml : fix AVX2 implementation

* sha : update hashes for 7B and 13B

* readme : update timings + remove warning banner

* llama : update v2 PR number to 1405

* ggml : fix WASM comments

* ggml : back to original bit order

* readme : add note that Q4 and Q5 have been changed

* llama : fix return for unknown version

---------

Co-authored-by: Stephan Walter <[email protected]>
phuongncn pushed a commit to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4 that referenced this pull request Apr 28, 2026
@saood06
Vibe coded script + constants from mainline + pip requirements (ggml-…
2161ee0 
…org#1405)
ljubomirj pushed a commit to ljubomirj/llama.cpp that referenced this pull request May 6, 2026
@ggerganov @sw
ggml : remove bit shuffling (ggml-org#1405)
7385ee4 
* ggml : remove Q4_0 bit shufling (ARM NEON)

* ggml : remove Q4_1 bit shuffling (ARM NEON + reference)

* ggml : nibbles_from_floats() + bytes_from_nibbles() (ARM NEON)

* ggml : remove Q4_2 bit shuffling (WIP, BROKEN)

* ggml : remove Q5_0 bit shuffling (ARM NEON)

* ggml : 2x faster scalar implementations

* ggml : remove Q5_1 bit shuffling (ARM NEON + scalar)

* ggml : simplify scalar dot

* ggml : remove WASM SIMD bit shuffling + remove vzip for ARM 32-bit

* ggml : fix Q4_1 quantization

* ggml : update cuBLAS + normalize variable names

* ggml : remove Q4_2 mode

* ggml : minor formatting

* ggml : fix Q5_0 quantization

* scripts : add script for measuring the time per token

* AVX implementations (ggml-org#1370)

* ggml : uniform 5th bit extraction

* llama : produce error upon loading old model files

* llama : fix model magic/version write

* ggml : speed-up Q5_0 + Q5_1 at 4 threads

* ggml : preserve old Q4 and Q5 formats

* ggml : simplify Q8_1 - no need for low / high sums anymore

* ggml : fix Q8_0 and Q8_1 rounding

* Revert "AVX implementations (ggml-org#1370)"

This reverts commit e73674401b3f75e1ea02291d6e69d2b943991598.

* ggml : fix AVX2 implementation

* sha : update hashes for 7B and 13B

* readme : update timings + remove warning banner

* llama : update v2 PR number to 1405

* ggml : fix WASM comments

* ggml : back to original bit order

* readme : add note that Q4 and Q5 have been changed

* llama : fix return for unknown version

---------

Co-authored-by: Stephan Walter <[email protected]>
AlexiAlp pushed a commit to minghaop/llama.cpp that referenced this pull request Jun 2, 2026
@ggerganov @sw
ggml : remove bit shuffling (ggml-org#1405)
8b14616 
* ggml : remove Q4_0 bit shufling (ARM NEON)

* ggml : remove Q4_1 bit shuffling (ARM NEON + reference)

* ggml : nibbles_from_floats() + bytes_from_nibbles() (ARM NEON)

* ggml : remove Q4_2 bit shuffling (WIP, BROKEN)

* ggml : remove Q5_0 bit shuffling (ARM NEON)

* ggml : 2x faster scalar implementations

* ggml : remove Q5_1 bit shuffling (ARM NEON + scalar)

* ggml : simplify scalar dot

* ggml : remove WASM SIMD bit shuffling + remove vzip for ARM 32-bit

* ggml : fix Q4_1 quantization

* ggml : update cuBLAS + normalize variable names

* ggml : remove Q4_2 mode

* ggml : minor formatting

* ggml : fix Q5_0 quantization

* scripts : add script for measuring the time per token

* AVX implementations (ggml-org#1370)

* ggml : uniform 5th bit extraction

* llama : produce error upon loading old model files

* llama : fix model magic/version write

* ggml : speed-up Q5_0 + Q5_1 at 4 threads

* ggml : preserve old Q4 and Q5 formats

* ggml : simplify Q8_1 - no need for low / high sums anymore

* ggml : fix Q8_0 and Q8_1 rounding

* Revert "AVX implementations (ggml-org#1370)"

This reverts commit 948d124.

* ggml : fix AVX2 implementation

* sha : update hashes for 7B and 13B

* readme : update timings + remove warning banner

* llama : update v2 PR number to 1405

* ggml : fix WASM comments

* ggml : back to original bit order

* readme : add note that Q4 and Q5 have been changed

* llama : fix return for unknown version

---------

Co-authored-by: Stephan Walter <[email protected]>
AlexiAlp pushed a commit to minghaop/llama.cpp that referenced this pull request Jun 2, 2026
@ggerganov @sw
ggml : remove bit shuffling (ggml-org#1405)
a8005d6 
* ggml : remove Q4_0 bit shufling (ARM NEON)

* ggml : remove Q4_1 bit shuffling (ARM NEON + reference)

* ggml : nibbles_from_floats() + bytes_from_nibbles() (ARM NEON)

* ggml : remove Q4_2 bit shuffling (WIP, BROKEN)

* ggml : remove Q5_0 bit shuffling (ARM NEON)

* ggml : 2x faster scalar implementations

* ggml : remove Q5_1 bit shuffling (ARM NEON + scalar)

* ggml : simplify scalar dot

* ggml : remove WASM SIMD bit shuffling + remove vzip for ARM 32-bit

* ggml : fix Q4_1 quantization

* ggml : update cuBLAS + normalize variable names

* ggml : remove Q4_2 mode

* ggml : minor formatting

* ggml : fix Q5_0 quantization

* scripts : add script for measuring the time per token

* AVX implementations (ggml-org#1370)

* ggml : uniform 5th bit extraction

* llama : produce error upon loading old model files

* llama : fix model magic/version write

* ggml : speed-up Q5_0 + Q5_1 at 4 threads

* ggml : preserve old Q4 and Q5 formats

* ggml : simplify Q8_1 - no need for low / high sums anymore

* ggml : fix Q8_0 and Q8_1 rounding

* Revert "AVX implementations (ggml-org#1370)"

This reverts commit 948d124.

* ggml : fix AVX2 implementation

* sha : update hashes for 7B and 13B

* readme : update timings + remove warning banner

* llama : update v2 PR number to 1405

* ggml : fix WASM comments

* ggml : back to original bit order

* readme : add note that Q4 and Q5 have been changed

* llama : fix return for unknown version

---------

Co-authored-by: Stephan Walter <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

3 more reviewers
@sw sw sw left review comments
@Excigma Excigma Excigma left review comments
@BobbyKay1 BobbyKay1 BobbyKay1 approved these changes
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

breaking change Changes that break ABIs, APIs, file formats, or other forms of backwards compatibility. performance Speed related topics

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Use different bit arrangement for quants (nibbles)

Uh oh!

There was an error while loading. Please reload this page.

9 participants

@ggerganov @sw @redthing1 @M00N-MAN @LostRuins @philpax @TheBloke @Excigma @BobbyKay1
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.