GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — msgpack-value v07 API by xerial · Pull Request #109 · msgpack/msgpack-java

Conversation

@xerial

@xerial xerial commented Jun 16, 2014

Copy link
Copy Markdown
Member

I implemented msgpack-value API for v07. Here is an overview of the API:
powerpoint___-_2014-06-17-msgpack-value_pptx

msgpack-value has the following components:

  • Cursor: traverses message-packed values. The user calls hasNext(), then next() : Value or nextRef() : ValueRef to retrieve values.
  • Value: Immutable representation of a message-packed value.
    • NumberValue : An interface for Integer and Float values. It provides number conversion methods (w/ or w/o truncation/rounding) and value range checkers.
  • ValueRef: A reference to a message-packed value. This reference is valid until cursor.next()/nextRef() is called. To make it immutable, call ValueRef.toValue() : Value, which creates a copy of the referenced value.
  • ValueHolder: An implementation of ValueRef, a sort of union (in C language) that can hold any type of a value. This is useful to reduce the cost of Value instance creation. MessageUnpacker directly feeds a value into the holder via unpackValue(ValueHolder) method.

Miscellaneous notes:

  • org.msgpack.value is now within the msgpack-core project because Cursor and ValueHolder need to interact with MessageUnpacker, and it makes difficult to separate msgpack-core and msgpack-value projects. For example, to switch the holder to use (e.g., long? or BigInteger?), adding unpackValue(ValueHolder) to the unpacker, then let it choose the target holder is the most straightforward way. And also ValueHolder can be used as an implementation of ValueRef. Moving org.msgpack.value package within msgpack-core project makes simple these interactions.
  • Traversing 10,000 binary values using Cursor.nextRef is about 2.7 times faster than using Cursor.next (returns immutable values):
    1__leo_weaver_local___work_git_msgpack-java__java_
  • MessageUnpacker:
    • Now supports counting total read bytes.
    • Added unpackValue(ValueHolder), unpackInteger(IntegerHolder), unpackFloat(FloatHolder)
  • Array/MapCursor
    • Materialising large array/map values requires huge memory. Array/MapCursor interfaces that can be obtained from Cursor provide the methods for traversing arrays and maps without constructing the whole array/map values.
    • In the draft code, these cursors extended List<Value> and Map<Value, Value> interfaces, but this design makes complicate the implementation since Java does not allow mixin of the List<Value> and Value API implementations (e.g., AbstractValue). Because of this, we needed to copy many codes to implement Array/MapCursor. Instead, I chose to provide only hasNext() and next() methods to traverse map/array values.
  • Upgraded to Scala 2.11 (Scala is used only for building and testing). TravisCI now runs sbt test
  • Added settings to deploy and release to sonatype: To sync with Maven Central, do sbt publishSigned, then sbt sonatypeRelease.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@xerial xerial modified the milestone: v07 Jun 16, 2014
komamitsu
komamitsu reviewed Jun 22, 2014
View reviewed changes
Comment thread msgpack-core/src/main/java/org/msgpack/value/holder/IntegerHolder.java Outdated

@komamitsu komamitsu Jun 22, 2014

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

type is better?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@xerial xerial Jun 22, 2014

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a private field, which is not visible to the user, but using type is OK.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@xerial
#109: Applied fixes suggested by @komamitsu
9dae25e
oza
oza reviewed Jun 29, 2014
View reviewed changes
Comment thread msgpack-core/src/main/java/org/msgpack/core/MessageOverflowException.java Outdated

@oza oza Jun 29, 2014

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we fix the comment to describe what this exception expresses?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@xerial xerial Jun 30, 2014

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. I will add a description.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
xerial added 3 commits June 30, 2014 09:37
@xerial
#109: Add a case for BINARY type in toByteBuffer
a6619fb
@xerial
#109: Add comment to MessageOverflowException
698a92c
@xerial
#109: Simply convert to uint32 (long) value using a bit mask
64d397c
This was referenced Nov 9, 2014
Releasing msgpack v0.7 #147
Closed
Updating getting started and tutorial page after releasing v0.7 #148
Open
xerial added 6 commits November 28, 2014 15:48
@xerial
Merge branch 'v07-develop' into v07-value
a2881a5
@xerial
0.7.0-M6-SNAPSHOT vesrion
d655c03
@xerial
Remove ValueRef interface
7a94130
@xerial
isImmutable is false for Array/MapCursor
84ac459
@xerial
Make Map/ArrayCursorImpl implement Value interface
52342fe
@xerial
Adjust benchmark
ad0bbbe
@xerial

xerial commented Nov 29, 2014

Copy link
Copy Markdown
Member Author

�Simplified the interface by removing Cursor and ValueRef. Instead the Value interface now has toImmutable and isImmutable method to create immutable representations.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@frsyuki

frsyuki commented Dec 1, 2014

Copy link
Copy Markdown
Member

why not adding ImmutableXxxValue interfaces?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@oza

oza commented Dec 6, 2014

Copy link
Copy Markdown
Member

@frsyuki Do you mean toImmutable methods return ImmutableXxxValue instead of XxxValue?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@frsyuki

frsyuki commented Dec 6, 2014

Copy link
Copy Markdown
Member

@oza that's right. ImmutableXxxValue are an interfaces extending XxxValue.

As an user, I want to know whether "this value" could be changed by other modules. With ImmutableValue and ImmutableXxxValue, I can use compiler's help to make sure this value is guaranteed to be immutable without writing unlimited amount of tests.
From this point of view, I think it's good design decision where MapValue.toMap returns non-changable map.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@frsyuki

frsyuki commented Dec 6, 2014

Copy link
Copy Markdown
Member

I'm assuming Cursor, MapCursor and ArrayCursor will be removed. Am I correct?
I can't find implementation of mutable values.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@oza

oza commented Dec 6, 2014

Copy link
Copy Markdown
Member

Hmm, latest patch still has Cursor, MapCursor and ArrayCursor.

@xerial could you clarify this point?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@xerial

xerial commented Dec 9, 2014

Copy link
Copy Markdown
Member Author

@frsyuki OK. I'm now working on adding ImmutableValue interfaces.
And the cursor interface will be removed, but the code of ArrayCursor and MapCusor will be used as an efficient implementation of (mutable) Values. I'm also thinking about implementing ValueHolder as a value, since writing ValueHolder.get().asIntegerValue() is a little bit cumbersome. Allowing to call ValueHolder.asIntegerValue() is much simpler.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@frsyuki

frsyuki commented Dec 9, 2014

Copy link
Copy Markdown
Member

@xerial Thank you for the explication. Overall, I agree with you.
I think we need only one mutable Value implementation called ValueBuffer because:

  • A purpose to have mutable value is removing overhead to create a new Value instance when we unpack a value.
  • Value should be able to be any types (including array and map) because Unpacker can unpack any types.
  • Current holder/cursor implementations (FloatHolder, RawHolder, ArrayCursor, etc.) can't be different types. We need to create a new value if unpacked type different from an expected type.

My draft code implemented this idea: https://github.com/msgpack/msgpack-java/blob/v07-value-sf/msgpack-value/src/main/java/org/msgpack/value/ValueBuffer.java

A difficulty is reusing nested values to store nested elements of a array and map Value.
For example, in my draft code above, ValueBuffer needs only one ValueUnion when it stores an integer value. However, it needs multiple ValueUnion instances for each element when it stores an array value (because valueBuffer.asArrayValue().get(i) returns Value). This is an essential problem because the same problem anyways happens to deserialize nested values

A known solution is to use an object pool (ValueBufferPool). Another difficulty here is the timing to reuse nested elements. A solution is to add ValueBuffer.release(). These ideas need following complex interfaces:

  • interface ValueBufferAllocator
    • ValueBuffer allocate()
  • interface ValueBufferDeallocator
    • void deallocate(ValueBuffer unused)
  • class ValueBufferPool implements ValueBufferAllocator, ValueBufferDeallocator
  • ValueBuffer.(ValueBufferAllocator allocateFrom, ValueBufferDeallocator releaseTo)
    • ValueBuffer.() will creates a new ValueBufferPool
    • void release() { valueBufferDeallocator.deallocate(this); }
      • this could be reference-counted like Netty's ByteBuf. But could be non-reference-counted like Jetty's ByteBufferPool. I think reference-counting causes too much overhead when the value is a deeply nested map/array.

You may have another idea.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@RoSkin

RoSkin commented Apr 17, 2015

Copy link
Copy Markdown

How can I pack a JavaBean?I found there is no method to pack JAVABean

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@xerial

xerial commented Jun 19, 2015

Copy link
Copy Markdown
Member Author

This work in progress at #246

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@xerial xerial closed this Jun 19, 2015
@xerial
xerial deleted the v07-value branch July 19, 2025 16:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

v07

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@xerial @frsyuki @oza @RoSkin @komamitsu
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.