GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Introduced new chunked transfer encoding parser to remove chunk markers by umennel · Pull Request #720 · cpp-netlib/cpp-netlib

Conversation

@umennel

@umennel umennel commented Dec 14, 2016

Copy link
Copy Markdown

I introduced a new client option called remove_chunk_markers. With this option selected, a new chunked transfer encoding parser is enabled which is able to parse a stream of small chunks and incrementally remove the chunk markers. The parser requires to keep internal state, so I had to parser function with a function object.
I selected the new option in some of the tests to cover the new code. Proper unit tests are missing yet.
Compiled with gcc 5.4 only.
Some issues might be worth to discuss a bit further:

  • How to handle parse errors (see BOOST_ASSERTs in the parser code)?
  • Performance??

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
umennel added 2 commits December 14, 2016 10:25
@umennel
Introduced new chunked transfer encoding parser to remove chunk marke…
673b853
…rs in streaming clients.
@umennel
Applied new remove chunk markers option to tests.
c4d758a
deanberris
deanberris reviewed Dec 15, 2016
View reviewed changes

@deanberris deanberris left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this @umennel! This has been one of the most-asked-for features to be added to the client and neither @glynos nor I had the time to make it happen.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread boost/network/protocol/http/client/connection/async_normal.hpp Outdated

chunk_encoding_parser() : state(state_t::header), chunk_size(0) {}

enum state_t { header, header_end, data, data_end };

@deanberris deanberris Dec 15, 2016

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can use an enum class here instead.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread boost/network/protocol/http/client/connection/async_normal.hpp Outdated
size_t chunk_size;
std::array<typename char_<Tag>::type, 1024> buffer;

void update_chunk_size(boost::iterator_range<typename std::array<typename char_<Tag>::type, 1024>::const_iterator> const& range) {

@deanberris deanberris Dec 15, 2016

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this is a really long line. We prefer using clang-format to make sure we have consistent formatting in the project, so if you don't mind running this patch through it, that would be really good for readability.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread boost/network/protocol/http/client/connection/async_normal.hpp Outdated
ss << std::hex << range;
size_t size;
ss >> size;
chunk_size = (chunk_size << (range.size()*4)) + size;

@deanberris deanberris Dec 15, 2016

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you help me understand this math?

If range has a size of 1024, this means chunk_size is shifted left 4096 times?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@umennel umennel Dec 18, 2016

Copy link
Copy Markdown
Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but range must correspond to the number of digits and 1024 digits would be an unrealistically large number (it would not throw an error though cause there is no overflow checking). I replaced the '+' with a '|' to make it more clear that this expression is just about appending the new bits to the lower end (and shifting the existing bits by the number of the new bits). Every hex digit corresponds to a quadruple of bits, that's why i multiply the number of digits by four. I didn't want to introduce another char buffer to accumulate the hex digits, so I decided to incrementally compute the chunk size.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread boost/network/protocol/http/client/connection/async_normal.hpp Outdated
// }

// return body;
// }

@deanberris deanberris Dec 15, 2016

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider just removing these lines?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread libs/network/test/http/client_get_different_port_test.cpp
typename client::options options;
options.remove_chunk_markers(true);
client client_;
typename TypeParam::request request("http://www.boost.org:80/");

@deanberris deanberris Dec 15, 2016

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this might be out-of-scope, but could you make this point to cpp-netlib.org instead? That would be great, thanks. :)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@umennel umennel Dec 16, 2016

Copy link
Copy Markdown
Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately cpp-netlib.org does not yield chunk encoded content. It does not matter much for this particular test, but for the streaming test we'd have all scenarios covered with www.boost.org: it yields chunk encoded content for http/1.1 clients and non encoded content for http/1.0 clients.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@deanberris deanberris Dec 16, 2016

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good point, thanks.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
umennel added 5 commits December 15, 2016 23:43
@umennel
Nicer formatting
e9bf5f7
@umennel
Use strongly typed enum.
9586a81
@umennel
Clarified and commented expression.
c70f774
@umennel
Removed commented lines.
ff5c7c8
@umennel
Nicer formatting and clean ups
192d520
@umennel

umennel commented Mar 30, 2017

Copy link
Copy Markdown
Author

Is there any chance to merge this PR in the near future? Is there still something missing?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@deanberris

deanberris commented Mar 31, 2017

Copy link
Copy Markdown
Member

Hi @umennel -- I'm so sorry about the delay, this just slipped through the cracks. :(

Let me have a quick look and merge.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@deanberris

deanberris commented Mar 31, 2017

Copy link
Copy Markdown
Member

LGTM

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@deanberris
deanberris merged commit 03f6d3b into cpp-netlib:master Mar 31, 2017
@umennel umennel mentioned this pull request Mar 5, 2018
Streaming API does not allow access to the HTTP status or the headers from the callback #827
Open
igorpeshansky
igorpeshansky reviewed Mar 7, 2018
View reviewed changes
Comment thread boost/network/protocol/http/client/connection/async_normal.hpp
body_string.append(this->part.begin(), this->part.begin() + bytes_transferred);
if (this->is_chunk_encoding) {
this->body_promise.set_value(parse_chunk_encoding(body_string));
if (this->is_chunk_encoding && remove_chunk_markers_) {

@igorpeshansky igorpeshansky Mar 7, 2018 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@umennel Careful -- this actually breaks previous behavior. Now, unless you set remove_chunk_markers to true, the following code will produce a chunked-encoded string:

http::client client;
http::client::request request("http://www.boost.org:80/");
http::client::response response = client.get(request);
std::cerr << body(response) << std::endl;

@deanberris Is this what we want?

To put it another way, can you think of a case where we wouldn't want remove_chunk_markers to default to true?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@umennel umennel Mar 15, 2018

Copy link
Copy Markdown
Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@igorpeshansky, indeed, the default should be true. Otherwise it is hard to keep the behavior consistent. We should decide on this before merging into 0.13.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@igorpeshansky igorpeshansky Mar 15, 2018

Copy link
Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, I'll send a PR shortly.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@igorpeshansky igorpeshansky Mar 15, 2018

Copy link
Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sent #830.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@igorpeshansky igorpeshansky mentioned this pull request Mar 15, 2018
Enable remove_chunk_markers by default. Fix a bug and add a comment. #830
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@deanberris deanberris deanberris left review comments
+1 more reviewer
@igorpeshansky igorpeshansky igorpeshansky left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@umennel @deanberris @igorpeshansky
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.