GeezerVison is showing github.com — program hanging in pthread_cond_wait() · Issue #386 · cpp-netlib/cpp-netlib
program hanging in pthread_cond_wait() #386DescriptionI have a program that uses cpp-netlib in a very simple way to download some files, and sometimes it hangs indefinitely during a download operation.
I have attached the debugger during the hang, and the stack trace is as follows:
#0 0x00007f68b1ade85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00000000004307bf in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
#2 0x000000000043204e in boost::detail::shared_state_base::wait_internal(boost::unique_lock<boost::mutex>&, bool) ()
#3 0x000000000043210c in boost::detail::shared_state_base::wait(bool) ()
#4 0x00000000004b1010 in boost::detail::shared_state<std::string>::get_sh() ()
#5 0x0000000000495a28 in boost::shared_future<std::string>::get() ()
#6 0x00000000004777cc in boost::network::http::async_message<boost::network::http::tags::http_async_8bit_udp_resolve>::body() const ()
#7 0x000000000045a3b1 in boost::network::http::impl::body_wrapper<boost::network::http::basic_response<boost::network::http::tags::http_async_8bit_udp_resolve> >::operator std::string() const ()
#8 0x000000000043726d in download_file(std::string const&) ()
download_file is a very simple function that looks like this:
std::string download_file(const std::string& url)
{
using namespace boost::network;
using namespace boost::network::http;
return body(client().get(client::request(url)));
}
Any ideas as to what is going wrong, or how I can diagnose the issue further? Activitytheres-waldo commented on Mar 2, 2014 I should mention that I am using version 0.11.0. HighCommander4 commented on May 21, 2014 Does anyone have any suggestion as to how to debug this? This is happening increasingle often, and it's basically making the library unusable. If this is not resolved I'm going to have to look for a different library for networking. deanberris commented on May 21, 2014 This usage pattern is ill-advised because:
- the client object is being destroyed in the call to return
- the body(...) wrapper will block
There will be a race condition and potential for deadlock at the destructor of the client object. The only advice I can give is to not do this.
What should be done instead is to make sure that the client object outlives any request/response objects. The async client object maintains state and shouldn't be treated as an object that can be temporarily created. You're going to have a bad time if you do this.
It's best to re-use a single client object to make multiple HTTP requests. This usage is an antipattern that shouldn't be encouraged. theres-waldo commented on May 24, 2014 I read up a bit of C++'s guarantees on order of evaluation.
- for most expressions, including function call expressions, the order of evaluation of subexpressions in unspecified
- within a full-expression, the order of destruction of temporaries is guaranteed to be the reverse of their order of construction (unless one of them has its lifetime extended by being bound to a reference)
In my case, client().get(client::request(url)) is a function call expression, and client() and client::request(url) are sub-expressions. Therefore, the order of evaluation between client() and client::request(url) is unspecified, and therefore the relative order in which the client and request objects are constructed is unspecified - and thus so is the order in which they are destructed.
To address this, I re-wrote the function as follows:
std::string download_file(const std::string& url)
{
using namespace boost::network;
using namespace boost::network::http;
client c;
client::request r{url};
auto response = c.get(r);
std::string result = body(response);
return result;
}
With this modified function, I experience the hang I described less frequently, but I still experience it.
Do you see anything that's still problematic with the revised function? deanberris commented on May 27, 2014 Does the "hang" not ever end? I suspect this has to do with connections that are kept open by the server, where the TCP FIN message hasn't quite gotten the ACK. We don't have a timeout configured for the implementation in 0.11.0, and it will depend on the platform's TCP implementation timeout to eventually reset the connection. On Linux the default is something like 30 seconds (on some systems its longer). I'm not aware of the default TCP connection timeouts in other environments.
Again I would suggest that you actually re-use the client object, by holding it in a shared pointer, or creating it from main() and passing it down to functions that need to make HTTP calls. There's really no need to be waiting for the client cleanup every time you just want to do an HTTP GET. theres-waldo commented on May 27, 2014
Does the "hang" not ever end?
No. I've had my program hang for days, maybe even weeks, before killing it.
There's really no need to be waiting for the client cleanup every time you just want to do an HTTP GET.
Looking at the stack trace, the hang seems to be happening in 'body_wrapper::operator std::string()', which doesn't look like "client cleanup", but code called from body(). deanberris commented on May 27, 2014 Interesting.
Are you getting data from a specific server/site so that I can reproduce? The best I can think of is the fact that the chunked transfer encoding code may be buggy there, or just that the server on the other end isn't closing the connection. What you can try is adding the "Connection: close" header to the request this way:
client::request r(url);
r << header("Connection", "close");
Before using the request object. This may or may not work depending on which URL you're trying to GET. theres-waldo commented on May 28, 2014 deanberris commented on May 30, 2014 Thanks -- does my suggested work-around work for you? theres-waldo commented on Jun 1, 2014
Thanks -- does my suggested work-around work for you?
Unfortunately not. In fact, with this added header, the download always hangs! theres-waldo commented on Jun 1, 2014
Unfortunately not. In fact, with this added header, the download always hangs!
Actually, it doesn't hang forever with this change. It hangs for about 2 minutes, and then a system_error with the message "Connection reset by peer" is thrown. deanberris commented on Jun 3, 2014 Interesting. I'll need to reproduce this locally before I can even start thinking about a fix. Can you write a self-contained gist that compiles with 0.11.0 and mention it here? Also, on what platform is this with what compiler and what version of Boost? theres-waldo commented on Jun 4, 2014
Interesting. I'll need to reproduce this locally before I can even start thinking about a fix. Can you write a self-contained gist that compiles with 0.11.0 and mention it here? Also, on what platform is this with what compiler and what version of Boost?
https://gist.github.com/theres-waldo/7a3871dc3744c57fbe04
Platform: Ubuntu 10.04
Compiler: GCC 4.8
Boost: 1.55
Note that the hang only happens intermittently, not every time the program is run. A convenient way to reproduce it might be to set up a cron job to run the program every so often, and then after a day or so see if any instances are still running, and if so attach a debugger to one. (Please don't run it too often - say, more often than a few times an hour - lest the maintainers of the target websites get angry :-)). leecoder commented on Jun 25, 2014 I think this issue was also fixed by #405.
Can you retry again after patch? theres-waldo commented on Jul 19, 2014 I tried with the latest code in the 0.11-devel branch, which includes the fix for #405. The hang still occurs, unfortunately. deanberris commented on Jul 21, 2014 @theres-waldo Can you make sure that you're re-using the client object instead? I think this has more to do with the fact that some underlying TCP operations are being missed (like a FIN not getting ACK'ed and therefore a system timeout not kicking in). In your gist the function will not return until the client object is destroyed. There's a potential for a deadlock there but I haven't dug in yet to properly root this out. Whitetigerswt commented on Aug 10, 2014 Hey,
I'm having I think a very similar issue, I'm using the following function:
using namespace boost::network;
http::client c;
std::string CWeb::GetWebsiteText(std::string url)
{
try
{
// Connect to the website
http::client::request req("http://www.boost.org/");
req << header("Connection", "close");
http::client::response res = c.get(req);
// put the result in an std::string.
std::string data = body(res);
return data;
}
catch (std::exception &e)
{
return e.what();
}
return "";
}It's hanging on body(res). (Adding logging indicated that it was body(res).)
I attached a debugger and got a backtrace:
(gdb) backtrace
#0 0xb7727424 in __kernel_vsyscall ()
#1 0xb76f8d4b in pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_wait.S:187
#2 0xb71f713c in boost::detail::shared_state_base::wait(bool) ()
from plugins/samp.so
#3 0xb71e9d68 in boost::detail::shared_state<std::string>::get_sh() ()
from plugins/samp.so
#4 0xb7222809 in CWeb::GetWebsiteText(std::string) ()
from plugins/samp.so
#5 0xb71e8230 in CServerUpdater::CheckForUpdate() () from plugins/samp.so
#6 0xb72143a5 in OnGameModeInit () from plugins/samp.so
#7 0xb72ee0d1 in _OnGameModeInit_handler () from plugins/samp.so
#8 0xb72efda7 in call_public_handler () from plugins/samp.so
#9 0xb72efe5c in sampgdk_callback_invoke () from plugins/samp.so
#10 0xb72f357b in _sampgdk_amxhooks_Exec () from plugins/samp.so
#11 0x080a39df in ?? ()
#12 0x080a9e2d in ?? ()
#13 0x080a86d2 in ?? ()
#14 0xb740aa83 in __libc_start_main (main=0x80a7b30, argc=1, argv=0xbff98654,
init=0x81475e0, fini=0x81475d0, rtld_fini=0xb7737180 <_dl_fini>,
stack_end=0xbff9864c) at libc-start.c:287
#15 0x0804b4d1 in ?? ()
(gdb)
I've tried all suggestions mentioned in this issue.
- not destroying the http::client object (re-using it)
- adding the "req << Header("Connection", "Close");" (the original author mentioned this causes a timeout after a few minutes, it doesn't cause a timeout after a few minutes for me)
- Switching to the 0.11-dlevel branch.
and, googling around, some suggestions to similar issues mentioned:
- Compiling with -std=c++11 cpp-netlib and my program
- Compiling without -std=c++11 cpp-netlib and my program.
Platform: Ubuntu 14.04 LTS
Compiler gcc 4.8
Boost: 1.55
Everything is compiled statically, don't know if that would matter?
The original author mentioned it was only hanging sometimes, for me, it's everytime. Another thing is, this does not happen, with the same code and website, on Windows(VC++ 2013), it only seems to happen on Ubuntu. I'm thinking because it's not occurring on windows, that it's something I'm doing wrong during compile. (I'm not compiling boost/cpp-netlib correctly on linux?) I may try compiling in another distro and getting back to you. Have there been any developments on this issue? Whitetigerswt commented on Aug 11, 2014 I've got kind of a follow up,
I've solved my problem, and I think, like I suspected it was I'm not compiling boost/cpp-netlib correctly. I ended up compiling on Debian 7. I compiled boost, and cpp-netlib libraries and transferred that to Ubuntu and it worked. But I also noticed, when I was re-compiling that my previous version of cpp-netlib was linked statically to boost 1.54, and my main program was linked to boost 1.55. So that may have been the reason of the conflict. However, now that this problem is solved and I literally spent all day (and half of last night) trying to fix it, I'm satisfied and not asking questions anymore. It's working and I'm happy, and I hope I helped someone else solve this problem. deanberris commented on Aug 11, 2014 Awesome, I was just about to give suggestions, but thank you @Whitetigerswt -- I hope it's useful to others too. theres-waldo commented on Jun 9, 2016
The hang most commonly occurs when trying to download one of these two pages:
http://www.apsc.utoronto.ca/timetable/fall.html
http://www.apsc.utoronto.ca/timetable/winter.html
I am beginning to think this is not a problem with cpp-netlib, after all, but with the server that's serving these pages. Even wget intermittently hangs when trying to download them, although wget eventually aborts and tries again (and then it typically succeeds on the second try).
Does cpp-netlib have "retry" functionality similar to wget --tries N, or is that something that needs to be implemented on top of it? deanberris commented on Jun 14, 2016 You can set a timeout value, and if it's timed out then retry the request yourself. The timeout is a configuration option in the constructor for the client class. theres-waldo commented on Jul 31, 2016
I am beginning to think this is not a problem with cpp-netlib, after all, but with the server that's serving these pages. Even wget intermittently hangs when trying to download them
Based on this observation, I believe this issue can be closed. Apologies for the goose chase.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|