GeezerVison

I noticed this since the test case with https://aip.scitation.org/doi/10.1063/1.5081715 started failing

Short-circuited the normalized equality so as to cut down on the number of attempts at resolving redirects -- otherwise, have been running into 403 errors that seem to arise from rate-limiting (since they go away with more attempts)

OK, so I've addressed most of my concerns -- I've documented the problem with unconditionally upgrading to https, have made the test case warn when it needs to fall back to resolving redirects, and removed the HTTPException (further testing revealed that the information it provided is accessible by invoking pytest with --showlocals).
However, further testing does indeed show that the test remains flaky, with scitation abusing HTTP 403 when they want to respond with 429. I'd recommend adding pytest-rerunfailures or similar to the dev requirements and marking this particular test as flaky and needing to be rerun on HTTPException.

Added pytest-rerunfailures to address the flakiness, I have no more notes.

No good, the test still fails with 403. Can't think of what else they're probing that we need to send -- curl -A Mozilla/5.0 works fine at the https endpoint.

OK, I've rebased this on top of #2 (including alexfikl#1 on the assumption @alexfikl will merge it), so now both can be merged with no issue.
I've dropped the pytest-rerunfailures commit, since ultimately what seems to be the correct advice is to just update the URLs when they change, as done in #2.

Hm. I'm not fully satisfied by my implementation -- I would like to figure out how come I'm managing to get the page to load in the browser but urllib returns 403. I have discovered that urllib is properly redirecting the URL, it's just that the final URL returns 403 (presumably because Cloudflare is detecting the scraper).

I have discovered that urllib is properly redirecting the URL, it's just that the final URL returns 403 (presumably because Cloudflare is detecting the scraper).

Yeah, Cloudflare anti-scraper stuff is making a lot of the downloaders in Papis also fail (even though they would work properly if fed the actual webpage). I'm not sure there's much we can do there.. from what I recall the "fix" for that is to use something like playwright that can fake being a known browser.

OK, cloudscraper indeed seems to work:

import cloudscraper
scraper = cloudscraper.create_scraper()
print(scraper.get('http://aip.scitation.org/doi/10.1063/1.5081715').url)
# https://pubs.aip.org/aip/jcp/article-abstract/150/7/074102/197572/Exact-two-component-equation-of-motion-coupled?redirectedFrom=fulltext

OK, added a [challenges] extra to bypass websites using CloudFlare (and friends, in the future) to protect against DDoS. Also added a test to check my redirect resolution code works, as @alexfikl asked for in #2 (comment), though ATM it only has testcases for cloudscraper-dependent sites. Still, I'm happy with the code as it stands now, and would like to have it merged.
@alejandrogallo, can we get this merged?

Note this still isn't perfect -- my testing must've gotten my on scitation's warnlist, because I'm starting to get redirected to their captcha page.

So I tried to investigate this further today, see if I could make the test more robust.
First problem is that scitation is very aggressive with their anti-scraping, so that testcase works at best 3-4 times before their countermeasures kick in.
In trying to investigate this, I tried using different test cases to replace it.
Found some irreproducible flakiness with this article. But even worse, trying to bulk out the testcase collection led me to discover multiple resolution, and the papers cited there all break our current system, to a point I'm no longer sure what the correct semantics are here. Does "a canonical URL" even exist? If not, is this PR useless?

EDIT: Reading further on that page, it seems that among the multiple valid URLs for the DOI, there will always be a "primary" URL, which can obtained by adding ?locatt=mode:legacy to the doi.org query string. But that means our equality needs to be at the level of DOIs, and that naively following redirects is insufficient.

From what I've seen this seems fine, thank you for your work, what do you think @alexfikl ?

@alejandrogallo Yeah, this looks good to me too! It doesn't seem like we can do much better these days + scraping seems to be disallowed more and more with AI wreaking havoc all over the place :(

It also includes #2, so I'll go ahead and close that.

First problem is that scitation is very aggressive with their anti-scraping, so that testcase works at best 3-4 times before their countermeasures kick in.

Might be worth looking for some other website then? The main point is to test that the DOI detection in the library works.. not necessarily that it works with scitation 😁