Description

benoit74

opened on Nov 7, 2026

Issue body actions

See #338 (comment)

We've introduced recent changes in rewriting logic to store original URLs everywhere in the DOM, and we have multiple PRs in wombat to be able to use these attributes during replay to "fake" JS scripts, so that they can interpret everything they want based on the URL.

Among these is the need to know if a URL has already been rewritten to not rewrite it twice.

The remark from @ikreymer is that we could theoretically not need to do that if we have a known prefix in every URLs (e.g. zimreplay/...).

This means that all URLs in HTML / DOM / JS must go up to that prefix so that we know it is rewritten (instead of just going "as-up-as-needed" like we do currently, e.g. for two HTML documents page1.html and page2.html in the same path, a link to page2 from page1 is just ./page2.html. Here we will need to go up to the prefix with as many ../ as needed.

While I agree this is definitely sufficient to solve the problem of URLs being rewritten twice, this is probably not sufficient to solve the issue of knowing what the original URL was.

We do have JS scripts which interpret various URLs (the document one, but also sometimes images src or anchor href) to do "something". For that we need to expose the true original value, not the one which allows to access the item inside the ZIM.

One example is that query parameters are escaped once rewritten, to be able to access the matching ZIM path.

This is however definitely a reversible operation (we know that query parameter are escaped, so we can just un-escape them).

However, as far as I remember we've already realized that the operation HTTP URL => ZIM path is not a perfectly reversible operation, i.e. once you have the ZIM path you know which URL you have to use to access it but you can't be certain about what the original URL was. I don't have exact example at hand, will look into it.

Activity

1. benoit74 commented on Nov 7, 2026

   

   benoit74

   on Nov 7, 2026

   CollaboratorAuthor

   More actions

   Found some concrete examples where the HTTP URL → ZIM path transformation in ArticleUrlRewriter.normalize is not reversible: two genuinely different original URLs produce the exact same ZIM path, so there's no way to know which one it was.

   (Scheme/port being dropped, and fuzzy-rule matching dropping query params, are also non-reversible, but those are obvious edge cases and not the interesting part here.)

   1. Percent-encoded / vs. literal /

   normalize() calls unquote(path), which can't distinguish an escaped slash from a real path separator:

   Original URL	Normalized ZIM path
   https://example.com/foo%2Fbar (one segment, containing a literal /)	example.com/foo/bar
   https://example.com/foo/bar (two segments)	example.com/foo/bar

   These can legitimately be two different resources server-side (many servers treat an encoded %2F differently from a raw / in the path), but we only ever get one ZIM path out of them.

   2. Dot-segments vs. literal _./_.. segments

   escape_dot_segments() rewrites a . or .. path segment to _./_.. so it isn't resolved away as navigation. But dot-segment resolution only happens in _urljoin when merging a relative reference onto the base URL — an already-absolute link (own scheme + host) skips it entirely. So a dot-segment that survives to normalize() and a genuinely literal _./_.. segment land on the same path:

   Original URL	Normalized ZIM path
   https://example.com/foo/./bar (absolute link, dot-segment never resolved)	example.com/foo/_./bar
   https://example.com/foo/_./bar (resource whose segment is literally _.)	example.com/foo/_./bar

   Same pattern applies to .. vs. _...

   3. +, %20, and a raw space in the query string

   These lines replace + with a space before unquoting, so all three of these collapse to the same stored character:

   Original query string	Normalized query
   ?a=1+2	?a=1 2
   ?a=1%202	?a=1 2
   ?a=1 2 (literal, unencoded space)	?a=1 2

   Once stored, there's no way to tell whether the original URL used +, %20, or a raw space — which matters for JS that re-reads location.search and expects one specific encoding.

   All three were verified against the current code (not just inferred).
2. pawank925 commented on Nov 7, 2026

   

   pawank925

   on Nov 7, 2026

   Contributor

   More actions

   @benoit74 will i take this ?
3. benoit74 commented on Nov 8, 2026

   

   benoit74

   on Nov 8, 2026

   CollaboratorAuthor

   More actions

   @pawank925 I'm really not sure we want to do anything, this is only a discussion at this stage

   Reacted by pawank925
4. benoit74

   added

   needs discussionA decision is needed before working on thisA decision is needed before working on this

   on Nov 8, 2026
5. benoit74

   added the

   Question

   issue type on Nov 8, 2026
6. ikreymer commented on Nov 10, 2026

   

   ikreymer

   on Nov 10, 2026

   More actions

   @benoit74 thanks for starting this and documenting the edge cases!

   However, as far as I remember we've already realized that the operation HTTP URL => ZIM path is not a perfectly reversible operation, i.e. once you have the ZIM path you know which URL you have to use to access it but you can't be certain about what the original URL was.

   I think it would be good to fix this, if possible.

   Since wombat generally assumes that the rewriting is reversible, the extractOriginalURL() can find the original from the rewritten.
   This is called in a few different places, and while the previous fixes address some of them, it is not all of them, and I'm concerned about having these one-off fixes that won't still fully address everything. I think an ideal solution would be to make the ZIM rewriting also reversible, and then just have a custom extractOriginalURL() that cleanly converts back..

   Perhaps this is doable with the prefix + some more fixes to the escaping?

   1. Percent-encoded / vs. literal /

   Perhaps this can be adjusted to just decode the query string, leaving path as is?

   2. Dot-segments vs. literal ./.. segments

   That seems like a good way to handle ., so just need to be able to rewrite _. to __. consistently?

   3. +, %20, and a raw space in the query string

   This might actually be ok. I think %20 is preferred, eg. setting history.replaceState("", "", "/foo?a b") results in a%20b.
   I believe in an HTTP request, the %20 and + should be treated the same in the query. Hopefully not likely that a site mixes both?
   I think rewriting to %20 should probably be safe.

   This is probably more work here, but I think it is a better approach for long term compatibility with wombat.

   Doing a quick search, extractOriginalURL( is called about 25 times in different places, and while the PRs you have fix some of them,
   I would be concerned about subtle issues that arise from all the different code paths.

   Reacted by benoit74
7. ikreymer commented on Nov 10, 2026

   

   ikreymer

   on Nov 10, 2026

   More actions

   Another edge which I forgot about but was detected in the issue above, is the unrewriting of dynamically generated HTML, which is handled to avoid duplicate rewriting in wombat via the wb_unrewrite_rx. It's not used often, I believe its needed for pages that change the DOM dynamically and checks if URL starts with the designated prefix.
8. benoit74

   self-assigned this

   on Nov 12, 2026

Issue actions