GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Edit Task
  • Edit Related Tasks...
  • Create Subtask
  • Edit Parent Tasks
  • Edit Subtasks
  • Merge Duplicates In
  • Close As Duplicate
  • Edit Commits
  • Edit Mocks
  • Mute Notifications
  • Protect as security issue
  • MediaWiki-Parser
  • (Backlog)
  • MediaWiki-Language-converter

This view

  • Original page
  • New address

GeezerVison is showing phabricator.wikimedia.org — T54661 Preprocessor/Parser irregularities with -{...}- variant constructs.

Page MenuHomePhabricator
Log In
Create Task
Maniphest T54661

Preprocessor/Parser irregularities with -{...}- variant constructs.
Open, MediumPublic
Actions

  • Edit Task
  • Edit Related Tasks...
  • Create Subtask
  • Edit Parent Tasks
  • Edit Subtasks
  • Merge Duplicates In
  • Close As Duplicate
  • Edit Related Objects...
  • Edit Commits
  • Edit Mocks
  • Mute Notifications
  • Protect as security issue
Assigned To
None
Authored By
cscott
Aug 9 2013, 2:38 AM2013-08-09 02:38:00 (UTC+0)
Project Tags
  • MediaWiki-Parser (Backlog)
  • MediaWiki-Language-converter (Core - Parser/Language converter)
  • Parsoid-Read-Views (Language Converter Support) (Backlog)
Referenced Files
None
Subscribers
Aklapper
Bawolff
cscott
Dalba
• Elitre
gerritbot
• GWicke
View All 15 Subscribers

Description

This is a parent task for various subtasks having to do with irregularities parsing -{...}- constructs, especially if they contain embedded vertical bar characters. See the subtasks for specific bugs having to do with different places in the parser and preprocessor where irregularities have been found.

Details

Reference
bz52661
Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Other language converter bugs (test case tweaks)C. Scott Ananianmediawiki/coremaster+1 -2
Customize query in gerrit

Related Objects
Search...

  • Task Graph
  • Mentions
  • Duplicates
View Standalone Graph
This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.
StatusSubtypeAssignedTask
· · ·
In ProgressNoneT43716 [EPIC] Support language variant conversion in Parsoid
OpenNoneT54661 Preprocessor/Parser irregularities with -{...}- variant constructs.
ResolvedcscottT146304 Preprocessor should handle -{...}- variant constructs in template arguments
ResolvedcscottT146305 Parser should protect -{...}- variant constructs in links
ResolvedcscottT54192 Markups in alt param of <gallery> are "eaten" during parsing
ResolvedcscottT54190 <gallery> with |link=<external link> doesn't work on wikis with LanguageConverter
ResolvedcscottT153135 doBlockLevels breaks with embedded language converter markup
ResolvedcscottT153140 -{ ... }- markup breaks tables
OpenNoneT153265 Language converter source text and language names cannot use <nowiki> escaping.
DuplicateBUG REPORTNoneT353501 new Parsoid cannot parse the converter wikitext syntax
· · ·
Mentioned In
T242466: [Research spike] Why does posting a new comment delete `</span>` from a previous one?
T189095: Template inclusion with -{ (minus followed by an opening curly brace) stops parser and prevents inclusion
T172306: Broken wikilinks can be parsed as wikilinks after preprocessing
T142803: [RFC] Spec for future wiki content processing model ("wikitext 2.0")
T119158: Language converter: unsafe attribute injection via glossary rules (CVE-2017-8815)
Mentioned Here
T166429: Getting a unclean output with {{#property:P856}} on site which enables Language Converter
T146305: Parser should protect -{...}- variant constructs in links
T153761: Incorrect parser output if -{{ appears in wikitext
Duplicates Merged Here
T74010: [[zh:MediaWiki:Wikimedia-copyright/zh]] shows unexpected -{R| } prefixes for URLs

Event Timeline

• bzimport raised the priority of this task from to Medium.Nov 22 2014, 2:01 AM2014-11-22 02:01:16 (UTC+0)
• bzimport added a project: MediaWiki-Parser.
• bzimport set Reference to bz52661.
• bzimport added a subscriber: Unknown Object (MLST).
cscott created this task.Aug 9 2013, 2:38 AM2013-08-09 02:38:00 (UTC+0)
cscott added a comment.Aug 9 2013, 2:39 AM2013-08-09 02:39:40 (UTC+0)
Comment Actions

From IRC:

07:36:15 PM) TimStarling: but it wouldn't be too hard to make the preprocessor annotate it, the same way it does with links
(07:36:33 PM) TimStarling: you know the preprocessor is responsible for expanding templates
(07:36:55 PM) TimStarling: but it marks up links for the sole purpose of getting correct template DOM
(07:37:28 PM) TimStarling: e.g. for parameter splitting in {{ a [[b|c]] }}
(07:37:45 PM) TimStarling: it would probably be beneficial for -{}- to be handled in the same way
(07:38:15 PM) TimStarling: then {{ a -{b|c}- }} would work in the intuitive way
(07:38:10 PM) cscott-free: yes. i think i'm going to add [[File:foobar.jpg|-{R|rawcaption}-]] as a parser test and open a bugzilla for that. for the future.

gerritbot added a comment.Aug 9 2013, 2:58 AM2013-08-09 02:58:30 (UTC+0)
Comment Actions

Change 78330 had a related patch set uploaded by Cscott:
Add parserTests for language converter markup.

https://gerrit.wikimedia.org/r/78330

liangent added a comment.Aug 9 2013, 3:02 AM2013-08-09 03:02:24 (UTC+0)
Comment Actions

btw There's another wikitext snippet that isn't handled well currently:

;-{zh-cn:AAA;zh-tw:BBB}-

Is this resolvable with the preprocessor change?

cscott added a comment.Aug 12 2013, 7:09 PM2013-08-12 19:09:32 (UTC+0)
Comment Actions

(In reply to comment #3)

;-{zh-cn:AAA;zh-tw:BBB}-
Is this resolvable with the preprocessor change?

Yes, I believe this has the same root cause.

• GWicke added a comment.Aug 14 2013, 12:39 AM2013-08-14 00:39:06 (UTC+0)
Comment Actions

(In reply to comment #4)

(In reply to comment #3)

;-{zh-cn:AAA;zh-tw:BBB}-
Is this resolvable with the preprocessor change?

Yes, I believe this has the same root cause.

Lists are not handled by the preprocessor. The issue here is that the list handler (doBlockLevels) is not aware of -{ }- either and (wrongly) recognizes the embedded colon as a single-line dt/dd pair.

cscott added a comment.Aug 14 2013, 8:51 PM2013-08-14 20:51:18 (UTC+0)
Comment Actions

Right. But if the preprocessor lifts out the -{...}- constructs, then doBlockLevels won't get confused. So yes, same root cause.

• GWicke added a comment.Aug 14 2013, 9:27 PM2013-08-14 21:27:11 (UTC+0)
Comment Actions

If you reintroduce language conversion blocks only after doBlockLevels is done, then you'll need to find a different way to parse the contents of those blocks independently of the main content.

liangent added a comment.Aug 16 2013, 7:05 AM2013-08-16 07:05:51 (UTC+0)
Comment Actions

One more thing being broken:

{|

-
-{RB}-
}
cscott added a comment.Aug 26 2013, 6:53 PM2013-08-26 18:53:12 (UTC+0)
Comment Actions

Also:
-{zh-cn:[[Category:A]];zh-tw:[[Category:B]];}-

This shouldn't be in both A and B (should it?). We don't want the category to depend on the variant. So maybe it *should* be in both?

cscott added a comment.Aug 26 2013, 6:56 PM2013-08-26 18:56:44 (UTC+0)
Comment Actions

I think it should be in neither. (gwicke agrees.)

[[Category:foo]] would add it to the 'foo' category. in a variant where foo=>bar, it might appear like [[Category:foo|bar]], and be edited that way by VE, but that wouldn't change the category of the page. Category links inside -{...}- would be forbidden (that is, parsed as plain text).

gerritbot added a comment.Nov 7 2013, 4:39 PM2013-11-07 16:39:59 (UTC+0)
Comment Actions

Change 78330 merged by jenkins-bot:
Add parserTests for language converter markup.

https://gerrit.wikimedia.org/r/78330

Aklapper added a comment.Nov 2 2014, 12:46 PM2014-11-02 12:46:43 (UTC+0)
Comment Actions
  • Bug 72875 has been marked as a duplicate of this bug. ***
Aklapper added a comment.Nov 2 2014, 12:46 PM2014-11-02 12:46:53 (UTC+0)
Comment Actions
  • Bug 72010 has been marked as a duplicate of this bug. ***
Jackmcbarn subscribed.Dec 1 2014, 3:21 AM2014-12-01 03:21:51 (UTC+0)
Liuxinyu970226 subscribed.Jun 20 2015, 8:05 AM2015-06-20 08:05:37 (UTC+0)
cscott mentioned this in T119158: Language converter: unsafe attribute injection via glossary rules (CVE-2017-8815).Dec 1 2015, 11:55 PM2015-12-01 23:55:42 (UTC+0)
gerritbot subscribed.Sep 20 2016, 10:29 PM2016-09-20 22:29:26 (UTC+0)
Comment Actions

Change 311849 had a related patch set uploaded (by C. Scott Ananian):
WIP: protect language converter markup in the preprocessor.

https://gerrit.wikimedia.org/r/311849

gerritbot added a project: Patch-For-Review.Sep 20 2016, 10:29 PM2016-09-20 22:29:27 (UTC+0)
cscott created subtask T146304: Preprocessor should handle -{...}- variant constructs in template arguments.Sep 21 2016, 6:08 PM2016-09-21 18:08:54 (UTC+0)
cscott created subtask T146305: Parser should protect -{...}- variant constructs in links.Sep 21 2016, 6:15 PM2016-09-21 18:15:37 (UTC+0)
cscott renamed this task from Preprocessor should handle -{...}- variant constructs. to Preprocessor/Parser irregularities with -{...}- variant constructs..Sep 21 2016, 6:17 PM2016-09-21 18:17:46 (UTC+0)
cscott updated the task description. (Show Details)
cscott added subtasks: T54192: Markups in alt param of <gallery> are "eaten" during parsing, T54190: <gallery> with |link=<external link> doesn't work on wikis with LanguageConverter.Sep 21 2016, 6:19 PM2016-09-21 18:19:56 (UTC+0)
cscott mentioned this in T142803: [RFC] Spec for future wiki content processing model ("wikitext 2.0").Sep 21 2016, 7:51 PM2016-09-21 19:51:15 (UTC+0)
gerritbot added a comment.Dec 13 2016, 7:22 PM2016-12-13 19:22:30 (UTC+0)
Comment Actions

Change 312066 had a related patch set uploaded (by C. Scott Ananian):
Other language converter bugs (test case tweaks)

https://gerrit.wikimedia.org/r/312066

cscott added a comment.Dec 13 2016, 8:48 PM2016-12-13 20:48:50 (UTC+0)
Comment Actions

There are also irregularities in how lists and tables with language converter markup are handled; see https://gerrit.wikimedia.org/r/312066

cscott added a subtask: T153135: doBlockLevels breaks with embedded language converter markup.Dec 13 2016, 10:09 PM2016-12-13 22:09:21 (UTC+0)
cscott added a subtask: T153140: -{ ... }- markup breaks tables.Dec 13 2016, 10:56 PM2016-12-13 22:56:33 (UTC+0)
gerritbot added a comment.Dec 14 2016, 10:38 PM2016-12-14 22:38:15 (UTC+0)
Comment Actions

Change 312066 abandoned by C. Scott Ananian:
Other language converter bugs (test case tweaks)

Reason:
Squashed into https://gerrit.wikimedia.org/r/327127

https://gerrit.wikimedia.org/r/312066

Nemo_bis added a subtask: T153761: Incorrect parser output if -{{ appears in wikitext.Dec 21 2016, 8:43 AM2016-12-21 08:43:37 (UTC+0)
Comment Actions

I added T153761 as blocker since it would be nice to have a test for that case (I see things are moving: T146305#2891350).

cscott closed subtask T146305: Parser should protect -{...}- variant constructs in links as Resolved.Mar 30 2017, 8:25 PM2017-03-30 20:25:53 (UTC+0)
cscott closed subtask T153140: -{ ... }- markup breaks tables as Resolved.
cscott removed a subtask: T153761: Incorrect parser output if -{{ appears in wikitext.
cscott closed subtask T54192: Markups in alt param of <gallery> are "eaten" during parsing as Resolved.Mar 30 2017, 8:30 PM2017-03-30 20:30:34 (UTC+0)
cscott closed subtask T54190: <gallery> with |link=<external link> doesn't work on wikis with LanguageConverter as Resolved.Mar 30 2017, 8:34 PM2017-03-30 20:34:23 (UTC+0)
cscott closed subtask T146304: Preprocessor should handle -{...}- variant constructs in template arguments as Resolved.Jun 5 2017, 2:10 AM2017-06-05 02:10:47 (UTC+0)
cscott closed subtask T153135: doBlockLevels breaks with embedded language converter markup as Resolved.Jun 5 2017, 2:13 AM2017-06-05 02:13:01 (UTC+0)
cscott added a comment.Jun 5 2017, 2:16 AM2017-06-05 02:16:34 (UTC+0)
Comment Actions

There's an issue with autolink URLs, like:

-{en-us:http://elevator.com;en-gb:http://lift.net}-

because the autolink regexp won't stop at the semicolon and thus will grab the en-gb and break the language converter nesting. See T166429: Getting a unclean output with {{#property:P856}} on site which enables Language Converter.

Not entirely sure this is fixable, it seems to be a genuine priority mismatch between autolink and language converter constructs.

cscott added a subtask: T153265: Language converter source text and language names cannot use <nowiki> escaping..Aug 2 2017, 5:45 PM2017-08-02 17:45:15 (UTC+0)
cscott mentioned this in T172306: Broken wikilinks can be parsed as wikilinks after preprocessing.Aug 2 2017, 6:03 PM2017-08-02 18:03:14 (UTC+0)
Dalba subscribed.Feb 11 2018, 2:25 AM2018-02-11 02:25:39 (UTC+0)
cscott mentioned this in T189095: Template inclusion with -{ (minus followed by an opening curly brace) stops parser and prevents inclusion.Aug 12 2019, 5:55 PM2019-08-12 17:55:16 (UTC+0)
cscott mentioned this in T242466: [Research spike] Why does posting a new comment delete `</span>` from a previous one?.Jan 14 2020, 6:14 PM2020-01-14 18:14:46 (UTC+0)
MGChecker removed a parent task: T73815: PDF should be rendered in the user's specified language converter variant, e.g. Serbian in Latin script (sr-el).Jun 3 2020, 7:14 AM2020-06-03 07:14:31 (UTC+0)
VulpesVulpes825 subscribed.Jul 18 2020, 2:57 PM2020-07-18 14:57:56 (UTC+0)
Pppery removed a project: Patch-For-Review.Mar 29 2023, 12:21 AM2023-03-29 00:21:17 (UTC+0)
Pppery added a project: MediaWiki-Language-converter.Apr 30 2023, 3:49 AM2023-04-30 03:49:11 (UTC+0)
Winston_Sung moved this task from Backlog to Core on the MediaWiki-Language-converter board.May 2 2023, 2:40 AM2023-05-02 02:40:51 (UTC+0)
Bugreporter added a subtask: T353501: new Parsoid cannot parse the converter wikitext syntax.Dec 15 2023, 4:51 PM2023-12-15 16:51:29 (UTC+0)
Winston_Sung reopened this task as Open.Dec 27 2023, 5:20 PM2023-12-27 17:20:39 (UTC+0)
Winston_Sung closed this task as a duplicate of T74010: [[zh:MediaWiki:Wikimedia-copyright/zh]] shows unexpected -{R| } prefixes for URLs.
Winston_Sung merged a task: T74010: [[zh:MediaWiki:Wikimedia-copyright/zh]] shows unexpected -{R| } prefixes for URLs.
Winston_Sung added subscribers: Aklapper, Bawolff.
MSantos added a project: Parsoid-Read-Views (Language Converter Support).May 13 2025, 10:42 AM2025-05-13 10:42:54 (UTC+0)
Winston_Sung moved this task from Core to Core - Parser/Language converter on the MediaWiki-Language-converter board.Apr 25 2026, 6:35 AM2026-04-25 06:35:29 (UTC+0)
Tiger3018 added a subtask: T417533: Round-trip testing regression with language converter markup.Sep 11 2026, 9:53 PM2026-09-11 21:53:56 (UTC+0)
Bewfip removed a subtask: T417533: Round-trip testing regression with language converter markup.Sep 12 2026, 12:49 PM2026-09-12 12:49:12 (UTC+0)
Log In to Comment
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.