GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | ||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | ||||||
GeezerVison is showing www.mediawiki.org — User:Mooeypoo/VisualEditor Development - MediaWiki Below are the full steps, as I've experienced them multiple times, in setting up a working development environment for MediaWiki and VisualEditor. Please note that this isn't an installation guide, this article is specifically meant to help contributors set up their system for contributions to MediaWiki core and VisualEditor. MotivationEverything in this guide already appears in the various guides online, either in GitHub or in MediaWiki documentation. However, from my experience, it tends to get a bit confusing to shift from one guide to the other, especially if you're just starting out with this type of development. So, I grouped everything together into one single newbie-friendly tutorial. If you have anything to add to this, feel free. AssumptionsThis guide assumes the user is working with a Linux-based system. I will concentrate on Debian / Ubuntu because that's the most beginner-friendly distribution out there, but each step also has information about other distributions. If your system is Windows, I recommend either using MediaWiki-Vagrant or install Linux under a Virtual Machine. VirtualBox or VMWare Player both work well in my experience. If you go with Vagrant, ignore this guide and follow the vagrant guide instead, it will handle setting up a development environment for MediaWiki and any extensions you set it up with. So, in this document I assume that:
Request a developer accountThe first step in getting into MediaWiki development is to ask for a developer access. This will allow you to make contributions through gerrit. Note: The 'instance shell account name' is used in the "username" field whenever you contact gerrit to pull, clone, push, or review code. Whenever you see this link "ssh://<username>@gerrit.wikimedia.org:29418" the 'username' is your instance shell account name you chose. It's also public, so choose wisely. You have to sign up to both sites. Use the same email details. Setting up the systemSet up gitThe best information about installing git is on GitHub documents. To save you the trouble, here it is in sequence for Debian / Ubuntu systems. Install$ apt-get install git For other distributions, follow the instructions at the git website. Set up your detailsThese details should match the details you signed up to gerrit and wikitech $ git config --global user.name "Your Name Here" $ git config --global user.email "[email protected] Optionally, you can set up colors in your terminal. This is useful to see errors or to spot trailing spaces. $ git config --global color.ui true Public keysTo be able to use ssh with gerrit, you must set up a public key gerrit can use to identify you are who say you are. This procedure was taken almost-verbatim from the GitHub tutorial. $ cd ~/.ssh # List the files in your .ssh directory: $ ls# Create a new ssh key, using the provided email as a label $ ssh-keygen -t rsa -C "[email protected]" Generating public/private rsa key pair. Enter file in which to save the key (/home/you/.ssh/id_rsa): # Add the key in the system $ ssh-add id_rsa Now you need to add the public key to both wikitech and gerrit. $ sudo apt-get install xclip # Downloads and installs xclip. If you don't have `apt-get`, you might need to use another installer (like `yum`) $ xclip -sel clip < ~/.ssh/id_rsa.pub # Copies the contents of the id_rsa.pub file to your clipboard Install git-reviewNext, we need to work with git review. This will allow us to send our work to gerrit so other developers can review it and (eventually) merge it into the codebase. The following instructions were taken from this page. You can see the full instructions there, as well as troubleshooting help. $ sudo apt-get install python-pip $ sudo easy_install pip $ sudo pip install git-review Configure git review to use your gerrit username: $ git config --global --add gitreview.username "gerritusername" Note: Do not install git-review via apt. It looks like it works, but doesn't. If you do not have easy_install set up, try sudo apt-get install python-setuptools Setting up MediaWikiThe next step is to install and set up MediaWiki from the development repository. It's also a good idea to set it up so you can test your code on a running local development wiki. Clone the repositoryIn your terminal, go to wherever you want your new mediawiki folder to be located and proceed with the following steps: $ git clone ssh://<username>@gerrit.wikimedia.org:29418/mediawiki/core.git <folder name> Where <username> is the 'instance shell account name' you chose when you signed up to wikitech and <folder name> is an optional entry for you to specify a folder for the code. If you don't specify the folder, the code will be cloned into a folder with the repo name -- in this case, a folder called 'core' After you've done that, you should initiailize git-review on this folder. $ git review -s The system will likely ask you for your gerrit username. Again, this is the same 'instance shell account name'. Get used to it, that's your ssh username from now on. Install LAMPIf your system already has some web service, php and mysql support, you should skip to Install Mediawiki (next section), although some php5 libraries indicated here are not standard and may not already be installed. $ sudo apt-get install apache2 $ sudo apt-get install mysql-server php5-mysql $ sudo mysql_install_db $ sudo apt-get install php5 libapache2-mod-php5 php5-mcrypt php5-curl php5-intl php5-gd Install MediaWikiAssuming your system already has some web service, php and mysql support, you should now install mediawiki so you can have a local working system you can use to test with.
$ ln -s <path to your mediawiki folder> /var/www/<symbolic name> The symbolic name is usually what you see in your browser url, like 'wiki' in http://localhost/wiki
Setting up ParsoidIn order to work with VisualEditor we will need to set up the parsoid service. These instructions are also available in the Parsoid installation page.
Before you beginIf you are using Parsoid in conjunction with VisualEditor, note that they are developed in parallel and upgrades to one often require a corresponding upgrade to the other. If in doubt, please check the Extension:VisualEditor page and follow its setup instructions first. Of course, Parsoid can be used stand-alone (to convert wikitext to HTML DOM and vice-versa) and if you don't need VisualEditor, you can ignore the above. If you are a developer, or if you lack access to sudo apt-get because you're on shared hosting, you probably want to follow the Developer Setup instructions. This page documents setup for a typical user of Parsoid using the native software packaging for your operating system. (Although, if your preferred operating system is not listed here, you might try the developer setup instructions — good luck!) If your shared hosting doesn't allow you to install Parsoid, you can also try this workaround: Installation on a shared host. If you run into problems, consult the troubleshooting pages. If you'd like to contribute your problems and suggested solutions to others, we encourage you to add that information to the troubleshooting pages, in order to keep this page of typical installation instructions as clear and simple as possible. First you will need to install Parsoid. After this is done, skip to the #Configuration section of this page in order to ensure that Parsoid can talk to your mediawiki instance. Please note that Parsoid 0.10.0 may not work with node 12, as described on Phabricator. InstallationUbuntu / Debian
Parsoid switched package repositories on 2015-09-17. If you installed Parsoid prior to this date, you will need to follow the instructions below to add the new Parsoid repository to get updated packages. If you are installing Parsoid as a new package, follow the instructions below.
Parsoid updated its GPG key on 2016-07-27 and on 2019-06-13 respectively. If you installed Parsoid prior to this date, you will need to follow the instructions below to add the new Parsoid GPG key before you will get updated packages. If you are installing Parsoid as a new package, follow the instructions below.
These packages work on all architectures and with current distros: Ubuntu 14+ and Debian testing, unstable or wheezy (stable) with backports enabled. See the manual installation on Linux or Mac OS X instructions if your distribution is older & doesn't have nodejs ≥ v8.x available. Import the repository gpg key: (key created on 2019-06-13 with validity 2029-04-23) sudo apt install dirmngr sudo apt-key advanced --keyserver keys.gnupg.net --recv-keys AF380A3036A03444 If the last command above isn't working (gpgkeys: key AF380A3036A03444 canot be retrieved), you can try another key server. This one should work : sudo apt-key advanced --keyserver pgp.mit.edu --recv-keys AF380A3036A03444 or (if the firewall is blocking) sudo apt-key advanced --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys AF380A3036A03444 Add the Wikimedia repository:
sudo apt-add-repository "deb https://releases.wikimedia.org/debian jessie-mediawiki main" On ubuntu 16.04 you may need to install "software-properties-common" in order to run "apt-add-repository" sudo apt-get install software-properties-common
With Jessie and Stretch, first make sure you have backports enabled, because nodejs 4.x is too old to run the latest release of Parsoid. Then, add the parsoid repository: echo "deb https://releases.wikimedia.org/debian jessie-mediawiki main" | sudo tee /etc/apt/sources.list.d/parsoid.list Install: sudo apt install apt-transport-https sudo apt update && sudo apt install parsoid Then, open the config file in Notes:
Caveats about the deb:
Error loading data from server: internal_api_error_Exception: [0db2f13b5ceecfae5a4c1a98] Exception caught: PHP cURL extension missing. Check https://www.mediawiki.org/wiki/Manual:CURL. Would you like to retry? solved with apt install php-curl systemctl reload apache2 PS: after posting this noticed https://www.mediawiki.org/wiki/Extension:VisualEditor#Troubleshooting mentions similar . ArchParsoid is available in AUR under aur/parsoid (release version) or aur/parsoid-git (development version). Install however you would usually install AUR packages. Enable and start the parsoid service ( This installs to RedHat/CentOS 7Parsoid is available from the Git repository. Try creating an empty directory and checking out a copy of Parsoid:
If you want to use a specific branch, use something like
This should dump everything you need into the current directory. Copy the default configuration (which configures the Parsoid server to listen at http://localhost:8000):
Edit this file per #Configuration (below), then start the server with:
At this point, opening a browser to localhost:8000 should display a page with links to the Parsoid documentation on www.mediawiki.org As one final step, change your startup files (init.d) to add a task to relaunch RedHat/CentOS 8dnf install git npm make python2 gcc-c++ -y cd /opt git clone --recursive https://gerrit.wikimedia.org/r/mediawiki/services/parsoid/deploy git clone https://gerrit.wikimedia.org/r/mediawiki/services/parsoid cd parsoid PYTHON=python2 npm install cp config.example.yaml config.yaml chown apache:apache /opt/parsoid Edit config.yaml. You may need to change uri parameter. Often localhost uri works, and default setting may be ok. Create this systemd unit file: # /etc/systemd/system/parsoid.service [Unit] Description=Mediawiki Parsoid web service on node.js Documentation=http://www.mediawiki.org/wiki/Parsoid Wants=local-fs.target network-online.target After=local-fs.target network-online.target [Install] WantedBy=multi-user.target [Service] Type=simple User=apache Group=apache WorkingDirectory=/opt/parsoid ExecStart=/usr/bin/node /opt/parsoid/bin/server.js KillMode=process Restart=on-failure PrivateTmp=true StandardOutput=syslog Create <?xml version="1.0" encoding="utf-8"?> <service> <short>Parsoid</short> <description>Wikitext converter for Mediawiki</description> <port protocol="tcp" port="8000"/> </service> Publish, add and activate the newly defined firewalld service. firewall-cmd --reload firewall-cmd --add-service=parsoid --permanent firewall-cmd --reload Enable the systemd service and verify if Parsoid is reachable. systemctl enable parsoid --now curl http://localhost:8000 VagrantIf you are using the MediaWiki-Vagrant virtual machine, the WindowsRequirements:
With Node.js you have to install the build tools using PowerShell 32-bit (as administrator) (this may take a while) npm install --global --production windows-build-tools You may need to update your npm version to avoid errors npm -g install npm@latest npm@next is broken. See https://github.com/npm/npm/issues/16037. If your current directory is in cd $env:userprofile Install Parsoid: npm install parsoid Note: If you get any error such as npm init Or enter the directory in which the %USERPROFILE%\node_modules\parsoid\ Copy the default configuration and configure Parsoid: copy %USERPROFILE%\node_modules\parsoid\localsettings.example.js %USERPROFILE%\node_modules\parsoid\localsettings.js If you use the copy %USERPROFILE%\node_modules\parsoid\config.example.yaml %USERPROFILE%\node_modules\parsoid\config.yaml And edit the content according to your wiki installation. Run Parsoid: C:\Users\USERNAME\node_modules\parsoid>node bin/server.js
Or on Windows 10 or later: cd %APPDATA%\npm\node_modules\parsoid node bin/server.js You should also open your
Otherwise you will have some trouble with Parsoid. DockerThis is a version of Parsoid created by community. The original repository can be found at TheNets's GitHub. Versions available: The images was created over Alpine image. RequirementsHow to runTo start Parsoid run the command below. Just pay attention to the MediaWiki version and choose a compatible Parsoid version. # For MediaWiki <= 1.30 docker run -d -p 8080:8000 -e PARSOID_DOMAIN_localhost=http://localhost/w/api.php thenets/parsoid:0.8 # For MediaWiki >= 1.31 & <= 1.32 docker run -d -p 8080:8000 -e PARSOID_DOMAIN_localhost=http://localhost/w/api.php thenets/parsoid:0.10 # For MediaWiki >= 1.33 docker run -d -p 8080:8000 -e PARSOID_DOMAIN_localhost=http://localhost/w/api.php thenets/parsoid:0.11 ExamplesHow to add more than one domain: docker run -d -p 8080:8000 \ -e PARSOID_DOMAIN_foobar=http://foobar.com/w/api.php \ -e PARSOID_DOMAIN_example=http://example.com/w/api.php \ -e PARSOID_DOMAIN_localhost=http://localhost/w/api.php \ thenets/parsoid:0.11 How to expose on a specific port: (You can use arbitrary port numbers which are not already in use) # Expose port 8081 docker run -d -p 8081:8000 -e PARSOID_DOMAIN_localhost=http://localhost/w/api.php thenets/parsoid:0.11 # Expose port 8142 docker run -d -p 8142:8000 -e PARSOID_DOMAIN_localhost=http://localhost/w/api.php thenets/parsoid:0.11 For more information about Docker setup, check the GitHub page. ConfigurationYou can find an example configuration file on github. Starting with Parsoid 0.6.0, the configuration file is located here:
If the The If your wiki is inside a reverse proxy configuration or similar, you can set the hostname in For example: you have a wiki at By default, Parsoid opens a UDP socket and send each minute some metrics about the Parsoid heap to a statsD server. If you want to send instead these metrics to the logging backend (with the "trace" log level), add in metrics: type: log localsettings.js (or settings.js) as configuration fileIf you prefer to use services: - module: lib/index.js entrypoint: apiServiceWorker conf: # For backwards compatibility, and to continue to support non-static # configs for the time being, optionally provide a path to a # localsettings.js file. See localsettings.example.js localsettings: ./localsettings.js and comment #mwApis: #- # This is the only required parameter, # the URL of you MediaWiki API endpoint. #uri: 'http://localhost/w/api.php' # The "domain" is used for communication with Visual Editor # and RESTBase. It defaults to the hostname portion of # the `uri` property below, but you can manually set it # to an arbitrary string. #domain: 'localhost' # optional In this approach the configuration file lives in one of the following locations:
Most configuration options are described in the file itself. The only required edit is to update it to reflect your API URL, something like: parsoidConfig.setMwApi({ uri: 'http://yoursite.com/w/api.php', domain: 'yoursite.com', prefix: 'myspecialwiki' }); The
Multiple wikis sharing the same parsoid serviceIf you have multiple wikis, make sure the Example of configuration in services: - module: lib/index.js entrypoint: apiServiceWorker conf: mwApis: - # First wiki uri: 'http://yoursite.com/w/api.php' domain: 'yoursite.com' # optional - # If you have another wiki on a different domain uri: 'http://yourothersite.com/w/api.php' domain: 'yourothersite.com' # optional - # If you have another wiki on the same domain uri: 'http://yoursite.com/w2/api.php' domain: 'wiki2' # optional Example of configuration in parsoidConfig.setMwApi({ uri: 'http://yoursite.com/w/api.php', domain: 'yoursite.com', prefix: 'myspecialwiki' }); // If you have another wiki on a different domain: parsoidConfig.setMwApi({ uri: 'http://yourothersite.com/w/api.php', domain: 'yourothersite.com', prefix: 'myotherspecialwiki' }); // If you have another wiki on the same domain: // Note that the domain and prefix can be arbitrary, they just need to be unique to this wiki. // (And they shouldn't contain slashes.) parsoidConfig.setMwApi({ uri: 'http://yoursite.com/w2/api.php', domain: 'wiki2', prefix: 'wiki2' }); See User:Mooeypoo/Troubleshooting#Configuration for additional troubleshooting help. See Parsoid/Setup/RESTBase for information on how to configure a local RESTBase instance between your local VisualEditor and local Parsoid. The Parsoid/Setup/RESTBase/Arbitrary domains page describes advanced RESTBase setup, but it may offer additional insight about the purpose of the
Setting up VisualEditorWe're now at the stage where we can set up VisualEditor for both development and testing environment. You can see a fully detailed explanation on how to set VisualEditor up in the official setup page. Clone the extensionIn your terminal, go to your mediawiki folder and into /extensions. The VisualEditor repository must be cloned into the mediawiki/extensions folder. $ git clone ssh://<username>@gerrit.wikimedia.org:29418/mediawiki/extensions/VisualEditor.git Activate the submodules $ git submodule update --init Add to LocalSettings.phpOpen your mediawiki's LocalSettings.php file. Add the following to the end: require_once("$IP/extensions/VisualEditor/VisualEditor.php"); // OPTIONAL: Enable VisualEditor in other namespaces // By default, VE is only enabled in NS_MAIN //$wgVisualEditorNamespaces[] = NS_PROJECT; // Enable by default for everybody $wgDefaultUserOptions['visualeditor-enable'] = 1; // Don't allow users to disable it $wgHiddenPrefs[] = 'visualeditor-enable'; // OPTIONAL: Enable VisualEditor's experimental code features //$wgVisualEditorEnableExperimentalCode = true; Notice that if you're contributing code to VisualEditor, it might be a good idea to enable experimental features by uncommenting the line with We also have to link VisualEditor with the parsoid service. Again, at the end of the LocalSettings.php file, add the following // URL to the Parsoid instance // MUST NOT end in a slash due to Parsoid bug $wgVirtualRestConfig['modules']['parsoid']['url'] = 'http://localhost:8000'; Congratulations! You have a working environment with MediaWiki, VisualEditor and parsoid running. Setting up grunt testsVisualEditor contains 'grunt', a javascript task runner that includes tasks to verify the code. This is crucial - it allows you to spot errors (to code and to style) before you commit your work. To set up grunt in your new VisualEditor folder, run the following commands: $ npm install $ cd lib/ve $ npm install Now go back to the main VisualEditor folder and test your environment. TestingTest your installation by going into your wiki. You should have "Edit" and "Edit Source" tabs. Click on "Edit" and make sure VisualEditor is loading properly. If it doesn't, or if there is only one "Edit" tab, please see the VisualEditor installation page for troubleshooting. Make sure parsoid's settings are correct and that the service is running. Make sure to test grunt by typing $ grunt test From your extensions/VisualEditor folder. Useful ToolsYour system is ready for development now. There are several tools that can help you start (and continue) working more easily. gerrit referenceThere's a very good gerrit reference here. Here's the quickie on how to work with gerrit: To start a new patch, make sure your repository is updated by checking out master and pulling: $ git checkout master $ git pull $ git submodule update Then, start a new branch for your code: $ git checkout -b [BRANCH NAME] # See tips about branch name in the tutorial Change your files, do what you do, then when you are done, check the changes you've made $ git status You should see a list of changed files. If you see 'lib/ve' in your list of changed files, do not do 'git commit --all'. Instead, add individual files to the commit manually: $ git add <path/to/file> And then commit without --all $ git commit The commit noteWrite a meaningful, helpful and explanatory commit note:
First commit in the branch$ git commit --all NOTE: Be careful. Since VisualEditor uses a Amending a previous commitIf the commit already exists -- either because you're fixing up some existing branch or reviewing one from gerrit, you should make sure to amend the commit $ git commit --all --amend Note again that Useful shortcutsYou'll work in the terminal quite a lot, so terminal shortcuts are very useful. Here are a couple that I am using. In order to set up shortcuts in the terminal, edit your Note: Some of these are shamelessly stolen and adapted from Krinkle's aliases, so thank him when you catch him online. Shortcut: update and activate parsoidYou can choose your own alias name, of course. This will update the Parsoid repository and run the service. It saves a lot of time instead of going to the parsoid folder and running alias goparsoid='cd [path/to/parsoid] && git remote update && git checkout origin/master && npm install && npm start' Shortcut: update MediaWiki and VisualEditorI use this every time I want to completely refresh my development system and make sure both MediaWiki and VisualEditor are up to date with the latest versions of the master branch. alias updateve='cd [path/to/mediawiki] && echo "Updating MediaWiki" && git checkout master && git pull && cd extensions/VisualEditor && echo "Updating VisualEditor" && git checkout master && git pull' Note: The new aliases won't work until you reload You can either close and reopen your terminal or run $ reload ~/.bashrc Useful Links | ||||||
| You are visitor 01402503 — GeezerVison does not store the pages you look up. Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required. |