5. Development¶
If you are only looking to contribute to XTools as a developer, it is recommended that you use a Toolforge account to connect to the WMF replicas so that you’ll have live data that matches production. Requesting an account should be the first thing you do, since it involves an approval process. Having access to Toolforge may benefit you in working on other Wikimedia projects, too.
5.1. Overview¶
- XTools is based on Symfony 4, which is a full MVC framework. We use Twig as our template engine.
- All the PHP lives in
src/
.- There is a single bundle called
AppBundle
, which contains the controllers, event listeners, helpers and Twig extensions.
- There is a single bundle called
- Views live in
templates/
and assetsassets/
. - Routes are configured using the
@Route
annotation. - By convention, each tool has it’s own controller that handles requests, instantiates a model, sets the repository, and returns it to the view. Not all the tools follow this convention, however. Each tool is also registered within
config/tools.yml
. - XTools was built to work on any MediaWiki installation, but it’s target wiki farm is Wikimedia. Some features are only available on the Wikimedia installation, which is what all the
app.is_labs
checks are for.
5.2. Running Development server¶
First make sure you meet the Prerequisites, and then follow these steps:
- Clone the repository:
git clone https://github.com/x-tools/xtools.git && cd xtools
- Run
composer install
and answer all the prompts. - For internal usage stats, create a new local database:
./bin/console doctrine:database:create
(ord:d:c
) and run the database migrations:./bin/console doctrine:migrations:migrate
(ord:m:m
). - Compile assets with
./node_modules/.bin/encore dev --watch
(for production, useproduction
instead ofdev
and without the watch flag). - Launch PHP’s built-in server:
./bin/console server:run
(ors:r
). - Visit
http://localhost:8000
in your web browser. - You can stop the server with
./bin/console server:stop
(ors:s
)
The Simple Counter is the simplest tool and should work as soon as you set up XTools.
Test it by going to http://localhost:8000/sc and put in Jimbo Wales
as the Username and en.wikipedia.org
as the Wiki.
After submitting you should quickly get results.
The development server does not cache application or session data; any changes you make are visible after refreshing the page.
However when you modify the config/parameters.yml
file or other things in config/
, you may need to clear the cache with php bin/console c:c --no-warmup
.
Assets are generated with Symfony Encore.
The logs are in var/logs/dev.log
.
If things are acting up unexpectedly, you might try clearing the cache or restarting the server.
5.3. Developing against WMF replicas¶
If you want to use the WMF database replicas (recommended), first make sure you have an account and shell access. Then you can open up the necessary tunnels and a shell session with:
ssh -L 4711:enwiki.web.db.svc.eqiad.wmflabs:3306 -L 4712:tools-db:3306 your-username@tools-dev.wmflabs.org
And set the following in app/config/parameters.yml
:
app.is_labs: 1
database_replica_host: 127.0.0.1
database_replica_port: 4711
database_replica_name: meta_p
database_meta_name: meta_p
database_replica_user: <your-db-username-here>
database_replica_password: <your-db-password-here>
database_toolsdb_host: 127.0.0.1
database_toolsdb_port: 4712
database_toolsdb_name: tools-db
Change the your-*-here
bits to your own values, which you can find in your replica.my.cnf
file in the home directory of your account on `Toolforge`_.
5.4. Table mappings¶
The replicas have different versions of tables that utilize indexing to improve performance. We’ll want to take advantage of that.
Go to the config directory with
cd app/config
Create the file table_map.yml from the template:
cp table_map.yml.dist table_map.yml
Set the contents of the file to the following:
parameters: app.table.archive: 'archive_userindex' app.table.revision: 'revision_userindex'
For the logging
table, sometimes we use logging_userindex
and other times logging_logindex
(depending on what we’re querying for). This is handled in the code via the getTableName()
method in Repository.php
.
For non-WMF installations, ignore all of the above unless your tables have different names than the MediaWiki defaults.
5.5. Caching¶
Caching should be done only in Repositorys, using this pattern:
public function doSomething($input)
{
$cacheKey = $this->getCacheKey(func_get_args(), 'something');
if ($this->cache->hasItem($cacheKey)) {
return $this->cache->getItem($cacheKey)->get();
}
$results = 'results of big query';
return $this->setCache($cacheKey, $results);
}
The cache key can be anything, so long as it is unique to the specific method.
A third parameter can be passed to setCache
to set the TTL, using the same syntax from the DateInterval class
(e.g. P1D
is one day, PT1H
is one hour).
The above methods are just wrappers around a PSR-6 implementation, intended to reduce the repetition of similar lines of code.
You can, of course, retrieve the underlying CacheItemPoolInterface whenever you want with $container->get('cache.app')
.
5.6. Style Guidelines¶
- It’s called “XTools”, with two capital letters.
- XTools conforms to PSR2 and Slevomat coding standards; use
./vendor/bin/phpcs -s .
to check your code. - Functions and routes must begin with the tool name.
- Version numbers follow Semantic Versioning guidelines.
5.7. Tests¶
Tests are located in the tests/
directory, and match the src/
directory structure. They are built with PHPUnit. Repositories only handle fetching data and do not need to be tested. Controllers also interact with the database, and while tests are most welcomed for these, they will not run on the continuous integration server (Travis and Scrutinizer) due to limitations.
There are also tests for linting, phpDoc blocks, and file permissions.
Use composer test
to run the full suite, or ./vendor/bin/phpunit tests/
to run just the unit tests.
5.8. Writing the docs¶
We use ReadTheDocs. To build this documentation locally, you need python-sphinx
installed,
as well as the sphinx_rtd_theme
theme.
Then, it’s simply a matter of running make clean && make html
in the docs/
directory,
and browsing to xtools/docs/_build/html/
to view the documentation.
Documentation sections use the following (standard Python) hierarchy of section symbols:
#
with overline for parts*
with overline for chapters=
for sections-
for subsections
5.9. Releases¶
Releases are made by tagging commits in the master branch. Before tagging a new release:
- Update the version numbers in
docs/conf.py
andapp/config/version.yml
. - Check the copyright year in
README.md
,docs/conf.py
, andapp/Resources/views/base.html.twig
. - If assets were modified, bump the version number in config.yml under framework/assets/version.
- Update
RELEASE_NOTES.md
with any notable new information for the end user.
Then tag the release (follow the Semantic Versioning guidelines, and annotate the tag with the above release notes) and push it to GitHub.
Lastly, update the version
and updated
parameters at https://www.mediawiki.org/wiki/XTools
5.10. Additional Help¶
- Email:
tools.xtools
@tools.wmflabs.org
- IRC: #wikimedia-xtools (Direct link - Requires an IRC client)
- MediaWiki talk page: Talk:XTools