Robots or bots are automatic processes that interact with Wikipedia (and other Wikimedia projects) as though they were human editors. This page attempts to explain how to carry out the development of a bot for use on Wikimedia projects and much of this is transferable to other wikis based on MediaWiki. The explanation is geared mainly towards those who have some prior programming experience, but are unsure of how to apply this knowledge to creating a Wikipedia bot.
Bots can automate tasks and perform them much faster than humans. If you have a simple task that you need to perform lots of times (an example might be to add a template to all pages in a category with 1000 pages), then this is a task better suited to a bot than a human.
It is often far simpler to request a bot job from an existing bot. If you have only periodic requests or are uncomfortable with programming, this is usually the best solution. These requests can be made at Wikipedia:Bot requests. In addition, there are a number of tools available to anyone. Most of these take the form of enhanced web browsers with Mediawiki-specific functionality. The most popular of these is AutoWikiBrowser (AWB), a browser specifically designed to assist with editing on Wikipedia and other Wikimedia projects. A mostly complete list of tools can be found at Wikipedia:Tools/Editing tools. Tools, such as AWB, can often be operated with little or no understanding of programming.
If you decide you need a bot of your own due to the frequency or novelty of your requirements, you don't need to write one from scratch. There are already a number of bots running on Wikipedia and many of these bots publish their source code, which can sometimes be reused with little additional development time. There are also a number of standard bot frameworks available. Modifying an existing bot or using a framework greatly speeds development time. Also, because these code bases are in common usage and are maintained community projects, it is far easier to get bots based on these frameworks approved for use. The most popular and common of these frameworks is Pywikibot (PWB), a bot framework written in Python. It is thoroughly documented and tested and many standardized Pywikibot scripts (bot instructions) are already available. Other examples of bot frameworks can be found below. For some of these bot frameworks, such as PWB, a general familiarity with scripts is all that is necessary to run the bot successfully (it is important to update these frameworks regularly).
Writing a new bot requires significant programming ability. A completely new bot must undergo substantial testing before it will be approved for regular operation. To write a successful bot, planning is crucial. The following considerations are important:
Just like a human editor, a Wikipedia bot reads Wikipedia pages, and makes changes where it thinks changes need to be made. The difference is that, although bots are faster and less prone to fatigue than humans, they are nowhere near as bright as we are. Bots are good at repetitive tasks that have easily defined patterns, where few decisions have to be made.
In the most typical case, a bot logs in to its own account and requests pages from Wikipedia in much the same way as a browser does – although it does not display the page on screen, but works on it in memory – and then programmatically examines the page code to see if any changes need to be made. It then makes and submits whatever edits it was designed to do, again in much the same way a browser would.
Because bots access pages the same way people do, bots can experience the same kind of difficulties that human users do. They can get caught in edit conflicts, have page timeouts, or run across other unexpected complications while requesting pages or making edits. Because the volume of work done by a bot is larger than that done by a live person, the bot is more likely to encounter these issues. Thus, it is important to consider these situations when writing a bot.
In order to make changes to Wikipedia pages, a bot necessarily has to retrieve pages from Wikipedia and send edits back. There are several Application Programming Interfaces (APIs) available for that purpose.
api.php). This library was specifically written to permit automated processes such as bots to make queries and post changes. Data is returned in JSON format (see output formats for more details).
action=raw&templates=expandGET request to index.php will give the unprocessed wikitext source code of a page. An API query with
action=query&prop=revisions&rvprop=content&rvexpandtemplates=1is roughly equivalent, and allows for retrieving additional information.
Some Wikipedia web servers are configured to grant requests for compressed (gzip) content. This can be done by including a line "Accept-Encoding: gzip" in the HTTP request header; if the HTTP reply header contains "Content-Encoding: gzip", the document is in gzip form, otherwise, it is in the regular uncompressed form. Note that this is specific to the web server and not to the MediaWiki software. Other sites employing MediaWiki may not have this feature. If you are using an existing bot framework, it should handle low-level operations like this.
Approved bots need to be logged in to make edits. Although a bot can make read requests without logging in, bots that have completed testing should log in for all activities. Bots logged in from an account with the bot flag can obtain more results per query from the Mediawiki API (api.php). Most bot frameworks should handle login and cookies automatically, but if you are not using an existing framework, you will need to follow these steps.
To log a bot in using the MediaWiki API, 2 requests are needed:
where TOKEN is the token from the previous result. The HTTP cookies from the previous request must also be passed with the second request.
A successful login attempt will result in the Wikimedia server setting several HTTP cookies. The bot must save these cookies and send them back every time it makes a request (this is particularly crucial for editing). On the English Wikipedia, the following cookies should be used: enwikiUserID, enwikiToken, and enwikiUserName. The enwiki_session cookie is required to actually send an edit or commit some change, otherwise the MediaWiki:Session fail preview error message will be returned.
Main-account login via "action=login" is deprecated and may stop working without warning. To continue login with "action=login", see Special:BotPasswords.
Wikipedia uses a system of edit tokens for making edits to Wikipedia pages, as well as other operations that modify existing content such as rollback. The token looks like a long hexadecimal number followed by '+\', for example:
The role of edit tokens is to prevent "edit hijacking", where users are tricked into making an edit by clicking a single link.
The editing process involves two HTTP requests. First, a request for an edit token must be made. Then, a second HTTP request must be made that sends the new content of the page along with the edit token just obtained. It is not possible to make an edit in a single HTTP request. An edit token remains the same for the duration of a logged-in session, so the edit token needs to be retrieved only once and can be used for all subsequent edits.
To obtain an edit token, follow these steps:
The token will be returned in the
edittoken attribute of the response.
If the edit token the bot receives does not have the hexadecimal string (i.e., the edit token is just '+\') then the bot most likely is not logged in. This might be due to a number of factors: failure in authentication with the server, a dropped connection, a timeout of some sort, or an error in storing or returning the correct cookies. If it is not because of a programming error, just log in again to refresh the login cookies. The bots may use Assert Edit Extension to make sure that they are logged in.
Edit conflicts occur when multiple, overlapping edit attempts are made on the same page. Almost every bot will eventually get caught in an edit conflict of one sort or another, and should include some mechanism to test for and accommodate these issues.
Bots that use the Mediawiki API (api.php) should retrieve the edit token, along with the
starttimestamp and the last revision "base" timestamp, before loading the page text in preparation for the edit;
prop=info|revisions can be used to retrieve both the token and page contents in one query ( ). When submitting the edit, set the
basetimestamp attributes, and check the server responses for indications of errors. For more details, see mw:API:Edit - Create&Edit pages.
Generally speaking, if an edit fails to complete the bot should check the page again before trying to make a new edit, to make sure the edit is still appropriate. Further, if a bot rechecks a page to resubmit a change, it should be careful to avoid any behavior that could lead to an infinite loop and any behavior that could even resemble edit warring.
Actually, coding or writing a bot is only one part of developing a bot. You should generally follow the development cycle below to ensure that your bot follows Wikipedia's bot policy. Failure to comply with the policy may lead to your bot failing to be approved or being blocked from editing Wikipedia.
Implementation (or coding) involves turning design and planning into code. It may be the most obvious part of the software engineering job, but it is not necessarily the largest portion. In the implementation stage you should:
A good way of testing your bot as you are developing is to have it show the changes (if any) it would have made to a page, rather than actually editing the live wiki. Some bot frameworks (such as pywikibot) have pre-coded methods for showing diffs. During the approvals process, the bot will most likely be given a trial period (usually with a restriction on the number of edits or days it is to run for) during which it may actually edit to enable fine-tuning and iron out any bugs. At the end of the trial period, if everything went according to plan, the bot should get approval for full-scale operation.
An important (and often overlooked) task is documenting the internal design of your bot for the purpose of future maintenance and enhancement. This is especially important if you are going to allow clones of your bot. Ideally, you should post the source code of your bot on its userpage or in a revision control system (see #Open-source bots) if you want others to be able to run clones of it. This code should be well documented (usually using comments) for ease of use.
You should be ready to respond to queries about or objections to your bot on your user talk page, especially if it is operating in a potentially sensitive area, such as fair-use image cleanup.
Maintaining and enhancing your bot to cope with newly discovered bugs or new requirements can take far more time than the initial development of the software. To ease maintenance, document your code from the beginning.
Major functionality changes of approved bots must be approved.
In addition to the official bot policy, which covers the main points to consider when developing your bot, there are a number of more general advisory points to consider when developing your bot.
If your bot is doing anything that requires judgment or evaluation of context (e.g., correcting spelling) then you should consider making your bot manually-assisted, which means that a human verifies all edits before they are saved. This significantly reduces the bot's speed, but it also significantly reduces errors.
It should be easy to quickly disable your bot. If your bot goes bad, it is your responsibility to clean up after it! You could have the bot refuse to run if a message has been left on its talk page, on the assumption that the message may be a complaint against its activities; this can be checked using the API
meta=userinfo query ( ). Or you could have a page that will turn the bot off when changed; this can be checked by loading the page contents before each edit.
Just like a human, if your bot makes edits to a talk page on Wikipedia, it should sign its post with four tildes (~~~~). Signatures belong only on talk namespaces with the exception of project pages used for discussion (e.g., articles for deletion).
A bot's edits will be visible at Special:RecentChanges, unless the edits are set to indicate a bot. Once the bot has been approved and given its bot flag permission, one can add the "bot-True" to the API call - see mw:API:Edit#Parameters in order to hide the bot's edits in Special:RecentChanges. In Python, using either mwclient or wikitools, then adding Bot=True to the edit/save command will set the edit as a bot edit - e.g.
PageObject.edit(text=pagetext, bot=True, summary=pagesummary).
Many bot operators choose to make their code open source, and occasionally it may be required before approval for particularly complex bots. Making your code open source has several advantages:
Open-source code, while rarely required, is typically encouraged in keeping with the open and transparent nature of Wikipedia.
Before sharing code, make sure that sensitive information such as passwords is separated into a file that isn't made public.
There are many options available for users wishing to make their code open. Hosting the code in a subpage of the bot's userspace can be a hassle to maintain if not automated and results in the code being multi-licensed under Wikipedia's licensing terms in addition to any other terms you may specify. A better solution is to use a revision control system such as SVN, Git, or Mercurial. Wikipedia has articles comparing the different software options and websites for code hosting, many of which have no cost.
Bots can be written in almost any programming language. The choice of a language depends on the experience and preferences of the bot writer, and on the availability of libraries relevant to bot development. The following list includes some languages commonly used for bots:
GNU Awk is an easy language for bots small and large, including OAuth.
If located on a webserver, you can start your program running and interface with your program while it is running via the Common Gateway Interface from your browser. If your internet service provider provides you with webspace, the chances are good that you have access to a perl build on the webserver from which you can run your Perl programs.
PHP can also be used for programming bots. MediaWiki developers are already familiar with PHP, since that is the language MediaWiki and its extensions are written in. PHP is an especially good choice if you wish to provide a webform-based interface to your bot. For example, suppose you wanted to create a bot for renaming categories. You could create an HTML form into which you will type the current and desired names of a category. When the form is submitted, your bot could read these inputs, then edit all the articles in the current category and move them to the desired category. (Obviously, any bot with a form interface would need to be secured somehow from random web surfers.)
The PHP bot functions table may provide some insight into the capabilities of the major bot frameworks.
|Key people[php 1]||Name||PHP Version||last update||Uses API[php 2]||Exclusion compliant||Admin functions||Plugins||Repository||Notes|
|User:Cyberpower678, User:Addshore, and User:Jarry1250||Peachy||5.2.1||2017||Yes||Yes||Yes||Yes||GitHub||Large framework, currently undergoing rewrite. Documentation currently non-existent, so poke User:Cyberpower678 for help.|
|User:Addshore||mediawiki-api-base||5.3–7||2018||Yes||N/A||N/A||extra libs||GitHub||Base library for interaction with the mediawiki api, provides you with ways to handle logging in, out and handling tokens as well as easily getting and posting requests.|
|User:Addshore||mediawiki-api||5.3||2019||Yes||No||some||extra libs||GitHub||Built on top of mediawiki-api-base this adds more advanced services for the api such as RevisionGetter, UserGetter, PageDeleter, RevisionPatroller, RevisionSaver etc. Supports chunked uploading.|
|User:nzhamstar||Wikimate||5.3.2||2019||Yes||No||No||No||GitHub||Supports main article and file stuff. Authentication, checking if pages exist, reading and editing pages/sections. Getting file information, downloading and uploading files. Tested and working. Aims to be easy to use.|
|Григор Гачев||Apibot||5.1||2015||Yes||Yes||Yes||Yes||on wiki||Full API support up to MW 1.21 incl., persistent connections, gzipped xfers, HTTPS, HTTP auth, GET sorting, auto site/user/paraminfo caching and usage, page bot exclusion compliance, close to 1000 functions, DB support, etc etc. Easily extendable modular structure. An UNIX-like overlayed 'assembly line' framework. AGPL 3.0 or later.|
|Fork of older wikibot.classes (used by ClueBot and SoxBot). Updated for 2010 and 2015 API changes. Supports file uploading.|
Microsoft .NET is a set of languages including C#, C++/CLI, Visual Basic .NET, J#, JScript .NET, IronPython, and Windows PowerShell. Using Mono Project, .NET programs can also run on Linux, Unix, BSD, Solaris and macOS as well as under Windows.