information and language processing systems

isla, university of amsterdam

WikiXML: XML format

Each page in the WikiXML collection (i.e., article, category, image or template) is provided as a separate XML document that (in most cases - see below) conforms to the WikiXML DTD which is an extension of the XHTML 1.0 Transitional DTD with elements and attributes specific to the WikipediaXML collection. XML elements and attributes extending the XHTML format all belong to the wx namespace. Stripping such elements and attributes from the files of WikipediaXML results in valid XHTML documents.

XML files for a small number (about 1%) of pages do not conform to the WikiXML DTD, although all files are guaranteed to contain well-formed XML. Such non-valid pages are marked in the page table of the database distributed with the collection.

The XML format of the collection differs from Wikipedia's regular XHTML output in several aspects. Sections below describe the differences.

Page content and meta-data

• Page decoration (navigation bars, header, footer, etc.) is removed;
• Page subtitle (From Wikipedia, the free encyclopedia) is removed;
• Table-of-contents is removed and its original location is marked with <div id="wx_toc"></div>;
• Per-section [edit] links are removed;
• <title> element reflects prefixed page name (e.g., Category:Operas_by_Pietro_Mascagni);
• <meta> elements provide the namespace, page id, page name, and redirection target for redirect pages;
• Article body is contained in a <div id="wx_article"> element;
• The converter adds id attributes to all XML elements on every page (with the exception of <wx:templateend> elements);
• The converter also performs some tweaking of HTML tags to fix common validity errors (making result a proper XML).

Sections

The XML converter marks sections of pages using <wx:section level="nnn" title="Header of the section"> (here 1 ≤ nnn ≤ 6). In addition, sections include the usual XHTML header elements <h[123456]>. Sections end at the following header of same or smaller level or at the end of the containing element. Sections can be nested.

Links

In general, the WikipediaXML collection uses XHTML <a...> elements to mark up all links in pages.

• Category links (links to categories of an article) are contained in a <div id="wx_categorylinks"> element at the end of pages;
• Language links (links to versions of an article in other languages) are contained in a <div id="wx_languagelinks"> element;
• Links to resources outsite Wikipedia (external links):
  • title attribute is removed from <a> element;
  • Extra attribute wx:linktype="external" is added;
  • Anchor text of auto-numbered links ([1], [2], etc.) are removed;
• Links to sections of the same page (intra-page links):
  • Extra attribute wx:linktype="fragment" is added;
  • Extra attribute wx:fragment contains the anchor name of the link target (name attribute of the target <a> element);
• Links to pages in the same Wikipedia collection (known links):
  • Extra attribute wx:linktype="known" is added;
  • Extra attribute wx:pagename contains the prefixed name of the target page;
  • Extra attribute wx:page_id contains the page id of the target page, unless the target namespace is not in the scope of our collection;
  • If the link refers to a specific section on the target page, extra attribute wx:fragment contains the anchor name of the target section;
• Links to Wikipedia pages that are not part of the collection (unknown links):
  • Extra attribute wx:linktype="unknown" is added;
  • Extra attribute wx:pagename contains the prefixed name of the target page;
  • Extra attribute wx:fragment contains the anchor name in the link target page, if provided;
  • href attribute points to a standard /wiki/Page_Name URL instead of the edit script;
• Wiki links to the Media namespace:
  • Attribute wx:linktype="media"
  • href attribute points to a standard /wiki/... URL instead of the target media item
• Links to pages in other wikis or other languages (interwiki links):
  • Extra attribute wx:linktype="interwiki" is added;
  • Extra attribute wx:pagename contains a qualified name of the target page, such as wiki:My_page or wiki:Category:My_category
  • Extra attribute wx:fragment contains the anchor name in the link target page, if provided;
• Links to the current page (self-links):
  • Treated as a usual wiki link (Wikipedia's MediaWiki would drop the link and simply display text in boldface);
  • Attribute wx:linktype="self" is added;
• Links to footnotes (created through <ref> markup in wiki-text):
  • Extra attribute wx:linktype="note" is added
  • Anchor text (autonumber) is suppressed;
• Backlinks from a footnote to the reference:
  • Extra attribute wx:linktype="noteref" is added;

Templates

Templates (e.g., {{infobox | param | param}}, see here) are expanded as usual. Additionally:

• If possible, the start of an expansion of a template is marked with <wx:template id="xxx" pagename="Template:name" page_id="nnn"/> and the end is marked with <wx:templateend start="xxx"/> (here xxx is an element id);
• Arguments of the templates used in a page are explicitely listed in the end of XML file, outside the actual content of the page: there is one element <wx:templatearguments for="xxx"> for each invocation of a template, containing zero or more elements <wx:argument name="name"> value </wx:argument> that specify names and values of the template arguments as defined in the original wiki-text of the page; for example, the article Germany uses the template Infobox Country or territory and its argument capital is a link to the article Berlin;
• The XML converter performs some limited wiki processing of template arguments as provided in the wiki-text, mostly to discover and mark up wiki links.
• Marking of template boundaries may fail, in which case the template will still be expanded correctly but will not be recognizable afterwards;

In general, interpreting Wikipedia's template argument values is very hard, because a template can transform its arguments in many ways: namespaces can be added, links inserted or removed, etc.; the converter only handles easily identifiable usages of template arguments;

The marking of the boudaries of template expansions with two empty elements <wx:template .../> expanded template <wx:templateend.../> is probably not the nicest solution and may be not convenient to work with. It is necessary, however, because expanded templates are not always balanced XML fragments and can therefore not be contained inside a single XML element such as <wx:template...> expanded template </template>

Images

Images included in Wikipedia pages are marked up with XHMTL img and a elements:

• <a> element gets extra attributes: wx:linktype="image", wx:pagename, wx:page_id (the name and id of the corresponding image page, if included in the collection);
• The src attribute of the <img> element is set to /wiki/Image:...
• If the image is framed, thumbnailed or floating, it is embedded in a <div> element together with the optional image caption;
• This markup is not currently provided for image galleries;
• On image pages, the list of pages that use the image is suppressed.

Miscellaneous

• The content of <math> markup (used in wiki-text to display formulas, see here) is passed to the XML output unmodified;
• <timeline> scripts (see here) are passed to the XML output unmodified;
• <hiero> markup (see WikiHiero) is passed to the XML output unmodified;
• On category pages, the list of member articles and subcategories is suppressed; the categorization information is provided in the XML versions of member article pages and subcategory pages, as well as in the page_category table in the database;
• Magic variables (e.g., {{FULLPAGENAME}}) are expanded as in MediaWiki's XHTML output, and moreover:
  • If possible, marked with <wx:variable name="FULLPAGENAME"> expanded text </wx:variable>
  • This marking may fail during conversion, in which case the variable will still be expanded correctly but will not be marked, so the expansion will not be recognizable in the XML output;

Examples of the XML format

• A sample of 100 English Wikipedia pages;
• A sample of 100 Dutch Wikipedia pages.