Micro-benchmarks

What is a micro-benchmark?

A micro-benchmark is an experimental tool that studies a given aspect (e.g., performance, resource consumption) of XML data management and querying, called the target of the micro-benchmark. The tool includes a parametric measure and guidelines, explaining which data and/or query parameters may impact the target, and suggesting value ranges for these parameters.

The methodology of applying a micro-benchmark consists in varying the parameters one at a time, while keeping the other parameters constant. We call each of these experiments a micro-benchmark running scenario. An running scenario will typically yield a family of curves where the varying parameter values are on the X axis, and the measure result on the Y axis. The number of running scenarios in a micro-benchmark depends on how many parameters vary in the documents and queries considered and how many different values they take. Each micro-benchmark lists several of these that the benchmark authors find particularly interesting.

Representation

The micro-benchmarks are represented as XML documents which are valid with respect to the DTD mb-structure.dtd.

Usage

Everybody is welcome to use the micro-benchmarks published on MemBeR. We recommend to follow the micro-benchmark's specifications with respect to the parameter values and with respect to the running scenarios. Still, we welcome well-justified changes and/or extensions to existing micro-benchmarks.

Please, refer to MemBeR and to the unique page of the micro-benchmark that you use when publishing (on the internet or in a paper publication) its results. Submitting your results to MemBeR will make them much more visible!

If none of the micro-benchmarks on MemBeR fits your target and you are forced to design your own micro-benchmark, please, submit it to MemBeR. This will get your idea (and experimental results) known, and may lead to your micro-benchmark being accepted as a standard yardstick to measure future systems against!