Introduction to tests

Table of Contents

• Overview
  • Name
  • Category
  • Environments
  • Variations
• Building
  • Creating a new test
  • Minimal main.c
  • Building
• Installing
• Running

Overview

A test has several important attributes, and are often referred to in the form:

test-$ENVIRONMENT-$NAME[~$VARIATION]

This identifies a single test instance, with the purpose of executing and providing a single result (See Running).

Name

A tests name is a unique identifier. The directory tests/$NAME/ contains all files related to a specific test. A test has one or more test instances.

Category

A tests category is a broad delineation of its purpose. All defined categories are:

ALL_CATEGORIES     := special functional xsa utility in-development

• xsa are dedicated tests to confirm the presence or absence of a specific security vulnerability.
• functional are tests covering a specific area of functionality.
• utility are not really tests, but exist to dump specific information about the VM environment.
• special covers the example and environment sanity checks.
• in-development covers tests which aren't yet complete, and are not ready to be run automatically yet.

Environments

A test is built for one or more environments. The environment encodes:

• The Xen VM ABI in use (PV or HVM).
• The compilation width (32bit or 64bit).
• The primary paging mode (none, PSE, PAE).

All available environments are:

ALL_ENVIRONMENTS   := pv64 pv32pae hvm64 hvm32pae hvm32pse hvm32

Some more specific collections for environments are also available:

PV_ENVIRONMENTS    := $(filter pv%,$(ALL_ENVIRONMENTS))
HVM_ENVIRONMENTS   := $(filter hvm%,$(ALL_ENVIRONMENTS))
32BIT_ENVIRONMENTS := $(filter pv32% hvm32%,$(ALL_ENVIRONMENTS))
64BIT_ENVIRONMENTS := $(filter pv64% hvm64%,$(ALL_ENVIRONMENTS))

An individual test, compiled for more than one environment, will end up with a individual microkernel binary for each specified environment.

Variations

Variations are optional. Some tests may only want to run the binary once, but some tests want to run the same binary multiple times in different configurations, reporting separate results.

The variation suffix (if present) is arbitrary and defined by the test itself.

Building

Creating a new test

A helper script is provided to create the initial boilerplate. It prompts for some information, and writes out some default files:

$ ./make-new-test.sh
Test name: example
Category [utility]: special
Environments [hvm32]: $(ALL_ENVIRONMENTS)
Extra xl.cfg? [y/N]: n
Writing default tests/example/Makefile
Writing default tests/example/main.c
$

Minimal main.c

The minimum for main.c is to include the framework:

#include <xtf.h>

Choose a suitable title:

const char test_title[] = "Hello World";

Implement test_main():

void test_main(void)
{

And make a call to xtf_success() somewhere before test_main() returns:

    xtf_success(NULL);
}

Building

Building this will generate a number of files.

$ make -j4
... <snip>
$ ls tests/example/
info.json
main.c
main-hvm32.d
main-hvm32.o
main-hvm32pae.d
main-hvm32pae.o
main-hvm32pse.d
main-hvm32pse.o
main-hvm64.d
main-hvm64.o
main-pv32pae.d
main-pv32pae.o
main-pv64.d
main-pv64.o
Makefile
test-hvm32-example
test-hvm32-example.cfg
test-hvm32pae-example
test-hvm32pae-example.cfg
test-hvm32pse-example
test-hvm32pse-example.cfg
test-hvm64-example
test-hvm64-example.cfg
test-pv32pae-example
test-pv32pae-example.cfg
test-pv64-example
test-pv64-example.cfg

The .o and .d files are build artefacts, leading to the eventual test-$ENV-$NAME microkernel. Individual xl.cfg files are generated for each microkernel. info.json contains metadata about the test.

Installing

If XTF in being built in dom0, all paths should be set up to run correctly.

# xl create tests/example/test-hvm64-example.cfg

If XTF is built elsewhere, it should be installed:

$ make install xtfdir=/path DESTDIR=/path

with paths appropriate for the system under test.

Running

A test will by default write synchronously to all available consoles:

# ./xtf-runner test-hvm64-example
Executing 'xl create -p tests/example/test-hvm64-example.cfg'
Parsing config from tests/example/test-hvm64-example.cfg
Executing 'xl console test-hvm64-example'
Executing 'xl unpause test-hvm64-example'
--- Xen Test Framework ---
Environment: HVM 64bit (Long mode 4 levels)
Hello World
Test result: SUCCESS

Combined test results:
test-hvm64-example                       SUCCESS

The result is covered by the report.h API. In short, the options are:

• Success
  • The test passed.
• Skip
  • The test cannot be completed for configuration reasons.
• Error
  • An error in the environment or test code itself.
• Failure
  • An issue with the functional area under test.