#
# Copyright (C) 2009 Red Hat, Inc.
# Copyright (C) 2009 Daniel P. Berrange
#
# This program is free software; You can redistribute it and/or modify
# it under the GNU General Public License as published by the Free
# Software Foundation; either version 2, or (at your option) any
# later version
#
# The file "LICENSE" distributed along with this file provides full
# details of the terms and conditions
#

=pod

=head1 libvirt TCK: Technology Compatability Kit

The libvirt TCK provides a framework for performing testing
of the integration between libvirt drivers, the underlying virt
hypervisor technology, related operating system services and system
configuration. The idea (and name) is motivated by the Java TCK

In particular the libvirt TCK is intended to address the following
scenarios

=over 4

=item *

Validate that a new libvirt driver is in compliance
with the (possibly undocumented!) driver API semantics

=item *

Validate that an update to an existing driver does not
change the API semantics in a non-compliant manner

=item *

Validate that a new hypervisor release is still providing
compatability with the corresponding libvirt driver usage

=item *
Validate that an OS distro deployment consisting of a
hypervisor and libvirt release is configured correctly

=back

Thus the libvirt TCK will allow developers, administrators and users
to determine the level of compatability of their platform, and
evaluate whether it will meet their needs, and get awareness of any
regressions that may have occurred since a previous test run

In relation to other libvirt testing, the split of responsibility
will be

=over 4

=item libvirt testsuite (aka $CHECKOUT/tests)

=over 4

=item *

unit testing of specific internal APIs

=item *

functional testing of the libvirtd using the 'test'+'remote' drivers

=item *

functional testing of the virsh command using the 'test' driver

=back

=item libvirt TCK

=over 4

=item *

functional/integration testing of the 'live' drivers

=back

=back


=head2 Framework requirements

The libvirt TCK is built using Perl in order to take advantage of
the advanced, but yet very simple, testing frameworks available
with Perl. Thus the libvirt interactions will all be done via the
libvirt Perl bindings, Sys::Virt (or perl-Sys-Virt RPMs)

The framework is thus built on the following Perl modules

=over 4

=item Test::More

simple framework for writing individual tests

=item TAP::Harness

simple framework for running sets of tests

=item Sys::Virt

binding for libvirt API

=item XML::Writer

module for generating XML documents

=item XML::Twig

module for parsing XML documents & XPath

=item Config::Record

module for parsing simple configuration files

=back

There are a handful of other modules these depend on, but these
are the most important 'top level' modules in use.

These are all currently available within Fedora 11, and later

These modules are all well tested, actively maintained parts of
Perl / CPAN, so easily available for every other operating system
in existence.

=head2 Overview of framework structure

For following discussions, it may be convenient to refer to the source
code of the framework. This is available from

   git clone  git://libvirt.org/libvirt-tck.git

First there are a couple of Perl modules to provide assistance when
dealing with libvirt / writing tests

=over 4

=item Sys::Virt::TCK  in lib/Sys/Virt/TCK.pm

The core module for connecting to libvirt, creating a clean
environment (ie blowing away all existing domains), and generating
simple XML configs for guests

=item Sys::Virt::TCK::DomainBuilder in lib/Sys/Virt/TCK/DomainBuilder.pm

A helper for constructing XML configs for guest domains

=item Sys::Virt::TCK::Capabilities  in lib/Sys/Virt/TCK/Capabilities.pm

A helper for parsing the libvirt capabilities XML

=item Sys::Virt::TCK::TAP::XMLFormatter in lib/Sys//Virt/TCK/TAP/XMLFormatter.pm

A plugin for TAP::Harness that is able to record all test results
in a structured XML document.

=back

As mentioned before, the framework is built about Test:More and the
TAP::Harness modules from Perl. This already comes with a simple
command called 'prove' for running tests & reporting on results. It
has a rather baffling array of options, so to make it simpler to run
the libvirt TCK, there is a small program

=over 4

=item bin/libvirt-tck  (installed to /usr/bin/libvirt-tck)

Given no arguments, this will connect using the default hypervisor
URI and a previously obtained kernel+initrd and run all the tests
currently available for the libvirt TCK, and report on failures.
It comes with a number of options to alter the output format or
choose different configurations. 'man 1 libvirt-tck' will produce
details

=back


The actual tests themselves are simply short Perl scripts using the
Test::More, Sys::Virt and Sys::Virt::TCK modules. Each test decides
on what aspect it wants to test, and then implements that logic and
tests results.

As a demonstration, there are 4 initial scripts

=over 4

=item scripts/domain/050-transient-lifecycle.t

Creates a guest from XML, destroys it, and then verifies that
it has actually gone away.

=item scripts/domain/060-persistent-lifecycle.t

Defines a guest config XML, starts it, destroys it, verifies
that the config still exists, and then undefines the config
and verifies that it has actually gone.

=item scripts/domain/070-transient-to-persistent.t

Creates a guest from XML, then defines a persistent config for
it, destroys the running guest, and then verifies the config
is still present.

=item scripts/domain/080-unique-identifiers.t

Defines a guest, and then tries to define / create more guests
with clashing name or UUID, and verifies that suitable errors
are raised by libvirt.

=back


At the time this was written, even these 4 simple proof of concept
scripts highlighted a couple of problems in the libvirt code

=over 4

=item *

The QEMU driver 'define domain' method didn't check for name
or UUID uniqueness correctly (well, at all)

=item *

After starting an inactive domain, the remote driver did not
update the 'ID' field in the virDomainPtr

=item *

After destroying an active domain, the remote driver did not
update the 'ID' field in the virDomainPtr

=item *

When defining a persistent config for an already running domain
the Xen XM driver blew away the current 'ID' field for the
running domain, replacing it with -1.

=item *

QEMU refuseed to boot kernel+initrd unless at least one disk
image is provided

=back


=head2 Output information

The libvirt-tck tool outputs results in a number of formats. The
default format is a simple plain test summary listing each test
case, and the pass/fail state, and details of each check failure

A more verbose text format otputs the full Perl TAP (Test Anything
Protocol) format results as described in 'man 3 TAP' or
'man 3 Test::Harness::TAP'.

For producing pretty web pages, it is possible to request an HTML
output format.

Ultimately though it will be desirable to do automated analysis,
and comparison of results across releases, OS, drivers, etc. To
assist in creating tools todo this, an XML format is also provided


=head2 Running the test suite


For those feeling brave it is possible to try out the current test suite.
It has only really been tested with the QEMU driver, and a little with
the Xen driver. Most other drivers will not yet work/pass with it.

The only thing that needs to be changed before running the test
script is to set a URI in the config file /etc/libvirt-tck/default.cfg

The test suite will automatically download kernels, initrds and
root filesystems needed to run the tests. The configuration file
can be used to override this though.

Now it should be possible to run

  # libvirt-tck

It may well show some failures. To get more information add the -v
flag

  # libvirt-tck -v


=head2 What's still todo

The code as it stands is the bare minimum to get a proof of concept working
for testing of domain APIs for Xen and QEMU drivers. The test suite though
is intended to be independant of any driver, and also allow for coverage of
all the libvirt APIs.

Of the top of my head, some important things that need doing

=over 4

=item libvirt-tck-prepare

For now it was sufficient to just grab the kernel+initrd from Fedora 10
pxeboot location, but not every kind of virtualization can boot off a
kernel + initrd. Older Xen HVM cannot do this. VMWare cannot do this.
OpenVZ / LXC have no concept of separate kernels, etc.

The C<libvirt-tck-prepare> command would automate the process of setting
up some pre-requisite pieces. Specifically it would

=over 4

=item *

Build a bootable ISO image using the kernel+initrd

=item *

Create a virtual root filesystem, populated with
busybox commands (for LXC/OpenVZ)

=back

=item intelligent XML generation

Be more intelligent about building domain XML configs. In
particular look at the capabilities XML to decide whether
to create a config that boots off kernel+initrd vs ISO
vs a virtual root filesystem for containers

=item More helper XML helper modules

Add helpers for building network, storage and interface XML
configs

=item Broader host configuration

Expand on configuration to allow admin to indicate some
resources that can be safely used during tests

=over 4

=item *
A friendly NFS server & some of its exports

=item *
A spare disk (eg /dev/sdb) that can be played with
and trashed

=item *
A spare network interface (or two) that can be played
with

=item *
A spare PCI device that can be detached from host

=item *
A spare USB device that can be detached from host

=item *
A path with X GB of free space to play with for
storage pool usage

=item *
A friendly iSCSI server & some of its targets

=back

If any of those resources were not available, the test cases
needing them would simply be skipped. This is very easy to
cope with in the Test::More framework

      plan skip_all "no iscsi server available" unless
            $tck->has_config("iscsi-server");

=item Fix libvirt bugs

Fix up all the horribly broken areas of libvirt that this
uncovers. This will entail deciding that the semantics for
various edge cases are with each API. Deciding what errors
codes need to be formally defined for each API. Figuring
out how to implement/fix the neccessary semantics in the
drivers.

=item Failure recording

A way to record specific test failures as 'known broken' for
a particular driver  / platform combination.

=item Problem reporting

A tool to take an XML report from the TCK, mask out all tests
that are 'known broken', and then report on the remaining
problems which need fixing.

=item Quality analysis

A tool to compare two XML reports and show interesting
differences in functionality, and / or bugs

=back


=head2 How this might be used

The original motivating goal for this is obviously to improve
the overall quality control of libvirt and things it interacts
with. There are a number of scenarios in which I see this being
used:

=over 4

=item *

Fedora rawhide updates to a  release of QEMU. Run the TCK to
make sure it didn't break anything (new) in libvirt

=item *

Declaring feature freeze for a libvirt release
Run the TCK to determine what state of each
driver is. Decide what problems should be
release blockers. It is expected that some
drivers may have long term failures, due to
features that will not be implemented

=item *

Released new libvirt. Provide online 'reference' reports for the
new libvirt release against various platforms. Allows OS distributors
to determine whether their changes cause regressions, or if it
is a known-broken item.

=item *

App developer looking to understand feature support. Look at the
TCK reports for the driver and decide if it implements enough of
the functionality to be worth supporting.

=back

The key factor I think is that it is unreasonable to expect that
the TCK will complete without failures for every libvirt driver,
let alone every OS. Some features will simply be impossible to
implement for certain platforms. Thus the key is in tracking
what areas are known to be broken, to make it possible to identify
regressions in areas that are expected to work. The known broken
areas may also provide motivation for new feature development in
associated tools.