Testing Quickstart

Overview

The automated testing system uses the Bitten extension to Trac which provides a slave and master system :

slave	any machine capable of running NuWa, with nosebit installed
master	dybsvn Trac instance http://dayabay.ihep.ac.cn/tracs/dybsvn

nosebit

nosebit denotes a collection of packages :

nose	test discovery/running, orchestration with coverage and profilers
setuptools	needed to install nose plugins, also provides easy_install tool
xmlnose	nose plugin to provide test results in the xml format needed by the bitten master
bitten-slave from bitten	automated build/test runner, pure python

Installation of nosebit into Nuwa python

Since r4360, nosebit is included with the dybinst externals, so the below installation should have been done automatically by dybinst, (if you find otherwise check ticket #19 ).

If which nosetests draws a blank when you are in your CMT environment then you(or your administrator) will need to rerun dybinst with commands like :

./dybinst trunk checkout
./dybinst trunk external nosebit

to get uptodate and install the currently non-default nosebit external. Which provides the tools :

nosetests	test running entry point
easy_install	install python packages into NuWa python
bitten-slave	communicates with bitten-master for instructions on tests (or builds) to be performed

Check your installation by getting into the CMT controlled environment of your default package, and trying these tools:

dyb__   ## see below for details   
which nosetests    ## also check easy_install  and bitten-slave

The path returned should be beneath the external/Python directory of your NuWa installation. Verify that the list of plugins includes xml-output (used by automated testing system) :

nosetests --plugins  | grep xml-output
Plugin xml-output

Utility dyb__* functions

These utility functions are used by the automated build and test system, so to investigate failures it is best to use these functions first to reproduce the failures, then proceed with using nosetests directly as you zero in on the failing tests.

Define the dyb__* functions in your environment with something like the below called from your $HOME/.bash_profile

. $NUWA_HOME/dybgaudi/DybTest/scripts/dyb__.sh

Use tab-completion in your bash shell to see the functions available :

dyb__<tab>   

Check your bash , if nothing is returned by the below function then your bash does not need the workaround of setting environment variable NUWA_HOME

 dyb__old_bash

Customize the defaults by overriding the dyb__buildpath function to feature your favourite repository path, for example by putting the below into your $HOME/.bash_profile. See the below section on bash functions if they are new to you.

dyb__buildpath(){    echo ${BUILD_PATH:-dybgaudi/trunk/Simulation/GenTools} ; }    

OR :

dyb__buildpath(){    echo ${BUILD_PATH:-dybgaudi/trunk/DybRelease} ; } 

• use BUILD_PATH if defined, otherwise the default path provided
• CAUTION: you must provide a repository path (with the trunk ) not a working copy path

dyb__usage

The dyb__usage function provides brief help on the most important functions :

dyb__usage

     Top level functions provided  :

           dyb__ [siteroot-relative-path]      
                   path defaults to dybgaudi/Simulation/GenTools  
                   jump into CMT controlled environment
                   and wc directory of a siteroot relative path, 
                   the relative path must either end with "cmt" or with 
                   a directory that contains the "cmt" directory 
       
                     eg 
                         dyb__ lcgcmt/LCG_Interfaces/ROOT/cmt  
                         dyb__ lcgcmt/LCG_Interfaces/ROOT
                         dyb__ dybgaudi/DybRelease
     
           dyb__update   
                        uses dybinst to svn update and rebuild 
                          dyb__checkout  :   ./dybinst    trunk checkout
                          dyb__rebuild   :   ./dybinst -c trunk projects  

           dyb__test [options-passed-to-nosetests]    

                        uses dyb__context to setup the environment and directory 
                        then invokes nosetests with the options passed 

Nose test running

Get into the CMT managed environment of your default build path and run nosetests with the function :

dyb__test  [arguments-are-passed-directly-to-nosetests]

The automated tests are run from the package directory, that is the parent directory of the cmt directory. Currently a simple nosetests invocation is performed, changing this simplest approach to avoid issues with test isolation may be necessary in future.

If you are already in the appropriate environment you can run tests with nosetests directly.

nosetests basic usage

Search for tests using the default test finding approach in the current directory, with command :

nosetests     

For your tests to be found,

• collect them in the tests folder of your project folder
• name the python modules test_*.py
• name the test functions test_*

See the example of the gentools tests in

• dybsvn:/dybgaudi/trunk/Simulation/GenTools/tests

You can also run the tests from specific directories or modules with eg

nosetests tests/test_look.py

nosetests options

Nosetests has a large number of options, see them listed with :

nosetests --help    

Some of the most useful ones are :

-v / -vv / -vvv	verbosity control, default is very terse just a "." for a successful test
-s / --no-capture	stdout/stderr for failing tests is usually captured and output at the end of the test run, use this to output immediately

nose documentation

nosetests is the command line tool that exposes the functionality of the nose python package. Access the pydoc for the nose package with :

pydoc nose 

Or see it online at

• http://www.somethingaboutorange.com/mrl/projects/nose/

Automated Test Running

Bitten recipes

The Bitten master manages XML recipes for automated building and test running. Recipes are associated with repository paths, to form the builds. Following svn checkins within the paths, builds are assigned "pending" status.

An example of a recipe :

• dybsvn:/dybgaudi/trunk/DybTest/recipes/nosetests.xml

<!DOCTYPE build [
  <!ENTITY  slav " export BUILD_PATH=${path} ; export BUILD_CONFIG=${config} ; export BUILD_REVISION=${revision} ; export BUILD_NUMBER=${build} ; " > 
  <!ENTITY  nuwa " export NUWA_HOME=${nuwa.home} ; export NUWA_VERSION=${nuwa.version} ; export NUWA_SCRIPT=${nuwa.script} ;  " >
  <!ENTITY  scpt  " pwd ; iwd=$PWD ; . $NUWA_HOME/$NUWA_SCRIPT ; "  >
  <!ENTITY  env  " &slav;  &nuwa;  &scpt; " > 
]>


<build
    xmlns:python="http://bitten.cmlenz.net/tools/python"
    xmlns:svn="http://bitten.cmlenz.net/tools/svn"
    xmlns:sh="http://bitten.cmlenz.net/tools/sh"
  >

  <!--
    the NUWA_HOME is needed to work with older bash that does not have BASH_SOURCE
  -->

<step id="update" description="dybinst checkout and rebuild " onerror="continue" >
      <sh:exec executable="bash" output="hello.out"  
           args=" -c &quot; &env;  dyb__hello  ; echo it returned $?  &quot; " />
      <sh:exec executable="bash" output="update.out"  
           args=" -c &quot; &env;  dyb__update ; echo it returned $?  &quot; " />
</step>

<step id="txttest" description="Run tests with txt output " onerror="continue"  >
     <sh:exec executable="bash"   output="txttest.out" 
           args=" -c  &quot; &env; dyb__test &quot;  "  />
</step>

<step id="xmltest" description="Run tests with xml output " onerror="continue" >
     <sh:exec executable="bash"  output="xmltest.out"
           args=" -c &quot;  &env; dyb__test --with-xml-output --xml-outfile=$iwd/nosetests.xml   &quot;  "  />
     <python:unittest file="nosetests.xml" />
</step>

</build>

A running Bitten slave polls the master (using HTTP) to see if there are any pending builds that it can perform. If there are it GETs the corresponding recipe and configuration parameters such as the BUILD_PATH from the master and follows the steps of the recipe reporting progress to the master after each step.

Setup/configure a bitten slave

Define some functions for configuration/control of the slave

. $NUWA_HOME/dybgaudi/DybTest/scripts/bitrun.sh

bitrun-usage

The slave is configured using two files, one for the credentials/url with which to connect to the master $HOME/.bitrunrc :

#
# name used for logfile and tmp directory identification  
#
local name=trialrun   
       
#
#  "builds" url of the master trac instance and credentials with which to connect
#    
local url=http://dayabay.ihep.ac.cn/tracs/dybsvn/builds   
local user=slave
local pass=youknowit

#
# absolute path to slave config file
#
local cfg=/disk/d3/dayabay/local/dyb/trunk_dbg/NuWa-trunk/dybgaudi/DybTest/recipes/grid1.cfg   

And a second defining the characteristics of the NuWa installation

• dybsvn:/dybgaudi/trunk/DybTest/recipes/grid1.cfg

#
#  configuration for a bitten slave 
#
#  the master deals in repository paths hence have to strip 
#  the "trunk" to get to working copy paths
# 
[nuwa]
home = /disk/d3/dayabay/local/dyb/trunk_dbg/NuWa-trunk
version = trunk
script = dybgaudi/DybTest/scripts/dyb__.sh

The quantites defined are accessible within the recipe context as eg:

${nuwa.home}  

slave running

The slave will poll the master looking for pending builds that can be performed by the slave, based on match criteria configured in the master. When a pending build is found that matches the slave, the steps of the build are done and reported back to the master. Test outcomes in XML are reported to the the master, which parses them and places into the Trac database ready for presentation.

To start the slave:

bitrun-start [options-passed-to-slave]

This function invokes the bitten-slave command which does the setup of working directories and configuration file access. Options to the function are passed to the command

Useful options for initial debugging :

-s, --single	exit after completing a single build
-n, --dry-run	do not report results back to master
-i SECONDS, --interval=SECONDS	time to wait between requesting builds, poll interval (default 300 s)

Bash Functions Primer

Define the function :

demo-func(){
  echo $FUNCNAME hello [$BASH_SOURCE]
}

Invoke it :

demo-func
demo-func hello []

Show the definition :

type demo-func
demo-func is a function
demo-func () 
{ 
    echo $FUNCNAME hello [$BASH_SOURCE]
}