[[TracNav(NuwaNav)]] = Testing Quickstart = == Overview == The automated testing system uses the '''Bitten''' extension to Trac which provides a '''slave''' and '''master''' system : || slave || any machine capable of running !NuWa, with '''nosebit''' installed || || master || dybsvn Trac instance http://dayabay.ihep.ac.cn/tracs/dybsvn || = '''nosebit''' = '''nosebit''' denotes a collection of packages : || [http://www.somethingaboutorange.com/mrl/projects/nose/ nose] || test discovery/running, orchestration with coverage and profilers || || [http://peak.telecommunity.com/DevCenter/setuptools setuptools] || needed to install nose plugins, also provides [http://peak.telecommunity.com/DevCenter/EasyInstall easy_install] tool || || xmlnose || nose plugin to provide test results in the xml format needed by the bitten master || || bitten-slave from [http://bitten.edgewall.org/ bitten] || automated build/test runner, pure python || == Installation of '''nosebit''' into Nuwa python == If '''which nosetests''' draws a blank when you are in your CMT environment then you(or your administrator) will need to rerun [db:Offline_Software_Installation dybinst] with commands like : {{{ ./dybinst trunk checkout ./dybinst trunk external nosebit }}} to get uptodate and install the currently non-default '''nosebit''' external. Which provides the tools : || nosetests || test running entry point || || easy_install || install python packages into NuWa python || || bitten-slave || communicates with bitten-master for instructions on tests (or builds) to be performed || Check your installation by getting into the CMT controlled environment of your default package, and trying these tools: {{{ dyb__ ## see below for details which nosetests ## also check easy_install and bitten-slave }}} The path returned should be beneath the '''external/Python''' directory of your !NuWa installation. Verify that the list of plugins includes '''xml-output''' (used by automated testing system) : {{{ nosetests --plugins | grep xml-output Plugin xml-output }}} = Utility dyb!__* functions = These utility functions are used by the automated build and test system, so to investigate failures it is best to use these functions first to reproduce the failures, then proceed with using '''nosetests''' directly as you zero in on the failing tests. Define the dyb!__* functions in your environment with something like the below called from your $HOME/.bash_profile {{{ . $NUWA_HOME/dybgaudi/DybTest/scripts/dyb__.sh }}} Use tab-completion in your bash shell to see the functions available : {{{ dyb__ }}} Check your bash , if nothing is returned by the below function then your bash does not need the workaround of setting environment variable NUWA_HOME {{{ dyb__old_bash }}} Customize the defaults by overriding the '''dyb!__buildpath''' function to feature your favourite repository path, for example by putting the below into your $HOME/.bash_profile. See the below section on bash functions if they are new to you. {{{ dyb__buildpath(){ echo ${BUILD_PATH:-dybgaudi/trunk/Simulation/GenTools} ; } }}} OR : {{{ dyb__buildpath(){ echo ${BUILD_PATH:-dybgaudi/trunk/DybRelease} ; } }}} * use BUILD_PATH if defined, otherwise the default path provided * CAUTION: you must provide a repository path (with the '''trunk''' ) not a working copy path == dyb!__usage == The '''dyb!__usage''' function provides brief help on the most important functions : {{{ dyb__usage Top level functions provided : dyb__ [siteroot-relative-path] path defaults to dybgaudi/Simulation/GenTools jump into CMT controlled environment and wc directory of a siteroot relative path, the relative path must either end with "cmt" or with a directory that contains the "cmt" directory eg dyb__ lcgcmt/LCG_Interfaces/ROOT/cmt dyb__ lcgcmt/LCG_Interfaces/ROOT dyb__ dybgaudi/DybRelease dyb__update uses dybinst to svn update and rebuild dyb__checkout : ./dybinst trunk checkout dyb__rebuild : ./dybinst -c trunk projects dyb__test [options-passed-to-nosetests] uses dyb__context to setup the environment and directory then invokes nosetests with the options passed }}} = Nose test running = Get into the CMT managed environment of your default build path and run '''nosetests''' with the function : {{{ dyb__test [arguments-are-passed-directly-to-nosetests] }}} The automated tests are run from the package directory, that is the parent directory of the cmt directory. Currently a simple '''nosetests''' invocation is performed, changing this simplest approach to avoid issues with test isolation may be necessary in future. If you are already in the appropriate environment you can run tests with '''nosetests''' directly. == nosetests basic usage == Search for tests using the default test finding approach in the current directory, with command : {{{ nosetests }}} For your tests to be found, * collect them in the '''tests''' folder of your project folder * name the python modules test_*.py * name the test functions test_* See the example of the '''gentools''' tests in * dybsvn:/dybgaudi/trunk/Simulation/GenTools/tests You can also run the tests from specific directories or modules with eg {{{ nosetests tests/test_look.py }}} == nosetests options == Nosetests has a large number of options, see them listed with : {{{ nosetests --help }}} Some of the most useful ones are : || -v / -vv / -vvv || verbosity control, default is very terse just a "." for a successful test || || -s / --no-capture || stdout/stderr for failing tests is usually captured and output at the end of the test run, use this to output immediately || == nose documentation == '''nosetests''' is the command line tool that exposes the functionality of the '''nose''' python package. Access the '''pydoc''' for the '''nose''' package with : {{{ pydoc nose }}} Or see it online at * http://www.somethingaboutorange.com/mrl/projects/nose/ = Automated Test Running = == Bitten recipes == The Bitten master manages XML '''recipes''' for automated building and test running. Recipes are associated with repository paths, to form the '''builds'''. Following svn checkins within the paths, builds are assigned "pending" status. An example of a recipe : * dybsvn:/dybgaudi/trunk/DybTest/recipes/nosetests.xml {{{ #!xml ]> }}} A running Bitten slave polls the master (using HTTP) to see if there are any pending builds that it can perform. If there are it GETs the corresponding recipe and configuration parameters such as the BUILD_PATH from the master and follows the steps of the recipe reporting progress to the master after each step. == Setup/configure a bitten slave == Define some functions for configuration/control of the slave {{{ . $NUWA_HOME/dybgaudi/DybTest/scripts/bitrun.sh }}} {{{ bitrun-usage }}} The slave is configured using two files, one for the credentials/url with which to connect to the master $HOME/.bitrunrc : {{{ # # name used for logfile and tmp directory identification # local name=trialrun # # "builds" url of the master trac instance and credentials with which to connect # local url=http://dayabay.ihep.ac.cn/tracs/dybsvn/builds local user=slave local pass=youknowit # # absolute path to slave config file # local cfg=/disk/d3/dayabay/local/dyb/trunk_dbg/NuWa-trunk/dybgaudi/DybTest/recipes/grid1.cfg }}} And a second defining the characteristics of the !NuWa installation * dybsvn:/dybgaudi/trunk/DybTest/recipes/grid1.cfg {{{ # # configuration for a bitten slave # # the master deals in repository paths hence have to strip # the "trunk" to get to working copy paths # [nuwa] home = /disk/d3/dayabay/local/dyb/trunk_dbg/NuWa-trunk version = trunk script = dybgaudi/DybTest/scripts/dyb__.sh }}} The quantites defined are accessible within the recipe context as eg: {{{ ${nuwa.home} }}} == slave running == The slave will poll the master looking for pending builds that can be performed by the slave, based on match criteria configured in the master. When a pending build is found that matches the slave, the steps of the build are done and reported back to the master. Test outcomes in XML are reported to the the master, which parses them and places into the Trac database ready for presentation. To start the slave: {{{ bitrun-start [options-passed-to-slave] }}} This function invokes the '''bitten-slave''' command which does the setup of working directories and configuration file access. Options to the function are passed to the command Useful options for initial debugging : || -s, --single || exit after completing a single build || || -n, --dry-run || do not report results back to master || || -i SECONDS, --interval=SECONDS || time to wait between requesting builds, poll interval (default 300 s) || = Bash Functions Primer = Define the function : {{{ demo-func(){ echo $FUNCNAME hello [$BASH_SOURCE] } }}} Invoke it : {{{ demo-func demo-func hello [] }}} Show the definition : {{{ type demo-func demo-func is a function demo-func () { echo $FUNCNAME hello [$BASH_SOURCE] } }}}