Changes between Initial Version and Version 1 of Testing_Quickstart

Timestamp:

08/22/08 11:08:37 (17 years ago)

Author:

blyth (IP: 140.112.102.77)

Comment:

--

Testing_Quickstart

@@ +1 @@
+[[TracNav(NuwaNav)]]
+= Testing Quickstart =
+
+== Overview ==
+
+The automated testing system uses the '''Bitten''' extension to Trac which provides a '''slave''' and '''master''' system : 
+   ||  slave   ||  any machine capable of running !NuWa, with '''nosebit''' installed || 
+   || master ||  dybsvn Trac instance http://dayabay.ihep.ac.cn/tracs/dybsvn  ||
+
+= '''nosebit''' =
+
+'''nosebit''' denotes a collection of  packages :
+
+   ||  [http://www.somethingaboutorange.com/mrl/projects/nose/ nose] ||  test discovery/running, orchestration with coverage and profilers   ||
+   ||  [http://peak.telecommunity.com/DevCenter/setuptools  setuptools] || needed to install nose plugins, also provides [http://peak.telecommunity.com/DevCenter/EasyInstall easy_install] tool  ||
+   ||  xmlnose ||    nose plugin to provide test results in the xml format needed by the bitten master ||
+   ||  bitten-slave  from [http://bitten.edgewall.org/ bitten]  ||   automated build/test runner,  pure python  || 
+
+== Installation of '''nosebit''' into Nuwa python ==
+
+If '''which nosetests''' draws a blank when you are in your CMT environment then you(or your administrator) will 
+need to rerun [db:Offline_Software_Installation  dybinst] with commands like :
+{{{
+./dybinst trunk checkout
+./dybinst trunk external nosebit
+}}}
+to get uptodate and install the currently non-default '''nosebit''' external. Which provides the tools :  
+   || nosetests ||   test running entry point ||
+   || easy_install ||   install python packages into NuWa python ||
+   || bitten-slave ||  communicates with bitten-master for instructions on tests (or builds) to be performed ||  
+
+Check your installation by getting into the CMT controlled environment of your default package, and trying these
+tools:
+{{{
+dyb__   ## see below for details   
+which nosetests    ## also check easy_install  and bitten-slave
+}}}
+
+The path returned should be beneath the '''external/Python''' directory of your !NuWa installation.
+Verify that the list of plugins includes '''xml-output''' (used by automated testing system) :
+{{{
+nosetests --plugins  | grep xml-output
+Plugin xml-output
+}}}
+
+= Utility dyb!__* functions =
+
+These utility functions are used by the automated build and test system, so to investigate failures 
+it is best to use these functions first to reproduce the failures, then proceed with using
+'''nosetests''' directly as you zero in on the failing tests.
+
+Define the dyb!__* functions in your environment with something like
+the below called from your $HOME/.bash_profile  
+{{{
+. $NUWA_HOME/dybgaudi/DybTest/scripts/dyb__.sh
+}}}
+
+Use tab-completion in your bash shell to see the functions available : 
+{{{
+dyb__<tab>   
+}}}
+
+Check your bash , if nothing is returned by the below function then your bash does not need the workaround of setting environment variable NUWA_HOME
+{{{
+ dyb__old_bash
+}}}
+ 
+Customize the defaults by overriding  the '''dyb!__buildpath''' function to feature your favourite repository path, for example by putting the 
+below into your $HOME/.bash_profile.  See the below section on bash functions if they are new to you.
+{{{
+dyb__buildpath(){    echo ${BUILD_PATH:-dybgaudi/trunk/Simulation/GenTools} ; }    
+}}}
+OR :
+{{{
+dyb__buildpath(){    echo ${BUILD_PATH:-dybgaudi/trunk/DybRelease} ; } 
+}}}
+
+   * use BUILD_PATH if defined, otherwise the default path provided 
+   * CAUTION: you must provide a repository path (with the '''trunk''' ) not a working copy path
+
+== dyb!__usage ==
+
+The '''dyb!__usage''' function provides brief help on the most important functions :
+{{{
+dyb__usage
+
+     Top level functions provided  :
+
+           dyb__ [siteroot-relative-path]      
+                   path defaults to dybgaudi/Simulation/GenTools  
+                   jump into CMT controlled environment
+                   and wc directory of a siteroot relative path, 
+                   the relative path must either end with "cmt" or with 
+                   a directory that contains the "cmt" directory 
+       
+                     eg 
+                         dyb__ lcgcmt/LCG_Interfaces/ROOT/cmt  
+                         dyb__ lcgcmt/LCG_Interfaces/ROOT
+                         dyb__ dybgaudi/DybRelease
+     
+           dyb__update   
+                        uses dybinst to svn update and rebuild 
+                          dyb__checkout  :   ./dybinst    trunk checkout
+                          dyb__rebuild   :   ./dybinst -c trunk projects  
+
+           dyb__test [options-passed-to-nosetests]    
+
+                        uses dyb__context to setup the environment and directory 
+                        then invokes nosetests with the options passed 
+
+
+}}}
+
+
+= Nose test running =
+
+Get into the CMT managed environment of your default build path  and run '''nosetests''' with the function : 
+{{{
+dyb__test  [arguments-are-passed-directly-to-nosetests]
+}}}
+The automated tests are run from the package directory, that is the parent directory of the cmt directory.
+Currently a simple '''nosetests''' invocation is performed, changing this simplest approach to avoid issues with test isolation 
+may be necessary in future.  
+
+If you are already in the appropriate environment you can run tests with 
+'''nosetests''' directly.  
+
+== nosetests basic usage ==
+
+Search for tests using the default test finding approach in the current directory, with command :
+{{{
+nosetests     
+}}}
+For your tests to be found, 
+    * collect them in the '''tests''' folder of your project folder 
+    * name the python modules test_*.py
+    * name the test functions test_* 
+
+See the example of the '''gentools''' tests in 
+   *  dybsvn:/dybgaudi/trunk/Simulation/GenTools/tests
+
+You can also run the tests from specific directories or modules with eg 
+{{{
+nosetests tests/test_look.py
+}}}
+
+== nosetests options ==
+
+Nosetests has a large number of options, see them listed with :
+{{{
+nosetests --help    
+}}}
+
+Some of the most useful ones are :
+  ||  -v  / -vv / -vvv       ||    verbosity control, default is very terse just a "." for a successful test   ||
+  || -s / --no-capture  ||   stdout/stderr for failing tests is usually captured and output at the end of the test run, use this to output immediately ||
+
+== nose documentation ==
+
+'''nosetests''' is the command line tool that exposes the functionality of the '''nose'''
+python package.  Access the '''pydoc''' for the '''nose''' package  with :
+{{{
+pydoc nose 
+}}}
+
+Or see it online at 
+   * http://www.somethingaboutorange.com/mrl/projects/nose/
+
+
+
+= Automated Test Running =
+
+== Bitten recipes ==
+
+The Bitten master manages  XML '''recipes''' for automated building and test running. 
+Recipes are associated with repository paths, to form the '''builds'''.
+Following svn checkins within the paths, builds are assigned "pending" status.
+
+An example of a recipe :
+  * dybsvn:/dybgaudi/trunk/DybTest/recipes/nosetests.xml
+
+{{{
+#!xml
+
+<!DOCTYPE build [
+  <!ENTITY  slav " export BUILD_PATH=${path} ; export BUILD_CONFIG=${config} ; export BUILD_REVISION=${revision} ; export BUILD_NUMBER=${build} ; " > 
+  <!ENTITY  nuwa " export NUWA_HOME=${nuwa.home} ; export NUWA_VERSION=${nuwa.version} ; export NUWA_SCRIPT=${nuwa.script} ;  " >
+  <!ENTITY  scpt  " pwd ; iwd=$PWD ; . $NUWA_HOME/$NUWA_SCRIPT ; "  >
+  <!ENTITY  env  " &slav;  &nuwa;  &scpt; " > 
+]>
+
+
+<build
+    xmlns:python="http://bitten.cmlenz.net/tools/python"
+    xmlns:svn="http://bitten.cmlenz.net/tools/svn"
+    xmlns:sh="http://bitten.cmlenz.net/tools/sh"
+  >
+
+  <!--
+    the NUWA_HOME is needed to work with older bash that does not have BASH_SOURCE
+  -->
+
+<step id="update" description="dybinst checkout and rebuild " onerror="continue" >
+      <sh:exec executable="bash" output="hello.out"  
+           args=" -c &quot; &env;  dyb__hello  ; echo it returned $?  &quot; " />
+      <sh:exec executable="bash" output="update.out"  
+           args=" -c &quot; &env;  dyb__update ; echo it returned $?  &quot; " />
+</step>
+
+<step id="txttest" description="Run tests with txt output " onerror="continue"  >
+     <sh:exec executable="bash"   output="txttest.out" 
+           args=" -c  &quot; &env; dyb__test &quot;  "  />
+</step>
+
+<step id="xmltest" description="Run tests with xml output " onerror="continue" >
+     <sh:exec executable="bash"  output="xmltest.out"
+           args=" -c &quot;  &env; dyb__test --with-xml-output --xml-outfile=$iwd/nosetests.xml   &quot;  "  />
+     <python:unittest file="nosetests.xml" />
+</step>
+
+</build>
+
+}}}
+
+A running Bitten slave polls the master (using HTTP) to see if there are any pending builds that it can perform.
+If there are it GETs the corresponding recipe and configuration parameters such as the BUILD_PATH from the master and
+follows the  steps of the recipe reporting progress to the master after each step.  
+
+== Setup/configure a bitten slave ==
+
+Define some functions for configuration/control of the slave 
+{{{
+. $NUWA_HOME/dybgaudi/DybTest/scripts/bitrun.sh
+}}}
+
+{{{
+bitrun-usage
+}}}
+
+The slave is configured using two files, one for the credentials/url  with which  to connect to 
+the master $HOME/.bitrunrc :
+{{{
+#
+# name used for logfile and tmp directory identification  
+#
+local name=trialrun   
+       
+#
+#  "builds" url of the master trac instance and credentials with which to connect
+#    
+local url=http://dayabay.ihep.ac.cn/tracs/dybsvn/builds   
+local user=slave
+local pass=youknowit
+
+#
+# absolute path to slave config file
+#
+local cfg=/disk/d3/dayabay/local/dyb/trunk_dbg/NuWa-trunk/dybgaudi/DybTest/recipes/grid1.cfg   
+
+}}}
+
+And a second defining the characteristics of the !NuWa installation 
+   * dybsvn:/dybgaudi/trunk/DybTest/recipes/grid1.cfg   
+
+{{{
+#
+#  configuration for a bitten slave 
+#
+#  the master deals in repository paths hence have to strip 
+#  the "trunk" to get to working copy paths
+# 
+[nuwa]
+home = /disk/d3/dayabay/local/dyb/trunk_dbg/NuWa-trunk
+version = trunk
+script = dybgaudi/DybTest/scripts/dyb__.sh
+}}}
+
+The quantites defined are accessible within the recipe context as eg:
+{{{
+${nuwa.home}  
+}}}
+ 
+== slave running ==
+
+The slave will poll the master looking for pending builds that can be performed by the slave, based on 
+match criteria configured in the master.
+When a pending build is found that matches  the slave, the steps of  the build are done and reported back to the master.
+Test outcomes in XML are reported to the the master, which parses them and places into the Trac database 
+ready for presentation.
+
+To start the slave:
+{{{
+bitrun-start [options-passed-to-slave]
+}}}
+
+This function invokes the '''bitten-slave''' command which does the setup of working directories and configuration file access. 
+Options to the function are passed to the command
+
+Useful options for initial debugging :
+   ||  -s, --single     ||   exit after completing a single build ||
+   ||  -n, --dry-run  ||     do not report results back to master ||
+   ||  -i SECONDS, --interval=SECONDS  ||   time to wait between requesting builds, poll interval  (default 300 s) ||
+
+
+= Bash Functions Primer  =
+
+Define the function :
+{{{
+demo-func(){
+  echo $FUNCNAME hello [$BASH_SOURCE]
+}
+}}}
+
+Invoke it :
+{{{
+demo-func
+demo-func hello []
+}}}
+
+Show the definition :
+{{{
+type demo-func
+demo-func is a function
+demo-func () 
+{ 
+    echo $FUNCNAME hello [$BASH_SOURCE]
+}
+}}}
+
+
+
+
+
+