Changes between Version 8 and Version 9 of NuWa_Slave

Timestamp:

09/02/10 17:06:03 (14 years ago)

Author:

blyth (IP: 140.112.102.77)

Comment:

--

NuWa_Slave

@@ -317 +317 @@
 }}}
  
-
-
-
- 
-
-
-
-
-
+ 
+== Investigating Issues ==
+
+The primary duty is to isolate the cause and report the problem to the author/responsible in the form of a Trac ticket that 
+enables the investigator to rapidly reproduce the issue.
+
+While investigating remember to stop the slave to avoid interference and resource competition from additional builds starting ...
+eg if using supervisord : 
+{{{
+[blyth@cms01 dybgaudi]$ supervisorctl
+dybslv                           RUNNING    pid 28651, uptime 1 day, 22:27:01
+C> stop dybslv
+dybslv: stopped
+}}}
+
+=== attach to python nuwa.py process with gdb ===
+
+Start the failing test :
+{{{
+[blyth@cms01 MDC10b]$ nosetests tests/test_mdc10b.py:test_dby0
+Warning in <TEnvRec::ChangeValue>: duplicate entry <Library.vector<short>=vector.dll> for level 0; ignored
+Run MDC10b.runLED_Muon.FullChain with double-pulsing of LEDs and no muons to produces 50 readouts ...
+}}}
+ 
+Attach gdb to the process and continue '''c''' :
+{{{
+[blyth@cms01 dybgaudi]$ gdb `which python` $(pgrep -f $(which nuwa.py))
+...
+Loaded symbols for /data/env/local/dyb/trunk/NuWa-trunk/dybgaudi/InstallArea/i686-slc4-gcc34-dbg/lib/libG4DataHelpers.so
+0xb6687b23 in ParticlePropertySvc::anti (this=0xaa28798, pp=0xaa66a98) at ../src/ParticlePropertySvc/ParticlePropertySvc.cpp:445
+445         const ParticleProperty* ap = *it ;
+(gdb)
+}}}
+
+Unfortunately this approach sometimes gets '''Killed''' for gdb '''Out of Memory'''.  
+
+=== running the command under gdb ===
+
+Grab the command from the source of the test(if simple) or process table :
+{{{
+ps --no-headers -o command  -p $(pgrep -f $(which nuwa.py)) > cmd
+}}}
+
+Edit the cmd file, fixup any missing quotes and prefixing with gdb command : '''set args'''
+
+Allowing :
+{{{
+[blyth@cms01 dybgaudi]$ gdb `which python` -x cmd
+GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
+Copyright 2004 Free Software Foundation, Inc.
+...
+}}}
+
+
+Capture the backtrace '''bt''' when meet problems :
+{{{
+ElecSimProc                           INFO Processing hit collections
+ToolSvc.EsIdealFeeTool                INFO Processing 73 pmt pulses.
+ToolSvc.TsMultTriggerTool             INFO Max multiplicity for DayaBayAD1 is 44
+*** glibc detected *** malloc(): memory corruption: 0x0fe95d10 ***
+
+Program received signal SIGABRT, Aborted.
+[Switching to Thread -1208318272 (LWP 17858)]
+0x00a1e7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
+(gdb)
+
+(gdb) bt
+#0  0x00a1e7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
+#1  0x00a5f915 in raise () from /lib/tls/libc.so.6
+#2  0x00a61379 in abort () from /lib/tls/libc.so.6
+#3  0x00a93e1a in __libc_message () from /lib/tls/libc.so.6
+#4  0x00a9b473 in _int_malloc () from /lib/tls/libc.so.6
+#5  0x00a9d0f1 in malloc () from /lib/tls/libc.so.6
+#6  0x04fa911e in operator new () from /usr/lib/libstdc++.so.6
+#7  0x032762ca in __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<char const* const, DybDaq::FeeTraits*> > >::allocate (this=0x32798c4, __n=1) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/ext/new_allocator.h:81
+#8  0x03276232 in std::_Rb_tree<char const*, std::pair<char const* const, DybDaq::FeeTraits*>, std::_Select1st<std::pair<char const* const, DybDaq::FeeTraits*> >, std::less<char const*>, std::allocator<std::pair<char const* const, DybDaq::FeeTraits*> > >::_M_get_node (this=0x32798c4) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_tree.h:356
+#9  0x03276159 in std::_Rb_tree<char const*, std::pair<char const* const, DybDaq::FeeTraits*>, std::_Select1st<std::pair<char const* const, DybDaq::FeeTraits*> >, std::less<char const*>, std::allocator<std::pair<char const* const, DybDaq::FeeTraits*> > >::_M_create_node (this=0x32798c4, __x=@0xbfe81c88) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_tree.h:365
+#10 0x03275ce5 in std::_Rb_tree<char const*, std::pair<char const* const, DybDaq::FeeTraits*>, std::_Select1st<std::pair<char const* const, DybDaq::FeeTraits*> >, std::less<char const*>, std::allocator<std::pair<char const* const, DybDaq::FeeTraits*> > >::_M_insert (this=0x32798c4, __x=0x0, __p=0xfe95b88, __v=@0xbfe81c88) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_tree.h:809
+#11 0x03275ac9 in std::_Rb_tree<char const*, std::pair<char const* const, DybDaq::FeeTraits*>, std::_Select1st<std::pair<char const* const, DybDaq::FeeTraits*> >, std::less<char const*>, std::allocator<std::pair<char const* const, DybDaq::FeeTraits*> > >::insert_unique (this=0x32798c4, __v=@0xbfe81c88) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_tree.h:929
+#12 0x0327583f in std::map<char const*, DybDaq::FeeTraits*, std::less<char const*>, std::allocator<std::pair<char const* const, DybDaq::FeeTraits*> > >::insert (this=0x32798c4, __x=@0xbfe81c88)
+    at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_map.h:360
+#13 0x032755cf in DybDaq::FeeTraits::defaultTraits () at ../src/FeeTraits.cc:52
+#14 0xb5880e3c in DayaBay::DaqReadoutPmtCrate::channel (this=0xfe97a80, channelId=@0xbfe81dc0) at ../src/DaqReadoutPmtCrate.cc:170
+#15 0xb5884bd5 in DayaBay::ReadoutPmtCrate::daqReadout (this=0xfe97780, run=0, event=0) at ../src/ReadoutPmtCrate.cc:77
+#16 0xaeb14500 in SingleLoader::execute (this=0xab6ec28) at ../src/SingleLoader.cc:112
+#17 0x03f95d2c in Algorithm::sysExecute (this=0xab6ec28) at ../src/Lib/Algorithm.cpp:558
+#18 0xaeb1f6fc in DybAlgorithm<DayaBay::ReadoutHeader>::sysExecute (this=0xab6ec28) at /data/env/local/dyb/trunk/NuWa-trunk/dybgaudi/InstallArea/include/DybAlg/DybAlgorithmImp.h:59
+#19 0x01825d45 in GaudiSequencer::execute (this=0xab6bc00) at ../src/lib/GaudiSequencer.cpp:100
+#20 0xb58d3823 in Stage::nextElement (this=0xab6ae78, pIStgData=@0xbfe8248c, erase=true) at ../src/Stage.cc:48
+#21 0xb58c0a4e in Sim15::execute (this=0xaae7608) at ../src/Sim15.cc:121
+Killed
+}}}
+
+
+Report findings in Trac tickets such as #565
+
+
+
+
+
+ 
+ 
+
+
+
+ 
+
+
+
+
+