View source

In case of Toolpack application crash, it's very useful to run the application inside the debugger (gdb) to catch the fault (even if the application binaries are compiled in 'release' without symbols, as gdb will still be able to provide call stack of all threads). <br> <br> Notes: <br> For Toolpack setups with two redundant hosts, this procedure can be done on a "live" system without interrupting the service (no active calls will be lost, though few transient (ringing) calls will be dropped during the forced transitions from one application to it's standby). <br> For Toolpack setups without redundant hosts, service will be interrupted while the application to debug is not running (no new calls can be received in that period, though active calls continue to work). <br> <br> Unfortunately, running a Toolpack application outside the Toolpack framework is not as simple as starting the application binary. A few environment variables need to be set first. <br> 

 In that procedure, please replace the following values by those that correspond to your setup:
  '2.5.42'           &nbsp;: Replace that value by the package you are actually running
  'toolpack_engine'  &nbsp;: Replace by the application you wish to debug (gateway, toolpack_sys_mgr, etc.)
  'toolpack_engine_1'&nbsp;: Replace by the application instance name in your database
                       (as seen in the Web Portal "Application-&gt;Instancea" or tboamapp's "Application launch manager" page)

Pre-requisites:
 gdb (debugger on linux) must be installed:
  - yum install gdb
 If you get an error running "yum install gdb", it's possible that the host does not have a DNS properly set.
 The procedure to configure a DNS can be found here: [[TBLinux:How_to_change_the_ip_address]]

Procedure: 

 First part of the procedure: stop the primary toolpack_engine application from running, and being re-launched by tboamapp
 1- Log to the primary server (ssh)
 2- cd /lib/tb/toolpack/pkg/2.5.42/bin/release/x86_64-linux64/
 3- start tbx_cli_tools_remote application (./tbx_cli_tools_remote)
 4- connect to "tboamapp" application
 5- Select "Application launch manager" page (using TAB if appropriate)
 6- press letter 's', will be prompted to choose 'Db id' of the application to stop manage
    Enter the 'Db id' of the toolpack_engine application (see list of application in the page)
    =&gt; This will prevent toolpack_engine to relaunch itself automatically when we stop it
 7- press letter 'd', will be prompted to choose the 'hProc' of the process to destroy.
    Enter the value for toolpack_engine application
    =&gt; This will terminate toolpack_engine application, secondary host will take over
 8- Press escape-escape, then 'control-C' to quit tbx_cli_tools_remote application

 Then Set some environment variables to simulate Toolpack environment:
 9- export TBDB_DRIVER=ODBC
    export TBDB_ODBC_DRIVER=MySQL
    export LD_LIBRARY_PATH=/lib/tb/toolpack/pkg/2.5.24/lib/release/x86_64-linux64/

 Then start toolpack_engine application in the debugger:
 10- gdb ./toolpack_engine run -db -name toolpack_engine_1
 11- Quit toolpack_engine on secondary host to force pass control back to primary in the debugger
    (use tbx_cli_tools_remote to connect to the secondary toolpack_engine, press letter 'q' then 'y' then 'enter' to quit)

 Wait for the crash to occur, then get the call stack of the crashed thread:
 12- bt
 13- thread apply all bt
     (press enter until backtrace of all thread has been printed)
 14- Take the backtraces, and send that to TelcoBridges support
     (during that time, don't worry, secondary toolpack_engine has taken control of calls while debugger has paused the app)

 Then restart toolpack_engine inside Toolpack as usual
 15- Quit debugger, kill application
 16- connect again to tboamapp using tbx_cli_tools_remote
 17- Press 'a' then 'enter' to re-activate auto-launch of toolpack_engine
     tboamapp should restart it within few seconds