Support:How To Run Toolpack Application In gdb

Latest revision as of 16:30, 18 February 2011

In case of Toolpack application crash, it's very useful to run the application inside the debugger (gdb) to catch the fault (even if the application binaries are compiled in 'release' without symbols, as gdb will still be able to provide call stack of all threads).

Notes:
For Toolpack setups with two redundant hosts, this procedure can be done on a "live" system without interrupting the service (no active calls will be lost, though few transient (ringing) calls will be dropped during the forced transitions from one application to it's standby).
For Toolpack setups without redundant hosts, service will be interrupted while the application to debug is not running (no new calls can be received in that period, though active calls continue to work).

Unfortunately, running a Toolpack application outside the Toolpack framework is not as simple as starting the application binary. A few environment variables need to be set first.

In that procedure, please replace the following values by those that correspond to your setup:
 '2.5.42'            : Replace that value by the package you are actually running
 'toolpack_engine'   : Replace by the application you wish to debug (gateway, toolpack_sys_mgr, etc.)
 'toolpack_engine_1' : Replace by the application instance name in your database
                      (as seen in the Web Portal "Application->Instancea" or tboamapp's "Application launch manager" page)

Pre-requisites:

gdb (debugger on linux) must be installed:
 - yum install gdb
If you get an error running "yum install gdb", it's possible that the host does not have a DNS properly set.
The procedure to configure a DNS can be found here: TBLinux:How_to_change_the_ip_address

Procedure:

First part of the procedure: stop the primary toolpack_engine application from running, and being re-launched by tboamapp
1- Log to the primary server (ssh)
2- cd /lib/tb/toolpack/pkg/2.5.42/bin/release/x86_64-linux64/
3- start tbx_cli_tools_remote application (./tbx_cli_tools_remote)
4- connect to "tboamapp" application
5- Select "Application launch manager" page (using TAB if appropriate)
6- press letter 's', will be prompted to choose 'Db id' of the application to stop manage
   Enter the 'Db id' of the toolpack_engine application (see list of application in the page)
   => This will prevent toolpack_engine to relaunch itself automatically when we stop it
7- press letter 'd', will be prompted to choose the 'hProc' of the process to destroy.
   Enter the value for toolpack_engine application
   => This will terminate toolpack_engine application, secondary host will take over
8- Press escape-escape, then 'control-C' to quit tbx_cli_tools_remote application

Then Set some environment variables to simulate Toolpack environment:
9- export TBDB_DRIVER=ODBC
   export TBDB_ODBC_DRIVER=MySQL
   export LD_LIBRARY_PATH=/lib/tb/toolpack/pkg/2.5.24/lib/release/x86_64-linux64/

Then start toolpack_engine application in the debugger:
10- gdb ./toolpack_engine 
11- run -db -name toolpack_engine_1
12- Quit toolpack_engine on secondary host to force pass control back to primary in the debugger
   (use tbx_cli_tools_remote to connect to the secondary toolpack_engine, press letter 'q' then 'y' then 'enter' to quit)

Wait for the crash to occur, then get the call stack of the crashed thread:
13- bt
14- thread apply all bt
    (press enter until backtrace of all thread has been printed)
15- Take the backtraces, and send that to TelcoBridges support
    (during that time, don't worry, secondary toolpack_engine has taken control of calls while debugger has paused the app)

Then restart toolpack_engine inside Toolpack as usual
16- Quit debugger, kill application
17- connect again to tboamapp using tbx_cli_tools_remote
18- Press 'a' then 'enter' to re-activate auto-launch of toolpack_engine
    tboamapp should restart it within few seconds