Interrogating Netrix

Standard > Troubleshooting Data: References > Netrix

Useful commands to use in an ssh session

df	- show partition usage, look for 100% usage in any of the writable partitions, /opt is not writable, but /opt/data is.

ps ax

- show processes

ls -l /proc/[1-9]*/fd | grep UTAApplication -

shows the processes file descriptors for the application log file.

If the logs have not been rotated properly you will see a UTAApplication.log.1 or .2 in the list indicating that the log rotate signal has not been processed.

ifconfig

- show interface addresses and usage, tx/rx packets, errors.

cat /proc/driver/i2c

- many counters of i2c usage. Look for values changing in successive calls. If there is no change, it is possible that the i2c has locked up.

cat /proc/driver/eth

cat /proc/driver/eth_status

/tmp/top*

- written by watchdog.sh every minute for debugging purposes.

Analysing the backtracefile

A Turret App crash should leave two files, backtracefile.txt and signalcaught.txt. The backtracefile.txt contains a record for each stack frame at the time of failure. There may be many stack frames stored in this file as the failure may cause other threads to fail too. Generally, the failing process can be determined from the application log file, /var/log/UTAApplication.log and search for the first occurrence of a "signal caught" by the "CombinedApp".

Note the thread number and look at the corresponding stack for that thread in backtracefile.txt.

Function Calls from the CombinedApp

There will be a number of records with just addresses within the turret app. This is due to the turret application executable being stripped of debug information when it was built to keep the size of the executable to a minimum. However, the function names can be found from the unstripped version of the code which should still exist in clearcase. First, generate the turretApp Symbol table from linux,

cleartool setview <corresponding view>

cd /SIPNetrix_vob/IPTNetrix/IPTNetrix_ppc82xx_so/build

nm -Cn CombinedApp >> symbols.txt

To determine the function name in the turret app, search the symbol table. The symbol table contains records with the function start address and function name. To find the function associated with an address, search for the address in the symbols file that is the closest match. Bear in mind that there will be an offset involved as the symbol table contains the start address of the function named. The symbol will probably be a mangled C== function name which can be passed through a utility such as c++filt on linux to yield the fully classified function that was called.

Linux Signal Problems

When log rotate is run it generates a SIGHUP which should be caught by the turret application.

Occasionally, the app gets in a state whee the signal is ignored.

A list of signals being caught or ignored can be found in the process status as follows,

Find the process id of the Turret Application using

ps ax | grep turret

886 ? S 106:56 /opt/apps/turret/turret ...

The process status file can be found in /proc/<processId/status

For Example, cat /proc/886/status produces...

Name: turret

State: S (sleeping)

Tgid: 886

Pid: 886

PPid: 1

TracerPid: 0

Uid: 0 0 0 0

Gid: 0 0 0 0

FDSize: 256

Groups:

VmSize: 213576 kB

VmLck: 0 kB

VmRSS: 41464 kB

VmData: 36192 kB

VmStk: 36 kB

VmExe: 18212 kB

VmLib: 6308 kB

SigPnd: 0000000000000000

SigBlk: 0000000080000000

SigIgn: 8000000000001006

SigCgt: 00000003c00144e9

CapInh: 0000000000000000

CapPrm: 00000000fffffeff

CapEff: 00000000fffffeff

As can be seen the first bit (LSB) is set 0 in SigIgn and 1 in SigCgt. This is the correct setting.

If the turret app hasn't crashed and is behaving strangely, when asked to do so by BT Support, the debug/trace levels can be changed through the web diagostics interface to give more detail in the Application log file.

The log file should be recovered manually or via a TSS request.

Debug/trace levels should be restored after getting the extra detail.