Watchdog

Depending on the nature of the turret not responding or crashed, the turret may not be accessible or unresponsive using a ssh session and the turret will need to be rebooted.  Any information available in the form of a compressed report file can then be extracted.

Firstly, check that the creation date and time of the compressed report file corresponds with the time of the problem observed as it is possible that the watchdog had run on a previously unnoticed occasion or may be left after a previous failure.  The report file is not automatically deleted but will be overwritten if watchdog is triggered again.  It won’t always be the case that these files are available, it depends on the type of failure.  Memory leakage is a possible cause of lockups.  However, it is possible that the watchdog had time and space to run before a lockup.

The watchdog generates a compressed report file after a known failure. Once generated and if the flag "application.watchdog.uploadreport" is set to true, then the watchdog attempts to upload the report to the TSS dump area.

The file named reason.txt contains the reason the report file was generated, the date and time that the watchdog detected the problem and the version of the turret software running.

The reason for the failure will be one of

1.        turret app crash

The turret app is no longer present in the process list for that turret.  A stack trace should always be generated by the turret application before it exits in the file named backtracefile.txt and when used with the turret application log file UTAApplication.log, can lead to discovering the fault directly.

2.        i2c lock up

There have been problems with the I2C hardware component and the watchdog attempts to detect a this.  The i2c is not responding to status requests made at 20 second then 60 second intervals.  These may time out (curl error 28) indicating that the turretApp is not respondng to the request.  The i2c regularly writes to /proc/drivers/i2c and the watchdog reads this file three times when generating the report.  Values in each of these three instances should change indicating the i2c driver is still active.

3.        cpu persistently high usage

This is detected by sampling the cpu usage every 20 second then 60 second intervals.  If the usage is persistently high triggers the watchdog to collect the logs.  Watchdog runs top and collects the output every minute snd these files can be interrogated to find which processes are using the cpu.  The top utility itself uses around 40% momentarily.

4.        Signal HUP being ignored

Log rotation occurs when the application log file reached around 6Mb in size.  When a log rotation is done, SIGHUP is temporarily ignored but should be caught again once the rotation has completed.  The turret app can occasionally get into a state where it is permanently ignoring this signal.  Ignoring this signal can lead to the application writing to the same log file eventually filling up the entire partition leading to the turret dying. Catching this condition will cause an additional file, /tmp/signalMask.txt,  to be written containing the details.

5.        Used disk space more than 90%

The flexible partitions, all but /opt are monitored and if usage goes above 90%, watchdog triggers a report.  Once the report is generated, watchdog continues.

6.        Manually requested

The report file generation was requested manually.  Watchdog has a facility that acts on the creation of a file named /tmp/watchdogTrigger.  The watchdog will continue monitoring after generating the report and deleting this temporary file.