Topics: AIX, PowerHA / HACMP, System Admin

NTP slewing in clusters

In order to keep the system time synchronized with other nodes in an HACMP cluster or across the enterprise, Network Time Protocol (NTP) should be implemented. In its default configuration, NTP will periodically update the system time to match a reference clock by resetting the system time on the node. If the time on the reference clock is behind the time of the system clock, the system clock will be set backwards causing the same time period to be passed twice. This can cause internal timers in HACMP and Oracle databases to wait longer periods of time under some circumstances. When these circumstances arise, HACMP may stop the node or the Oracle instance may shut itself down.

Oracle will log an ORA-29740 error when it shuts down the instance due to inconsistent timers. The hatsd daemon utilized by HACMP will log a TS_THREAD_STUCK_ER error in the system error log just before HACMP stops a node due to an expired timer.

To avoid this issue, system managers should configure the NTP daemon to increment time on the node slower until the system clock and the reference clock are in sync (this is called "slewing" the clock) instead of resetting the time in one large increment. The behavior is configured with the -x flag for the xntpd daemon.

To check the current running configuration of xntpd for the -x flag: 

# ps -aef | grep xntpd | grep -v grep
    root  409632  188534   0 11:46:45      -  0:00 /usr/sbin/xntpd
To update the current running configuration of xntpd to include the -x flag: 
# chssys -s xntpd -a "-x"
0513-077 Subsystem has been changed.
# stopsrc -s xntpd
0513-044 The /usr/sbin/xntpd Subsystem was requested to stop.
# startsrc -s xntpd
0513-059 The xntpd Subsystem has been started. Subsystem PID is 40932.
# ps -f | grep xntpd | grep -grep
    root  409632  188534   0 11:46:45      -  0:00 /usr/sbin/xntpd -x




If you found this useful, here's more on the same topic(s) in our blog:


UNIX Health Check delivers software to scan Linux and AIX systems for potential issues. Run our software on your system, and receive a report in just a few minutes. UNIX Health Check is an automated check list. It will report on perfomance, capacity, stability and security issues. It will alert on configurations that can be improved per best practices, or items that should be improved per audit guidelines. A report will be generated in the format you wish, and the report includes the issues discovered and information on how to solve the issues as well.

Interested in learning more?