UNIX Health Check documentation

Index
  1. Obtaining Support
  2. Using UNIX Health Check
  3. Reports generated by UNIX Health Check
  4. Versions of UNIX Health Check
  5. Installing UNIX Health Check
  6. Determining the version of UNIX Health Check
  7. Running checks individually
  8. Using the checkall master script
  9. Using options with the checkall master script
  10. Return codes
  11. Running categories of scripts
  12. Automating the download of UNIX Health Check
  13. Enabling password-less access through SSH on cluster nodes
  14. Resolving email issues
  15. Resolving issues found by UNIX Health Check

1. Obtaining Support
Getting support for UNIX Health Check is easy: Send us an email at support@unixhealthcheck.com or complete our online Contact form. When you ask for support, please include the following information:
  • Your contact information.
  • A description of the issue you've encountered.
  • The output file generated by UNIX Health Check that includes the issue.
For more information:
2. Using UNIX Health Check
We provide UNIX Health Check services to scan entire UNIX systems. Just like going to a doctor for a full check-up. You'll get a report of all features of your system, and how the system is doing. If the system is properly installed, configured and maintained, an all green report with a score of 100% will be given. If for some reason some things are off, we'll let you know in the report what is wrong, and how you can fix it.

Your goal should be to get the UNIX system(s) on a UNIX Health Check score of 95% or higher. There is no need to aim for a 100% score; there are always reasons why a certain system is configured (slightly) differently than how the best practices indicate how a system should be configured.

UNIX Health Check can be beneficial in several stages of a UNIX system lifecycle. Most of our customers simply run the entire health check daily, to provide them with a report of their server status. But doing a UNIX Health Check is also very useful at certain important points in the life cycle of a UNIX system:
  • Before putting a new UNIX system into production - to make sure the system was installed and configured correctly.
  • Before and after doing a change on a UNIX server - to make sure no issues exist before applying patches/changes, and that no adverse situations were introduced as part of a change.
  • Before or during audits - to make sure a system is compliant with company and federal regulations.
  • During yearly performance reviews - to determine if the system administrators have taken good care of the UNIX servers.
  • Before or during any Bare Metal Restore or Disaster Recovery exercises - to validate that a UNIX server is recovered correctly.
  • When doing a security review of a UNIX system - ensuring your system's security against outside vulnerabilities, such as hackers.
  • When taking over support of UNIX systems from others or other accounts to perform an initial system scan.
  • ... and of course:
  • As part of the daily routine to check and monitor UNIX systems, because not daily reviewing UNIX systems, can have costly consequences.

3. Reports generated by UNIX Health Check
Reports can be generated by UNIX Health Check in different formats. By default, it will generate a report in TEXT format (also known as "log" format). You can also select to have a report generated in web page (HTML) format, or comma separated (CSV) format, or in Extended Markup Language (XML) format. Use one of these options to select a specific format:

Option
Extension
Description
-l
.log
TEXT format (log) (default)
-h
.html
Web page format (HTML)
-c
.csv
Comma separated format (CSV)
-x
.xml
Extended Markup Language format (XML)

If you combine any of the options above, the order of the options used, determines which format is used to generate the report. The option that is mentioned last, will determine the format. For example, if you select options "-l", "-x" and "-h", the format of the report is HTML, because the "-h" option (for HTML output) was mentioned last.

A report generated by UNIX Health Check consists of multiple parts:

  1. The UNIX Health Check copyright message.

    A sample of the copyright message.

    This copyright message states that the use of UNIX Health Check is subject to license agreements, along with displaying information on how to make changes based on issues discovered by UNIX Health Check, and information about how to get additional support. You can suppress displaying this information by using the -b option.

  2. An overview section of the options selected when running UNIX Health Check.

    A sample of the section that shows the options selected.

    This section will display the options selected when UNIX Health Check was run; when the report was created; what version of UNIX Health Check was used; the name of the output file and the number of check scripts that were run. You can suppress this output by using the -b option.

  3. A system configuration section.

    A sample of the system configuration section.

    This is a short overview of many configuration items of the server, which includes items like the serial number of the server, the amount of memory, the CPU clock rate, and the OS level installed. The system configuration section is not displayed when the -b option is used.

  4. The output of individual check scripts.


    Two samples of the output of individual check scripts.

    In this part of the report you'll see the output of individual check scripts after each other. For each check script, it will show the name check script that was run, a description of the check script (if the -d option was used; the -d option displays descriptions of check scripts), the output of the check script, and if applicable, any errors that were generated when the check script was run.

    At the end, you'll also see the return code of the script (More information about return codes in Return codes), along with the amount of time in seconds that it took to run the check script. If the -g option (lowercase G) was used, then only the output of individual check scripts is shown in this section, when any of the check scripts returns a non-zero return code (meaning either an ERROR or WARNING status occurred).

    In the HTML output version of UNIX Health Check different colors are used to indicate the result for each individual check script. Check scripts that completed successfully are listed as green. Check scripts that ended with a WARNING status (return code 2), are listed in yellow. And check scripts that have ended with an error, are listed as red. This helps to identify the severity level of each individual check script.

  5. A results summary.

    A sample of the results section.

    This section displays the total run time for all check scripts, the number of check scripts that were run, the number of checks that returned an OK (return code 0), WARNING (return code 1) and ERROR (return code 2), and a total score for the system. It will finally point to the location of the output file that was generated by UNIX Health Check.

4. Versions of UNIX Health Check
Two versions of UNIX Health Check are available:
  • UNIX Health Check for AIX

    This version can be used on AIX systems only.
  • UNIX Health Check for Red Hat Enterprise Linux

    This version can be used on Red Hat Enterprise Linux, CentOS, Oracle Linux and Scientific Linux.

The default shell used in AIX is the Korn Shell, and to indicate that, all the check scripts in the AIX version of UNIX Health Check use extension "ksh". The default shell in Red Hat Enterprise Linux and all its derivatives is the Bash shell, and to indicate that, all the check scripts in the Red Hat Enterprise Linux version of UNIX Health Check use extension "sh". The Korn shell style check scripts of UNIX Health Check cannot be used on Red Hat Enterprise Linux (and its derivatives such as CentOS, Oracle Linux and Scientific Linux). The Bash shell style scripts of UNIX Health Check cannot be used on AIX.

When downloading UNIX Health Check software, two main files are available:
  • ahc_latest.tar

    This file is for AIX. AHC stands for AIX Health Check. This version can be used both on AIX and VIOS systems. And it can also be used on WPARs.

  • rhc_latest.tar

    This file is for Red Hat Enterprise Linux, CentOS, Oracle Linux and Scientific Linux. RHC stands for Red Hat Health Check.

5. Installing UNIX Health Check
UNIX Health Check software will be provided to you once a purchase order has been received, or when you've acquired UNIX Health Check online. The software will be made available to you through download from our website. As soon as we've received a purchase order or your online payment, we'll send you a login and password, so you can download the software: Two versions of UNIX Health Check are available:
  • UNIX Health Check for AIX
  • UNIX Health Check for Red Hat Enterprise Linux
Download the correct version for your system. The downloaded file will either be called ahc_latest.tar (for AIX) or rhc_latest.tar (for Red Hat Enterprise Linux, CentOS, Oracle Linux and Scientific Linux). Transfer the UNIX Health Check tar image onto the UNIX system you wish to run it on, and un-pack the file. It is recommended to un-pack the file in a separate folder. Any folder will do; you can transfer the tar image into the folder of your preference and un-pack it there. The example below assumes that you will be installing the software in folder /uhc. We recommend that you create a separate file system of at least 64 MB, and use that file system for storing the UNIX Health Check software and any reports generated by it. We also recommend that you run UNIX Health Check initially on one of your test servers, before running it on any production server, even though it is perfectly safe to run UNIX Health Check on production systems.

Note: All the commands shown in this documentation are to be run as user root; UNIX Health Check requires root, and no support is available if UNIX Health Check is run using any other user account.

UNIX Health Check requires 64 MB to be free in the /var and /tmp file systems. If that space is not available, either try clearing out these file systems by deleting old files, or you may try extending the file systems.

UNIX Health Check for AIX requires that the ncargs setting on the AIX system is set to 256 or higher. This setting determines the maximum number of command line arguments that can be used on the system, and due to the large number of check scripts available in UNIX Health Check for AIX, this value should be 256 or higher. The default value for ncargs in AIX 6 and AIX 7 is 256. On older AIX versions, you may encounter that it is still set to 6. Also, on some older AIX versions, ncargs can't be configured higher than 128.

To check the current ncargs setting, run:
# lsattr -El sys0 -a ncargs
To determine the maximum value ncargs can be configured to, run:
# lsattr -R -l sys0 -a ncargs
Set ncargs to either 256, or if 256 is not supported, set it to the highest available value, such as 128, by running:
# chdev -l sys0 -a ncargs=256
Create the installation folder:
# mkdir /uhc
# cd /uhc
Move the UNIX Health Check tar image into the folder (assuming you have placed the tar image in /tmp on the system that you wish to run UNIX Health Check on):
# mv /tmp/ahc_latest.tar /uhc
# ls *tar
ahc_latest.tar
Next, you may verify the check sum of the downloaded tar file. When you download the tar file from the UNIX Health Check website, you can find both the file size and the checksum in the download section. For example:


Note: This is an example. Please verify the actual file size and check sum while downloading UNIX Health Check software.

The check sum of UNIX Health Check tar image can be easily determined, for example for the AIX version of UNIX Health Check, by using the sum command:
# cd /uhc
# sum ahc_latest.tar
46389  6760 ahc_latest.tar
The first number indicated (in the example above listed as 46389) is the checksum of the file. The second number (6760 in the example above) indicates the block count).

Please note that a similar command, cksum, is also available, but this command will generate different check sums for files, and should not be used for verifying the downloaded software images of UNIX Health Check.

Also, please verify the file size matches with the file size provided when downloading UNIX Health Check, to ensure that you've correctly downloaded the UNIX Health Check tar file:
# ls -als ahc_latest.tar
6712 -rw-r--r-- 1 root system 6871040 Apr 26 13:53 ahc_latest.tar
Then, un-pack the image:
# tar -xvf ahc_latest.tar
x checkactivatedrpcservices.ksh, 1625 bytes, 4 media blocks.
x checkadaptersdefined.ksh, 297 bytes, 1 media blocks.
x checkadapters.ksh, 319 bytes, 1 media blocks.
x checkaiooa.ksh, 334 bytes, 1 media blocks.
x checkaiostatus.ksh, 1765 bytes, 4 media blocks.
x checkall.ksh, 26286 bytes, 52 media blocks.
x checkaudit.ksh, 235 bytes, 1 media blocks.
...
...
[lines removed]
...
...
x checkxntpd.ksh, 2557 bytes, 5 media blocks.
x checkzombies.ksh, 407 bytes, 1 media blocks.
x COPYRIGHT, 930 bytes, 2 media blocks.
x DESCRIPTIONS, 124444 bytes, 244 media blocks.
Once the tar command completes un-packing the image file, you will find many check scripts in the same folder, along with the master script, checkall.ksh on AIX and checkall.sh on Red Hat Enterprise Linux, and some other files like CHANGES, COPYRIGHT, COUNT, DOCUMENTATION, FAQ, SUPPORT, VERSION and DESCRIPTIONS.

File checkall.ksh or checkall.sh is the script that can be used to run more than one check script, and is referred to as the master script. You will generally use it to run all the check scripts, or groups of check scripts. Throughout this documentation, the master check script is either referred to as checkall, checkall.ksh (for AIX) or checkall.sh (for Red Hat Enterprise Linux, CentOS, Oracle Linux and Scientific Linux). Please ensure that you use the correct version, depending on the UNIX operating system used.

Files that start with check and end with sh (except for checkall.ksh or checkall.sh) are the actual check scripts that are run by UNIX Health Check. You should find several hundreds of those, depending on the version downloaded. Each check script checks a single function or a single item on the UNIX system. You can run each check script individually if you like, but it is more common to run all or groups of check scripts through the checkall master script. No check script will ever change anything on your system. It is safe to run them at any time, without affecting the system in any way. However, due to the nature of the commands run by some check scripts, it may be possible that some harmless errors may occur in log files or error report (for example when unavailable disks are attempted to be accessed by a check script). Please do not be alarmed by this; it is normal behavior.

File APARS is a file containing all high severity APARs available for AIX, as released by IBM. The source of this file is http://www-304.ibm.com/webapp/set2/flrt/doc?page=aparCSV and is used by UNIX Health Check for AIX to check for the correct APARs installed on the AIX system. This file is not included on the UNIX Health Check version for Red Hat Enterprise Linux.

File CHANGES is an overview of the most recent changes to UNIX Health Check.

File COPYRIGHT is the copyright message that is part of UNIX Health Check.

File COUNT shows the number of checks that is available in the full version of UNIX Health Check.

File DEMO provides additional information about the demo version of UNIX Health Check. This file is only present if the demo version of UNIX Health Check is used. The demo version can be found at www.unixhealthcheck.com/demo.

File DESCRIPTIONS contains a description for each check script available in the UNIX Health Check software package. It will provide you information of what each check script does. Additionally, each check script has a description in the header of the file. By simply viewing a check script in an editor such as VI, you will be able to see what the check script is checking. When the master check script checkall is used, UNIX Health Check will include the description for each check in the report, if the "-d" option is used; we'll discuss that later in this documentation.

File DOCUMENTATION provides a link to the UNIX Health Check documentation (this web page), so you will never have to search for the latest documentation for UNIX Health Check.

File FAQ provides a link to our frequently asked questions, also on the UNIX Health Check website.

File SUPPORT provides a link to our support page, in case you need additional support when using UNIX Health Check.

File VERSION contains the version number of UNIX Health Check. The version is an inverted date. For example, version 17.05.24 indicates the version of UNIX Health Check as released on May 24, 2017.

When the ahc_latest.tar or rhc_latest.tar image file has been un-packed, you can safely remove the tar image:
# rm -f *hc_latest.tar
The mode for all the check scripts should already be set to 755. Please verify that this is the case. If not, please run:
# chmod 755 check*sh
Please repeat the installation of UNIX Health Check on every UNIX system, that you wish to check with UNIX Health Check.

6. Determining the version of UNIX Health Check
To determine the version of UNIX Health Check that you are using, run the checkall.ksh script (on AIX) or checkall.sh script (on Red Hat Enterprise Linux, CentOS, Oracle Linux and Scientific Linux) using the "-u" option. The "-u" option can be used to show all the different options for UNIX Health Check, along with the version number:
# ./checkall.sh -u

Copyright (c) 2004-2017 UNIX Health Check - All Rights Reserved

www.unixhealthcheck.com

This is confidential and unpublished work of authorship subject to limited
use license agreements and is a trade secret, which is the property of 
UNIX Health Check (www.unixhealthcheck.com). All use, disclosure and/or 
reproduction not specifically authorized in writing by UNIX Health Check 
is strictly prohibited.

Any expressed or implied warranties are disclaimed. In no event shall UNIX 
Health Check be liable for any direct, indirect, incidental, special, 
exemplary, or consequential damages (including, but not limited to, loss 
of use, data, profits, or business interruption) however caused and on any 
theory of liability, whether in contract, strict liability, or tort 
(including negligence or otherwise) arising in any way out of the use of 
these scripts, even if advised of the possibility of such damage.

Version: 17.05.24

Usage : checkall.sh [ -b ] [ -c ] [ -C ] [ -d ] [ -D ] [ -e ]
                    [ -E script1,script2,... ] [ -f file ] [ -g ] [ -G ]
		    [ -h ] [ -i ] [ -l ] 
		    [ -m emailaddress1,emailaddress2,... ]
                    [ -n ] [ -s script1,script2,... ] [ -u ] [ -v ]
                    [ -w # ] [ -x ]

 -b : Basics: no system configuration, end results and copyright message.
 -c : Write comma separated output (CSV).
 -C : Comma separated list of categories of scripts to run.
 -d : Add descriptions of checks to output.
 -D : Enable the debug option - for UNIX Health Check support use only.
 -e : Exclude ALL check scripts (will only run system configuration).
 -E : Comma-separated list of check scripts to exclude from the run.
 -f : Specify location of output.
 -g : Suppress showing output of check scripts that complete successfully.
 -G : Display check scripts that complete with ERROR; no WARNINGs.
 -h : Write web page output (HTML).
 -i : Inventory. Deprecated option. Use "-C inventory" instead.
 -l : Write text output to log file (TEXT) (default).
 -m : Comma-separated list of email addresses to send output to.
 -n : Only send email when result is less than 100% (errors only).
 -s : Comma-separated list of check scripts to run.
 -u : Display usage information.
 -v : Verbose output to screen.
 -w : Specify the width of the output.
 -x : Write Extended Markup Language output (XML).
The example above shows the version number after the copyright message. Another way to determine the version of UNIX Health Check is to read the VERSION file:
# cat VERSION
17.05.24
The version of UNIX Health Check is a reversed date. Version 17.05.24 is the release of UNIX Health check on May 24, 2017.

The example above also shows the different options that can be used with UNIX Health Check. These are discussed in more detail in Using options with the checkall master script.

Be sure to check regularly for updated versions of UNIX Health Check, as they are released often. The latest version is always available on our Support page (Click on the "download the latest version of UNIX Health Check" link in the download section, and log in with the user credentials that were provided to you upon purchase of a license for UNIX Health Check).

There's also a possibility to automate the download of UNIX Health Check directly to your UNIX systems. For more information see: Automating the download of UNIX Health Check.

7. Running checks individually
Once the tar image has been un-packed, you will notice a lot of files within the chosen folder. Most of the files are individual check scripts that start with "check". Each individual script (except for script checkall.sh or checkall.ksh) will check a certain function or configuration of the system. You can run each script individually, if you like. For example, to check if wget is installed, run checkwget.ksh (on AIX) or checkwget.sh (on Red Hat Enterprise Linux). To determine the model name of the UNIX server, run checkmodelname.ksh or checkomodelname.sh.

For example, on AIX:
# ./checkwget.ksh
wget-1.9.1-1
# ./checkmodelname.ksh
9117-MMB
And an example on Red Hat Enterprise Linux:
# ./checkwget.sh
wget-1.14-13.el7.x86_64
# ./checkmodelname.sh
Dell Inc. PowerEdge R820
Please note that UNIX Health Check is designed to run as user root only. Many scripts will still run using a different user account, but UNIX Health Check is only supported by running via the root account. Root access is required, because UNIX Health Check runs several root-level commands, and not using the root user account may result in unwanted errors generated by the check scripts. Also, UNIX Health Check never makes any change to the UNIX system; it only reports. UNIX Health Check is not designed to automatically resolve any issues found, because the configuration can depend on many different factors within your environment or infrastructure. From the output of the check script(s), you can determine what issue was found (that is, if an issue was found), and what possible action can or should be taken to remediate the issue.

8. Using the checkall master script
As described earlier, most users will usually use the checkall.ksh (for AIX) or checkall.sh (for Red Hat Enterprise Linux, CentOS, Oracle Linux and Scientific Linux) master script to run either a selection or all the check scripts to generate a report. Running check scripts individually can be a tedious task with hundreds and hundreds different check scripts. You can use the checkall master script to run a combination of different check scripts. For example, if you wish to run check scripts checkwget.ksh and checkmodelname.ksh on your AIX server, you can use the "-s" (for "Script") option:
# ./checkall.ksh -s checkwget.ksh,checkmodelname.ksh
Multiple check scripts can be specified using the "-s" option. Scripts need to be separated using the comma (",") only.

Please note that when you run the above command, that no output will be shown on the screen. When the checkall.ksh script completes running all or the selected scripts, it will simply return the command prompt. UNIX Health Check will log its output by default to a log file. After running the above command, you will be able to find a log file within the folder in which checkall.ksh is located:
# ls *log
checkall_hostname.log
You can open the log file with an editor and review the output. If you rather have UNIX Health Check both log its output to a log file and display the output on the screen while it is running, then add the "-v" (for "Verbose output") option:
# ./checkall.ksh -s checkwget.ksh,checkmodelname.ksh -v
You can determine the width of the output using the -w (for "Width") option. For example, if you wish UNIX Health Check to generate a report that is only 70 characters wide, run:
# ./checkall.ksh -s checkwget.ksh -v -w 70
######################################################################

2017-05-24 14:41:45: UNIX HEALTH CHECK FOR AIX

Copyright (c) 2004-2017 UNIX Health Check - All Rights Reserved

www.unixhealthcheck.com

This is confidential and unpublished work of authorship subject to
limited use license agreements and is a trade secret, which is the
property of UNIX Health Check (www.unixhealthcheck.com). All use,
disclosure and/or reproduction not specifically authorized in writing
by UNIX Health Check is strictly prohibited.

Any expressed or implied warranties are disclaimed. In no event shall
UNIX Health Check be liable for any direct, indirect, incidental,
special, exemplary, or consequential damages (including, but not
limited to, loss of use, data, profits, or business interruption)
however caused and on any theory of liability, whether in contract,
strict liability, or tort (including negligence or otherwise) arising
in any way out of the use of these scripts, even if advised of the
possibility of such damage.

This report is generated by UNIX Health Check for AIX. It is an
overview of check scripts run on an AIX system, and depending on the
options selected when the checkall.ksh script was run, it will list
each check script, the returncode of the check script, the output of
the check script and a description. At the end of this report is an
overview of all scripts run and a score report.

Any individual implementing changes should completely understand the
change and deem each change appropriate before it is applied to the
system. As a standard rule, please take into consideration the impact
on other components before implementing the change. Also, we
encourage all to conduct a peer review of all changes before
implementation. Most importantly, if the effect of a change is not
fully understood, do not implement that change until a satisfactory
explanation can be given as to what the effects of the change are. We
recommend implementation of one change at a time. The application of
many changes all at once will increase the likelihood of confusion,
if issues arise.

For more information, check website http://www.unixhealthcheck.com.
For support, email to support@unixhealthcheck.com.

######################################################################

2017-05-24 14:41:45: OPTIONS SELECTED
-------------------------------------

2017-05-24 14:41:45: Version:         17.05.19
2017-05-24 14:41:45: Start at:        05/24/2017 14:41:44 PDT
2017-05-24 14:41:45: Options:         -s checkwget.ksh -v -w 70
2017-05-24 14:41:45: Output file:     checkall_server01.log
2017-05-24 14:41:45: Width:           70
2017-05-24 14:41:45: Display:         All checks
2017-05-24 14:41:45: Descriptions:    No
2017-05-24 14:41:45: Output type:     TEXT
2017-05-24 14:41:45: # Checks:        1
2017-05-24 14:41:45: Script:          checkwget.ksh

######################################################################

2017-05-24 14:41:50: SYSTEM CONFIGURATION
-----------------------------------------

2017-05-24 14:41:50: Hostname:        server01
(server01.unixhealthcheck.com)
2017-05-24 14:41:50: IP Address:      172.16.42.43
2017-05-24 14:41:50: IP Assignment:   Static
2017-05-24 14:41:50: Subnet Mask:     255.255.255.0
2017-05-24 14:41:50: Default Gateway: 172.16.42.1
2017-05-24 14:41:50: Name Server(s):  172.16.42.56 172.16.42.57
2017-05-24 14:41:50: LPAR / VM:       1 SERVER01
2017-05-24 14:41:50: OS Level:        AIX 7.1.4.32 7100-04
2017-05-24 14:41:50: PowerHA Level:   7.1.3.6
2017-05-24 14:41:50: Model:           IBM,8286-42A IBM Power S824
2017-05-24 14:41:50: Serial Number:   21B096W
2017-05-24 14:41:50: Firmware Level:  SV860_082
2017-05-24 14:41:50: Kernel:          64 bit
2017-05-24 14:41:50: Hardware:        64 bit
2017-05-24 14:41:50: Processor Type:  PowerPC_POWER8
2017-05-24 14:41:50: CPU Clock Rate:  3525 MHz
2017-05-24 14:41:50: rPerf:           421.90 rPerf  based on 24.00
Virtual CPU cores
2017-05-24 14:41:50: CPUs:            24
2017-05-24 14:41:50: Logical CPUs:    192
2017-05-24 14:41:50: Capacity:        Min: 1.00 Entitled: 24.00 Max:
24.00 Increment: 1.00
2017-05-24 14:41:50: Physical CPUs:   24
2017-05-24 14:41:50: System type:     mode=Capped
type=Dedicated-SMT-8 weight=0 smt=8
2017-05-24 14:41:50: Virtual CPUs:    Desired: 24 Min: 1 Online: 24
Max: 24
2017-05-24 14:41:50: Memory:          255744MB
2017-05-24 14:41:50: Memory Settings: min: 256MB, desired: 255744MB,
max: 262144MB, online: 255744MB
2017-05-24 14:41:50: Paging Space:    32768MB (1% in use)
2017-05-24 14:41:50: Uptime:          02:41PM up 9 days, 2:41, 1
user, load average: 2.28, 2.29, 2.47

######################################################################

2017-05-24 14:41:51: CHECK SCRIPTS
----------------------------------

----------------------------------------------------------------------
2017-05-24 14:41:50: Running check script 1 of 1: checkwget.ksh
2017-05-24 14:41:51: Check checkwget.ksh completed successfully:
returncode 0
2017-05-24 14:41:51: Runtime: 1 second(s)
2017-05-24 14:41:51: 100% complete - 0 checks to go.
----------------------------------------------------------------------

######################################################################

2017-05-24 14:41:51: RESULTS
----------------------------

2017-05-24 14:41:51: Run time for all checks              : 7 seconds
2017-05-24 14:41:51: Total number of checks               : 1
2017-05-24 14:41:51: # Checks with result OK              : 1
2017-05-24 14:41:51: # Checks with result WARNING         : 0
2017-05-24 14:41:51: # Checks with result ERROR           : 0
2017-05-24 14:41:51: Score [Percentage OK/WARNING]        : 100.00 %

2017-05-24 14:41:51: For details see logfile              :
/uhc/checkall_server01.log

######################################################################
Note that the width setting should be between 70 or 1024 characters. By default, UNIX Health Check will produce output width a width that matches the current terminal, or if that cannot be determined, a width of 130 characters wide. This width only applies to regular text output, and does not apply to different output formats, such as CSV, XML and HTML (discussed later in this documentation).

To simply run all the check scripts, for example on AIX, run:
# ./checkall.ksh
By not specifying the -s option, UNIX Health Check's master script, checkall, will run all the check scripts, and generate an output file. Please note that when run without the -s option, the checkall.ksh script will determine which scripts should be run on your system, and may not run all scripts available. For example, if your AIX system is not a VIOS, or is not part of a PowerHA/HACMP cluster, checkall.ksh will skip any checks related to VIOS and/or PowerHA/HACMP systems, reducing the number of scripts that it should run, and thus shortening the time needed to run all the scripts. Therefore, you may see that checkall runs less check scripts than what is included in the full UNIX Health Check software image.

9. Using options with the checkall master script
Shown below are all the options you can use with the UNIX Health Check master script checkall:

OptionDescription
-b
Shows only the basic output. It will not show any of the end results, the configuration section will be skipped and no copyright message will be shown.
-c
This option will write a CSV file output instead of the default log file (-l) output. CSV is comma separated output, which can be useful if you wish to use the information generated by UNIX Health Check in Microsoft Windows Excel or in a database.

Note: The order of the options determine which type of output is written. If you specify both the -l and -c option, the last option provided will determine the type of output written. This also applies to the -h (for HTML output) and -x (for XML output) options.

By default, UNIX Health Check will write CSV output to a file called checkall_hostname.csv in the same folder where UNIX Health Check is run from.

When using the CSV output format, no descriptions will be added to the output, even when option -d has been specified, as this will not fit in the comma separated output format.
-C
This option allows running scripts of a certain category. Specify a single category, or a comma separated list of categories of scripts to be run. For example, to run all security scripts, use the -C option as follows:
-C security
Or to run both memory and storage related scripts, use the -C option as follows:
-C memory,storage
You can choose from the following categories:

backupRun all scripts that check the backup of the system, including TSM, EMC Avamar/Networker, OSSV Snapvault, Commvault Simpana, and Veritas Netbackup.
bootingRun all check scripts that check items of the boot sequence, including inittab, rc.d files, autorestart, and start scripts.
capacityRun all check scripts that check the capacity of the system, including storage, CPU, network and memory.
clusterRun all scripts that check cluster configuration and status, including PowerHA/HACMP (on AIX), Red Hat Cluster (on Red Hat Enterprise Linux) and Veritas clustering.
daemonsRun all check scripts that check daemons and services on the system, including inetd.
datetimeRun all check scripts that check date and time settings on the server, including ntp configuration.
debuggingRun all check scripts that check debug items, such as debug filesets, core dump settings, error logging and system dumps.
hardwareRun all check scripts that check the physical hardware of the system, including adapters, devices, power supplies, firmware, and LPAR configuration.
inventoryRun all check scripts that generate inventory information and only display output of commands for inventory purposes.
loggingRun all check scripts for checking log files, alog and (r)syslog configuration.
mailRun all check scripts that check mail related items, such as the mail configuration and aliases, Sendmail and Postfix.
memoryRun all check scripts that check memory usage, vmo options (on AIX), swap space and memory cache.
monitoringRun all check scripts that check the monitoring of the system, including BMC Patrol, Ganglia, Marimba and Nimsoft.
networkRun all check scripts that check network settings, DHCP, DNS, and NFS.
performanceRun all check scripts that check the performance of the system, including those that check Active Memory Expansion (on AIX).
powervmRun all check scripts that check PowerVM configuration and settings, including dlpar, VIOS, and WPARs (AIX only).
printingRun all check scripts that check printing and printer queues, including Cups.
redundancyRun all check scripts that check redundancy features and configuration on the server (does not include cluster items).
schedulingRun all check scripts that check the scheduling of jobs on the system, including at and cron.
securityRun all check scripts that check permissions/owner of files, the ssh configuration, old files, user accounts and user limits.
softwareRun all check scripts that check the software installed on the system, such as filesets (lslpp) and rpm packages, and additional software installed on the server.
storageRun all check scripts that check the storage of the system, including SDD, SDDPCM, multi-pathing, disks, and EMC Powerpath.

Please note that check scripts can belong to multiple categories. For example, a script that checks if xmdaily on AIX is disabled in the root crontab, checkxmdaily.ksh, belongs to both the category "scheduling" and the category "monitoring", as the xmdaily entry in the root crontab controls running xmwlm/topasrec processes for monitoring purposes on the system.
-d
This option will add descriptions of each individual check script to the output. This is very useful if you wish to know how to resolve certain issues. The descriptions for check scripts can also be found in the DESCRIPTIONS file.
-D
This option will enable the debugging feature of the checkall.ksh master check script. Use this option only when instructed to do so by the UNIX Health Check technical support team, for the purpose of gathering information regarding an issue with UNIX Health Check.
-e
This option will exclude running all scripts. This may sound weird, but this can be useful to only generate the system configuration section.

The -e option is overruled by the -s option. If one or more check scripts to run were specified using the -s option, these check scripts will still be run, even if the -e option was used as well.

The -e option however, overrules the -C and -E options. If you choose to run a certain category of scripts using the -C option, or if you exclude specific scripts using the -E option, then including the -e option will result in all the scripts being excluded.
-E
This option can be used to exclude specific scripts from being run. This option is useful if one or more check scripts generate a warning or error, and you have made sure that the warning or error does not apply to your environment. In that case, you can use the -E option to exclude specific scripts. Provide a comma-separated list of scripts that you would like to have excluded. For example:
# ./checkall.ksh -v -E checkbit.ksh,checkuptime.ksh
The -e option (lowercase) overrules the -E option (uppercase). If both are specified, the -e option will exclude all check scripts.
-f
You can use this option to specify a different location to write the output to. By default, the output will be written into the same folder from which the checkall.ksh script is run. You need to supply the filename, or the full path to the filename for this option, for example:
# ./checkall.ksh -f filename
# ./checkall.ksh -f /full/path/to/filename
-g
This option suppresses the output of any checks that complete successfully. This is specifically very useful, when running all scripts through the master check script checkall.ksh (AIX) or checkall.sh (Red Hat Enterprise Linux, CentOS, Oracle Linux and Scientific Linux). Many of the check scripts will most likely return a null (zero) error code, indicating they have completed successfully, and thus a report of those check scripts is usually not worth reviewing, because they complete successfully and no issue is identified. Using the -g option will make sure the checks that complete successfully will not end up in the report. By default, the output of all scripts is shown.
-G
This option suppresses the output of any checks that complete successfully - and check scripts that return a WARNING (return code 2). Only check scripts that return an ERROR (return code 1) will be displayed. The -G option supersedes the -g option. This option is useful if you only wish to review the most critical items identified on the system, and when you do not care about any WARNING messages.
-h
This option will write an HTML file output instead of the default log file (-l) output. By generating HTML output, it is easier to see which check scripts return an error or warning, and which check scripts complete successfully. This is accomplished by using different colors for each script. A script that completes successfully (or return code 0) will be listed as green. Check scripts that end with a WARNING (or return code 2) will be listed as yellow. Check scripts that end with an ERROR (or return code 1) will be listed as red.

Note: The order of the options determine which type of output is written. If you specify both the -l and -h option, the last option provided will determine the type of output written. This also applies to the -c (for CSV output) and -x (for XML output) options.

By default, UNIX Health Check will write HTML output to file called checkall_hostname.html in the same folder where UNIX Health Check is run from.

A sample of HTML output can be seen on the Sample reports page.

To view the HTML reports that UNIX Health Check generates, one of the following browsers is required:
  • Microsoft Internet Explorer 8 or higher
  • Microsoft Edge
  • Mozilla Firefox
  • Google Chrome
Other browsers may or may not work.
-i
This is an old option, and its use is deprecated. It was used to generate an inventory section. If you wish to run any scripts of the inventory category, you should use option "-C inventory" instead.
-l
This option will write a text log file output. This is the default output type if neither of the options -l, -c, -x or -h is provided. By default, UNIX Health Check will write a text log file to file name checkall_hostname.log in the same folder where UNIX Health Check is run from.

Note: The order of the options determine which type of output is written. If you specify both the -h and -l option, the last option provided will determine the type of output written. This also applies to the -c (for CSV output) and -x (for XML output) options.
-m
This option can be used to provide a comma-separated list of email addresses to send output to, for example:
# checkall.ksh -m info@redhathealthcheck.com,info@aixhealthcheck.com
Mail is sent through the mailx command. Before you attempt to use this option, please make sure that it is possible to send email by running:
# cat /etc/hosts | mailx -v -s subject my@emailaddress.com
Email will be sent in the type of output that you've specified. If you didn't specify any of the -c, -h, -x or -l options, a text file output (by default, the -l option) will be sent through email. If you specified -c, a CSV style output will be sent through email. If you specified -h, an HTML type report will be sent through email. If you specified -x, an XML type report will be sent through email.

To view the HTML reports by email, only Microsoft Outlook 2003/2010/2016 and Windows Live Mail are supported. Other email clients may or may not be able to view the HTML report generated by UNIX Health Check properly.

Multiple email addresses may only be separated by a single comma.
-n
The -n option will ensure that only output is sent by email if there are any check scripts that didn't complete successfully (return code 0).

This is very useful to use in combination with the -m option. For example, if you have several crontab entries set up to monitor your systems, you can use the -n option to only alert you by email if there is indeed a check that generates an ERROR message (return code 1). No email is sent if all the check scripts complete successfully (return code 0) or if only check scripts completed with a WARNING message (return code 2).

For example, you could add the following to the root user's crontab on an AIX system, to alert you if there's an issue on the UNIX system:
# Check every hour:  Free space in file systems, 
# paging space usage, memory utilization, 
# high CPU using processes, printer queues, 
# error report, zombie processes
0 * * * *   /uhc/checkall.ksh -gdhbnm email@address.com 
	    -s checkfreespaceinfs.ksh,checkpgspusage.ksh,
            checkmemoryutilization.ksh,checkhighcpu.ksh
5 * * * *   /uhc/checkall.ksh -gdhbnm email@address.com 
            -s checkenq.ksh,checkerrpt.ksh,checkzombies.ksh
# Check once a day: TSM backups
9 9 * * *   /uhc/checkall.ksh -gdhbnm email@address.com 
	    -s checktsmsched.ksh
# Check once a week: mksysb backups, logical volume/file 
# system structure
0 8 * * 1,5 /uhc/checkall.ksh -gdhbnm email@address.com 
            -s checkmksysb.ksh,checklvfscreate.ksh
A useful combination for the -n option, is to also use the -b option. This way, also any end results and copyright messages are eliminated, thus further reducing the emails sent out.
-s
You can use the -s option to provide a comma-separated list of check scripts to run. The default action of checkall.ksh is (without supplying the -s option) to run all the check scripts. With the -s option, you can specify which check scripts to run. For example:
# ./checkall.ksh -s checkbit.ksh,checkpgspusage.ksh
Multiple check scripts may only be separated by a single comma. Do not specify any folder names when specifying any check scripts to run. Only the name of a check script including the .ksh extension (on AIX) or the .sh extension (on Red Hat Enterprise Linux, CentOS, Oracle Linux and Scientific Linux) is allowed. For example: "-s /tmp/checkbit.ksh" will not work, but "-s checkbit.ksh" will work. UNIX Health Check assumes to find all the check scripts in the same folder where checkall.ksh or checkall.sh is run from. UNIX Health Check will skip running any scripts it can't find in the same folder where checkall.ksh or checkall.sh is run from.

The -s option overrules the -e option. Scripts specified with the -s option will be run, even if the same scripts were excluded using the -e option. The -s option also overrules the -E option. If specific scripts were excluded using -E option, and still included using the -s option, then the scripts will be run. Finally, the -s option also overrules the -C option. If certain scripts were specified using the -s option, then the -C option (for specifying categories of scripts) will be ignored.
-u
The -u option displays the usage information, and then exits checkall.ksh. Using the question mark (-?) instead of the -u option has the same effect.
-v
The -v option displays verbose output information on screen, while checkall.ksh is running.
-w
With the -w option you can specify the width of the output. By default, UNIX Health Check will generate output of 130 characters wide or at the same width of the terminal that it is run in. You can specify any value with -w from 70 to 1024 characters:
# ./checkall.sh -w 512
Specifying the width of the output has no effect if HTML output (-h) is selected.
-x
This option will write an XML file output instead of the default log file (-l) output. By generating XML output, output of UNIX Health Check can be loaded into various other systems/applications or can be viewed in tree-like form with the use of a XML reader.

Note: The order of the options determine which type of output is written. If you specify both the -l and -x option, the last option provided will determine the type of output written. This also applies to the -c (for CSV output) and -h (for HTML output) options.

By default, UNIX Health Check will write HTML output to file checkall_hostname.xml in the same folder where UNIX Health Check is run from.

To view the XML reports that UNIX Health Check produces, an XML reader is required, for example the free XML editor, Foxe, from First Object, or the free MindFusion XML Viewer from MindFusion.

10. Return codes
Each check script ends with one of the following three exit/return codes:

0
The script completed successfully.
1
The script found an ERROR.
2
The script ended with a WARNING.

Errors are usually something that requires some form of remediation, and should be resolved, if possible. Warnings are just that, warnings. These may be ignored, but it might still be useful to consider looking into the warnings.

You can see the return code of a single check script by checking the $? value. For example:
# ./checkpgspsize.ksh
# echo $?
0
This particular script ended with return code 0, and thus ended successfully (no ERROR or WARNING is generated) and thus all it checked is okay.

The following is an example of a check script that ends with an ERROR status:
# ./checkbosadtdebug.ksh
Fileset bos.adt.debug not installed.
# echo $?
1
As you can see, a script that ends with an ERROR (or a WARNING) status, will also output a message about the error found.

11. Running categories of scripts
UNIX Health Check includes a feature to run certain categories of scripts. Instead of running all the check scripts that are included in UNIX Health Check, this option allows you to focus on those features deemed relevant to you at that time. For example, you may want to check all the security items of the UNIX system, and not be bothered by anything else. Or sometimes, you may wish to generate only an inventory, by running only those scripts that provide inventory type information.

You can use the "-C" option available in UNIX Health Check, to run a category of scripts. For example, to run all the security type scripts, run:
# ./checkall.ksh -C security
Likewise, if you wish to generate the inventory information of the UNIX system, run:
# ./checkall.ksh -C inventory
You can also combine several categories of scripts by providing a comma-separated list of categories to be run. For example, if you wish to run all the storage and security type scripts, you can run:
# ./checkall.ksh -C storage,security
A detailed list of available categories can be found in section Using options with the checkall master script.

Please note that scripts may belong to multiple categories. For example, a script that checks if xmdaily is disabled in the root crontab, checkxmdaily.ksh, belongs to both the category "scheduling" and the category "monitoring", as the xmdaily entry in the root crontab controls running xmwlm/topasrec processes for monitoring purposes on the system.

Also keep in mind that by selecting one or more categories, that this determines how many scripts will be run. For example, the security category is a large category of scripts, that includes over 300 scripts to be run, while the datetime category is a much smaller category, that includes only a few scripts.

12. Automating the download of UNIX Health Check
It is easy to automate the download of the latest version of UNIX Health Check to your UNIX system. Doing so, removes the need for downloading UNIX Health Check manually each time a new version of UNIX Health Check is released, and removes the need to transfer the downloaded file onto your UNIX systems manually.

12a. What do you need?
  • Install either curl or wget on your UNIX system, if not already installed. For AIX: These tools are not part of a regular AIX installation. You can download the RPM for these tools from the AIX Toolbox for Linux Applications. For Red Hat Enterprise Linux: You can use the familiar yum command to install the packages.
  • Your UNIX system needs Internet access, and needs to be able to resolve the "unixhealthcheck.com" domain through DNS.

    For example, test with a simple ping:
    # ping unixhealthcheck.com
  • You need a valid license for UNIX Health Check, which provides you a login (in the format of an email address) and a password to access the UNIX Health Check download page. If you are unable to download the UNIX Health Check software manually from the download page, then you won't be able to automate the download as well.
12b. Examples of using curl and wget to download UNIX Health Check

Here's an example of how you can use curl to download the latest version of UNIX Health Check (assuming your login/email address is "user@email.com" and your password is "mypassword"):

For AIX:
curl -d "emailaddress=user@email.com&password=mypassword" \
   --referer http://www.unixhealthcheck.com \
   http://www.unixhealthcheck.com/downloadauto.php \
   -o ahc_latest.tar
For Red Hat Enterprise Linux, CentOS, Oracle Linux and Scientific Linux:
curl -d "emailaddress=user@email.com&password=mypassword" \
   --referer http://www.unixhealthcheck.com \
   http://www.unixhealthcheck.com/downloadauto.php \
   -o rhc_latest.tar
This command will download file ahc_latest.tar or rhc_latest.tar in your current folder. Please note that providing the email address, password and referer is required. Without it, the download will fail.

The same download can be accomplished with wget:

For AIX:
wget --post-data 'emailaddress=user@email.com&password=mypassword' \
   --referer=http://www.unixhealthcheck.com \
   http://www.unixhealthcheck.com/downloadauto.php \
   -O ahc_latest.tar
For Red Hat Enterprise Linux, CentOS, Oracle Linux and Scientific Linux:
wget --post-data 'emailaddress=user@email.com&password=mypassword' \
   --referer=http://www.unixhealthcheck.com \
   http://www.unixhealthcheck.com/downloadauto.php \
   -O rhc_latest.tar
Once you have downloaded the ahc_latest.tar or rhc_latest.tar file, you will be able to un-tar the file and use UNIX Health Check.

12c. Automating download with a script

You can automate downloading UNIX Health Check easily with the use of a shell script. The script that is shown below downloads UNIX Health Check with the use of wget, will un-pack the downloaded file, run UNIX Health Check, send an HTML style report through email, and clean up afterwards.

For AIX:
#!/usr/bin/ksh
 
# VARIABLES
emailaddress="user@email.com"
password="mypassword"
folder="/uhc"
wget="/usr/bin/wget"
myfile="ahc_latest.tar"
 
# Test if folder already exists.
if [ -d ${folder} ] ; then
        echo "Folder ${folder} already exists. Aborting."
        exit
fi
 
# Test if wget is executable.
if [ ! -x ${wget} ] ; then
        echo "Wget does not exist or is not executable. Aborting."
        exit
fi
 
# Create a folder to store UNIX Health Check:
echo "Creating folder ${folder}..."
mkdir -p ${folder}
 
# Download the file.
echo "Downloading UNIX Health Check..."
${wget} --post-data "emailaddress=$emailaddress&password=$password" \
   --referer=http://www.unixhealthcheck.com \
   http://www.unixhealthcheck.com/downloadauto.php \
   -O ${folder}/${myfile} >/dev/null 2>&1
 
# Un-pack the downloaded file.
if [ -s ${folder}/${myfile} ] ; then
        cd ${folder}
        echo "Un-packing the downloaded file..."
        tar -xvf ${folder}/${myfile} >/dev/null 2>&1
        cd - >/dev/null 2>&1
        echo "Removing downloaded file..."
        rm -f ${folder}/${myfile}
 
        # Run UNIX Health Check.
        if [ -x ${folder}/checkall.ksh ] ; then
                echo "Running UNIX Health Check..."
		${folder}/checkall.ksh -hdm ${emailaddress}
                echo "Removing folder ${folder}..."
                rm -rf ${folder}
        else
                echo "Encountered an error with checkall."
                echo "Removing folder ${folder}..."
                rm -rf ${folder}
                exit
        fi
else
        echo "Error downloading UNIX Health Check."
        echo "Removing folder ${folder}..."
        rm -rf ${folder}
        exit
fi
For Red Hat Enterprise Linux, CentOS, Oracle Linux and Scientific Linux:
#!/bin/bash
 
# VARIABLES
emailaddress="user@email.com"
password="mypassword"
folder="/uhc"
wget="/bin/wget"
myfile="rhc_latest.tar"
 
# Test if folder already exists.
if [ -d ${folder} ] ; then
        echo "Folder ${folder} already exists. Aborting."
        exit
fi
 
# Test if wget is executable.
if [ ! -x ${wget} ] ; then
        echo "Wget does not exist or is not executable. Aborting."
        exit
fi
 
# Create a folder to store UNIX Health Check:
echo "Creating folder ${folder}..."
mkdir -p ${folder}
 
# Download the file.
echo "Downloading UNIX Health Check..."
${wget} --post-data "emailaddress=$emailaddress&password=$password" \
   --referer=http://www.unixhealthcheck.com \
   http://www.unixhealthcheck.com/downloadauto.php \
   -O ${folder}/${myfile} >/dev/null 2>&1
 
# Un-pack the downloaded file.
if [ -s ${folder}/${myfile} ] ; then
        cd ${folder}
        echo "Un-packing the downloaded file..."
        tar -xvf ${folder}/${myfile} >/dev/null 2>&1
        cd - >/dev/null 2>&1
        echo "Removing downloaded file..."
        rm -f ${folder}/${myfile}
 
        # Run UNIX Health Check.
        if [ -x ${folder}/checkall.sh ] ; then
                echo "Running UNIX Health Check..."
		${folder}/checkall.sh -hdm ${emailaddress}
                echo "Removing folder ${folder}..."
                rm -rf ${folder}
        else
                echo "Encountered an error with checkall."
                echo "Removing folder ${folder}..."
                rm -rf ${folder}
                exit
        fi
else
        echo "Error downloading UNIX Health Check."
        echo "Removing folder ${folder}..."
        rm -rf ${folder}
        exit
fi
The script above can be used to download UNIX Health Check to your UNIX system in folder /uhc. Please make sure to provide the correct login/email address and password combination in the beginning of the script for variables email address and password, and if you wish to specify a different location for storing UNIX Health Check, you can change this in the script as well, by updating the folder variable.

The output of the script above, will look like this (assuming you have named the script "run-uhc"):
# ./run-uhc
Creating folder /uhc...
Downloading UNIX Health Check...
Un-packing the downloaded file...
Removing downloaded file...
Running UNIX Health Check...
Removing folder /uhc...

13. Enabling password-less access through SSH on cluster nodes
When running UNIX Health Check on cluster nodes, such as PowerHA/HACMP, it is important that password-less access for user root is available from one cluster node to another. If this is not set up correctly, some scripts may fail.

Here are the steps to perform on each cluster node:

First, make sure OpenSSH and OpenSSL are installed on the two clustered UNIX servers, serverA and serverB.

On each server, as user root, type:
# ssh-keygen -t rsa
This will prompt you for a secret passphrase. If this is your primary identity key, use an empty passphrase (which is not secure, but the easiest to work with). You will get two files called id_rsa and id_rsa.pub in the .ssh sub-folder in the root user's home directory.

Copy the id_rsa.pub file to the other host's .ssh sub-folder with the name authorized_keys. For example, on serverA:
# scp ~root/.ssh/id_rsa.pub serverB:~root/.ssh/authorized_keys
Now serverB should be ready to accept your SSH key from serverA. Please note that if the authorized_keys file already exists on serverB, that instead of copying over the file, you probably rather want to append the contents of id_rsa.pub from serverA to the authorized_keys file on serverB, to avoid overwriting any existing keys already present in the authorized_keys file.

To test, type:
# ssh serverB
This should allow you to log in without typing a password or passphrase.

14. Resolving email issues
When sending email from UNIX Health Check's master script (checkall.ksh or checkall.sh) using the -m option, you may run into issues while sending email.

Before you attempt to use the email feature of UNIX Health Check, please make sure to verify that you can send email from your UNIX system, by sending a test email as follows:
# cat /etc/hosts | mailx -v -s test email@domain.com
Please replace "email@domain.com" with a valid email address within your organization.

By using the -v option with the mailx command, you will see verbose output of the email you're sending. If mail does not arrive at the designated mailbox, check the mail queue on the UNIX system to determine if mail has been sent or not:
# mailq
If mail is not being sent, this is usually due to two common issues:
  • DNS is not properly configured
    For the mail function of your UNIX system, it is important that your system can resolve the fully qualified domain name of your system. Make sure that the short hostname, the fully qualified domain name, e.g. host.domain.com, and the IP address of your server can be resolved, by testing with the nslookup command.

    For example:
    # nslookup myserver
    Name:   myserver.domain.com
    Address: 10.43.242.82
    
    # nslookup myserver.domain.com
    Name:   myserver.domain.com
    Address: 10.43.242.82
    
    nslookup 10.43.242.82
    82.242.43.10.in-addr.arpa name = myserver.domain.com.
    
    If this does not work in the example above, please check with your network administrator to correct the DNS settings for your server.
  • Missing SMTP server entry in /etc/sendmail.cf or /etc/postfix/main.cf file
    Mail can also not be sent properly, if the name of the SMTP server is missing in /etc/sendmail.cf file. Look for an entry in this file that starts with "DS", for example:
    # "Smart" relay host (may be null)
    DSrelay.domain.com
    
    If no entry is present, discuss with your email administrator which SMTP relay host can be used to relay email from the UNIX system to your desired mailbox.

    On many Linux distributions, postfix is used for sending email, and the main configuration file for postfix can be found in /etc/postfix/main.cf, in which the target SMTP server should be defined using the relayhost entry, such as:
    relayhost = relay.domain.com
  • Issues on the SMTP server
    Issues on the SMTP server side may also prevent email from being received. An SMTP server may not allow email being sent from the client server, or there may be other issues on the SMTP server preventing email from being transmitted. Please consult with your SMTP server administrator if that is the case. If email remains in the mail queue (as seen by running the mailq command), especially when "read errors" are displayed when the mailq command is run, this may be an indication of issues on the SMTP server side.

15. Resolving issues discovered by UNIX Health Check
When reviewing reports generated by UNIX Health Check, you may see issues that need to be remediated.

Please take note of the warning in the report:

"Any individual implementing changes should completely understand the change and deem each change appropriate before it is applied to the system. As a standard rule, please take into consideration the impact on other components before implementing the change. Also, we encourage all to conduct a peer review of all changes before implementation. Most importantly, if the effect of a change is not fully understood, do not implement that change until a satisfactory explanation can be given as to what the effects of the change are. We recommend implementation of one change at a time. The application of many changes all at once will increase the likelihood of confusion, if issues arise."

It is important to follow change procedures properly while resolving any issues found by UNIX Health Check.

We recommend starting out by running UNIX Health Check on a single UNIX system first. And when an issue is found, attempt to resolve the issue not only on the one system, but if it is an issue that applies to multiple UNIX systems in your organization, apply the solution to multiple servers, obviously by following proper change procedures, and doing changes on non-production systems first, and when successful, followed by applying the same changes to the production systems. Repeat that for each issue found, until you've exhausted the issues to be remediated. Then move forward by running UNIX Health Check on the next UNIX system. You'll most likely find that many issues have already been resolved, by resolving issues and applying appropriate changes on all the UNIX systems, from running UNIX Health Check on the previous UNIX system. Quite often, UNIX administrators configure UNIX systems more or less the same way, or also, may make similar mistakes on multiple systems. As such it is good practice, when resolving an issue, to determine if the same issue is also present on other UNIX systems within your organization, and resolving them on all the UNIX systems. That helps to standardize your UNIX environment.