Analyze ESXi Logs for Security-Related Messages

In the course of writing about VCAP-DCA objectives, I’ve written previously about working with ESXi log files. In terms of security the main ESXi log files to be aware of, along with the hostd and vmkernel logs are:

•  /var/log/auth.log: ESXi Shell authentication success and failure attempts.
• /var/log/shell.log: ESXi Shell usage logs, including enable/disable and every command entered.
• /var/log/esxupdate.log: ESXi patch and update installation logs.

ESXi auth.log

The auth.log file records ESXi shell authentication success and failure attempts, which is useful in terms of security. The example entries below show a successful connection using the root account:

2012-02-01T22:01:38Z sshd[23752]: Connection from 10.0.0.2 port 57994
2012-02-01T22:01:41Z sshd[23753]: pam_per_user: create_subrequest_handle(): doing map lookup for user "root"
2012-02-01T22:01:41Z sshd[23753]: pam_per_user: create_subrequest_handle(): creating new subrequest (user="root", service="system-auth-generic")
2012-02-01T22:01:44Z sshd[23752]: Accepted keyboard-interactive/pam for root from 10.0.0.2 port 57994 ssh2
2012-02-01T22:01:44Z sshd[23752]: pam_per_user: create_subrequest_handle(): doing map lookup for user "root"
2012-02-01T22:01:44Z sshd[23752]: pam_per_user: create_subrequest_handle(): creating new subrequest (user="root", service="system-auth-generic")
2012-02-01T22:01:44Z sshd[23752]: pam_unix(system-auth-generic:session): session opened for user root by (uid=0)
2012-02-01T22:01:44Z sshd[23754]: Session opened for 'root' on /dev/char/pty/t1
2012-02-01T22:03:18Z sshd[12041]: pam_unix(system-auth-generic:session): session closed for user root
2012-02-01T22:03:18Z sshd[12041]: Session closed for 'root' on /dev/char/pty/t0

We can see in the output that this session was over SSH, from 10.0.0.2. The excerpt below shows an attempted, failed, login by the root account:

2012-12-18T19:36:27Z sshd[30399]: Connection from 192.168.88.135 port 35703
2012-12-18T19:36:40Z sshd[30426]: pam_per_user: create_subrequest_handle(): doing map lookup for user "root"
2012-12-18T19:36:40Z sshd[30426]: pam_per_user: create_subrequest_handle(): creating new subrequest (user="root", service="system-auth-generic")
2012-12-18T19:36:42Z sshd[30426]: pam_unix(system-auth-generic:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.88.135  user=root
2012-12-18T19:36:44Z sshd[30399]: error: PAM: Authentication failure for root from 192.168.88.135
2012-12-18T19:36:44Z sshd[30427]: pam_per_user: create_subrequest_handle(): doing map lookup for user "root"
2012-12-18T19:36:44Z sshd[30427]: pam_per_user: create_subrequest_handle(): creating new subrequest (user="root", service="system-auth-generic")

You can quickly find failed authentication attempts by running:

/var/log # cat auth.log | grep failure

This command will display all the ‘failure’ instances found in the log file.

ESXi shell.log

The shell.log file lists all the commands entered into the ESXi shell. For example:

2012-05-17T16:09:58Z shell[4236]: esxcli
2012-05-17T16:10:05Z shell[4236]: esxcli vm process list
2012-05-17T16:10:08Z shell[4236]: esxcli vm process
2012-05-17T16:10:55Z shell[4236]: esxcli system
2012-05-17T16:10:58Z shell[4236]: esxcli system time
2012-05-17T16:11:00Z shell[4236]: esxcli system time get
2012-05-17T16:11:09Z shell[4236]: esxcli system stats
2012-05-17T16:11:14Z shell[4236]: esxcli system stats uptime
2012-05-17T16:11:17Z shell[4236]: esxcli system stats uptime get

This is useful in tracking what has been run on a host, and from a security perspective is useful if unauthorized access is suspected.

ESXi esxupdate.log

The esxupdate.log file logs activity related to ESXi patching and upgrades, an excerpt is shown below:

2012-01-31T10:04:25Z esxupdate: BootBankInstaller.pyc: DEBUG: Using /bootbank/sata-sat.v03 from bootbank as source
2012-01-31T10:04:25Z esxupdate: BootBankInstaller.pyc: DEBUG: Total of 8941 bytes were written.
2012-01-31T10:04:25Z esxupdate: BootBankInstaller.pyc: DEBUG: About to write payload 'vmware-fdm' of VIB VMware_bootbank_vmware-fdm_5.0.0-455964 to '/altbootbank'
2012-01-31T10:04:25Z esxupdate: BootBankInstaller.pyc: DEBUG: Using source from LiveImageInstaller
2012-01-31T10:04:36Z esxupdate: BootBankInstaller.pyc: DEBUG: Total of 13578808 bytes were written.
2012-01-31T10:04:36Z esxupdate: BootBankInstaller.pyc: DEBUG: About to write payload 'ata-pata' of VIB VMware_bootbank_ata-pata-via_0.3.3-2vmw.500.0.0.469512 to '/altbootbank'

This log can be used to see what has been installed on the host, including patches and drivers.