If there’s one thing that’s apparent when supporting vSphere environments, its that it’s never good news when a datastore has ran out of space. This article will look at how you can configure datastore alarms. Datastores can run out of space due to a number of reasons, the most common ones being snapshots and thin provisioning. The use of thin provisioning increases the possibility of oversubscribing storage at the datastore level.
There are a number of potential issues that can occur when a datastore is low on or out of space. These include virtual machine startup failure, vmotion failures, snapshot creation and commit failures and overall poor performance. This is where datastore alarms are extremely useful:
Datastore usage on disk alarms can be used to monitor the percentage of disk space usewd on each datastore in your vSphere environment. The default settings of the Datastore usage on disk alarm is to trigger a Warning when disk usage exceeds 75% and an Alert at 85%. These thresholds can be adjusted to suit the needs of your environment, although it is good practice to begin taking action as your arrays approach 80% utilized. This may vary depending on the size of your datastores and how they are being used.
We can see how the triggers are configured by viewing the alarm’s settings:
Along with the built in datastore alarms, custom alarms can be configured for datastores and datastore clusters.
Creating a Custom Datastore Alarm
The steps for creating a custom datastore alarms using the vSphere client are as follows:
- Go to the Datastores and Datastore Clusters view.
- Right-click the datastore to be monitored then click Alarm > Add Alarm. This will bring up the alarm’s settings screen.
- On the General tab enter a name for the alarm. For the Alarm Type, ensure that Datastore is selected and that Monitor for specific conditions or state and Enable this alarm are selected.
- On the Triggers tab, choose to trigger an alert if any or all of the conditions are met. Click Add. The Datastore Disk Usage Trigger will populate. Enter the desired settings for both Warnings and Alerts.
- On the Reporting tab, enter the desired settings for the Range and Frequency of the triggered alarm.
- On the Actions tab, choose to Send a notification email, Send a notification trap and/or Run a command. Once configuration is complete, click ok to save the custom alarm.
Working with and Analysing Datastore Alarms
The datastore usage alarm will send notifications when a datastore has exceeded it’s configured thresholds. To test this you can create a custom datastore usage alarm and set the thresholds to be along the lines of Warning when Datastore Disk Usage exceeds 10% and an Alert at 15%. On the datastore where I configured the alarm these thresholds were enough to generate a notification.
Where can you see these alerts?
Within the vSphere client you can see the alerts in a couple of places. The first is the vCenter Events screen, access from the Home screen:
The other is in the Datastores and Datastore Clusters view. We should see a red exclaimation mark over any datastore that has triggered an alert. Selecting the datastore and then the alarms tab will give detail about which alarm has triggered the alert or warning:
When you right click on a triggered alarm you get the option to clear the alarm or acknowledge it. If it is cleared then the alert will disappear, but will alert again depending on what its set in the alarm settings. For example, the default datastore usage alarm will trigger again if the thresholds are still exceeded after 5 minutes. Acknowledging the alarm won’t make it disappear, but it will become greyed out. This feature is to show that the alarm/fault is being worked on by an administrator. Clearing or acknowledging an alarm is treated as an event and is recorded in the vCenter events screen.
What can you do if a datastore is running out of space?
The most common remediation steps include:
- Increasing capacity by adding extents or extending the size of the LUN and VMFS volume.
- Use svMotion to move virtual machines to free up space
- Add additional datastores to migrate virtual machines to
- Delete unneeded snapshots on virtual machines on the datastore in question.
PowerCLI and Datastore Alarms
You can also list and interact with datastore alarms using PowerCLI. For example, to list the alarms in use for a particular datastore you can run:
get-datastore datastorename | get-alarmdefinition | select name,description | ft -auto
