Splunk has a powerful alerting system that can be used to send emails or run arbitrary commands whenever a specific condition is met, as defined in a Splunk search.
In this walkthrough we demonstrate how to alert when the duration of user logons exceeds a certain threshold.
Defining the Condition
An easy way to find the right Splunk search for defining the alert condition is to use an element already present on one of uberAgent's dashboards. In this example we are interested in logon duration which is displayed on the User Logon Duration dashboard. Navigate to the table near the bottom and click the button with the i on it:
This opens the search job inspector in another window. Scroll to the bottom and copy the search that was run to populate the data table:
Paste the search command into a regular Splunk search box. At this stage you can add a condition (highlighted below) to filter for specific events:
Setting up the Alert
Once you are happy with the results returned from your conditional search copy it.
Click Settings > Searches, reports and alerts:
Configure the alert as shown in the following very large screenshot:
Configure EMail Alerting
To configure alerting via email perform the following from the alert configuration screen shown above.
Edit the email action:
Specify email addresses. It might also be a good idea to check Inline so that you get the actual data causing the alert right in the email:
Back on the alert configuration screen navigate to the email setup:
The following mail server settings apply to Google Apps:
The Resulting Alerts
With that you are all done. With the sample configuration you will receive an email like the following whenever a user logon takes more than 10 seconds:
You can also see all alerts in Activity > Triggered Alerts: