Fine Tuning SentryOne Alerts

SentryOne can provide a wealth of information about your environment right out of the box when you accept the recommended default settings within the Setup Wizard. However, every environment is different and it’s likely that you will discover some opportunities for fine tuning of SentryOne notifications after your first week or two with the product. Below are some recommended practices to fine tune your notifications and reduce the number of unwanted alerts you may receive.

1. Global Message Throttling

Global Message Throttling is a feature designed to automatically throttle any message for the same condition and recipient that is sent x times within a given timeframe. This allows SentryOne to better handle those situations that have the potential to occur repeatedly in a short timeframe where one notification may be sufficient, as opposed to one every minute, etc.

The default behavior is SentryOne will not send more than 10 emails within 1 hour for the same object, condition, and alert target. You may want to further reduce this to perhaps 3 per hour. This setting can be changed by going to the Monitoring Service Settings node in the Navigator pane, and selecting the Message Processor tab.

To keep you informed, SentryOne also has the ability to alert you when messages are being throttled. This can be configured by enabling a Send Email action for the Message Processor: Message Action Throttled failsafe condition.

2. Blocking Source Settings

SentryOne Performance Advisor can detect and notify on any blocking SQL on a monitored server. The default behavior is to detect any block that lasts longer than 15 seconds. Additionally there is a Response Ruleset applied to the Send Email action for the Blocking SQL condition to send an email immediately after the block is detected, and then every 5 minutes while that same block exists.

For example, if you are receiving a high number of Blocking SQL emails, you may want to increase the Minimum Block Duration from the default of 15 to 30 seconds, and/or increase the Block Notification Response Ruleset to email after 30 seconds. Minimum Block Duration is a Blocking SQL Source setting, and can be adjusted globally for all servers, or per server, by selecting either the Shared Groups root or a SQL Server instance in the Navigator pane respectively, then go to the Settings tab. The Block Notification ruleset can be adjusted by selecting Tools > Manage Response Rulesets > select Block Notification > click Edit Ruleset > increase Process Actions After time.

3. Runtime Threshold Settings

SentryOne Event Manager can notify you when jobs and other objects exceed a Max Runtime Threshold. The global default is 250% of average duration, which has been determined to be an effective initial setting in most environments. Many environments, however, will still have some jobs with more volatile runtimes, such as transaction log backups. For these jobs, you may want to change the threshold. You have a couple of different options to accomplish this.

Job by Job basis

If you only have a handful of jobs you may wish to do this on a job by job basis. To do so, select the job in the calendar or Navigator pane, then go to the Settings tab. In the Runtime Thresholds section, set Inherit From Parent to False. This allows you to override the global settings for this particular job. Then, configure the appropriate threshold. In this case you’ll likely want to use an explicit threshold instead of a percentage of average.

Large number of jobs and Object Groups

For a large number of jobs, it is recommended that you use Object Groups. Object Groups were introduced in SQL Sentry v7 and allow you to easily manage these types of jobs. Object Groups allow you to configure policies (both Settings and Conditions/Actions) for a set of objects, which may not necessarily exist within the same sub-tree of the SentryOne Hierarchy. Object Groups can be configured through the Object Groups node found in the Navigator pane. You can add all of your transaction log backup jobs to a group quickly, searching either by name or category, and then apply changes to the Runtime Threshold settings for the entire group. For a complete explanation of Object Groups please see the Object Groups User Guide topic

4. Online/Offline Notifications and Transient Network Issues

Transient network issues can produce Failsafe notifications as the SentryOne Monitoring Service temporarily loses connectivity with a monitored server. If these are truly transient, the Monitoring service will recover automatically, and you may not necessarily want emails for these events. In this case, you can use a count based Response Ruleset to skip alerts on the first occurrence of these events. You should then only be alerted if the issue persists.

Failsafe alert configuration is available by selecting the Failsafe Actions tab on the right side of the SentryOne Client, or by going to the View > Failsafe Actions menu item. Note there are two types of rulesets, count based and time based. A count based ruleset is most appropriate for any Failsafe conditions that end in “Error”. Time based rulesets can be used with Offline conditions to wait until a server has been offline for X minutes before sending an alert. See the Response Ruleset Examples topic in the User Guide for more information. This same strategy can be employed for non-critical recurring jobs which sometimes fail once, then recover.

5. Failsafe Conditions and Maintenance Windows

If you perform regular nightly maintenance on your servers, the increased overhead can contribute to the same Failsafe notifications. For these regularly scheduled events, you can specify maintenance windows for the SentryOne database, for the Computers in your environment, or for the individual Connections you are monitoring. Failsafe notifications will be disabled during any Maintenance Windows you specify.

More details on configuring maintenance windows are available here.

6. SQL Server Agent Log Errors and Warnings

Monitoring the SQL Agent Error Log is critical to ensure SQL Agent is running properly and all scheduled activities on your SQL Server execute as expected. However, some error or warning entries may be less critical. A solution to filter the more common non-critical SQL Agent Log Errors is available here.

7. Alerting and Response System Windows

If you find that you need further control of when notifications are sent, you can also configure windows within SentryOne. Windows as they relate to notifications have a couple of different applications:

Ruleset Windows for Configured Actions

To control the time frame of when an action will be processed you can apply a window directly to a configured action. An action with a window applied will only be processed during that window's active time frame. For instance, if you created a window with a daily active time frame of 12:00 – 20:00, and then applied it to a configured send email action, you would only receive notifications about that condition during the hours of 12:00-20:00. If that condition occurred outside of the window’s active time frame the notification would be suppressed.

To apply a window directly to a configured action, first select the action in the Actions pane. Next open theRuleset tab for the selected action. Use the Window drop-down list to select your desired window. You may also choose to edit an existing window with the Edit button, or create a new window with the New button.

Email and Pager Windows for Users and Groups

You may assign email and pager windows to users and groups to filter notifications during a specified time. When you assign an email or pager window to a user or group, alerts will not be sent to that user or group during that window's time frame. For example, if you created a window with a daily active time frame of 16:00 – 23:00, and then set it as a user’s email window, that user would not receive any notifications during that window’s active time frame (the hours of 16:00-23:00).

To apply an email or pager window to a user or group, first double-click the user or group in the Navigator pane. This will open an Edit User/Group tab. Use the email or pager window drop-down list box to choose the desired window. -- Following the above steps can eliminate 95+% of unwanted alerts in most environments. Please see the SQL Sentry User Guide for more information on the Alerting and Response SystemResponse RulesetsObject GroupsWindows, and other topics mentioned in this article, contact Support for assistance, or leave a comment.

To quickly determine where most of your alerts are coming from so you can target your efforts, run this query against the SentryOne database, adjusting the date as needed:

SELECT

ParentObjectName

, ObjectName

, ConditionTypeName

, ActionTypeName

, COUNT(*) AS RecCt

, MIN(EventStartTime) AS FirstTime

, MAX(EventStartTime) AS LastTime

FROM vwObjectConditionActionHistory

WHERE MessageCreationTimestamp > '2011-01-01 00:00:00'

AND ActionTypeName IN ('Send Email', 'Send Page')

GROUP BY ObjectName

, ConditionTypeName

, ParentObjectName

, ActionTypeName

ORDER BY COUNT(*) DESC

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.