Saved Searches¶
bh_stats_gen¶
The bh_stats_gen search is responsible for generating statistics about data coming into Splunk.
The results are written to the summary index, to be picked up and read by other searches for
alerting purposes. It can be fine-tuned using the bh_stats_gen_contraints and
bh_stats_gen_additions macros. By default, it will look over all data sources that have sent
logs over the past 36 hours. If you have a data source that is regularly delayed longer that you
wish to monitor, this time range will need to be adjusted.
Broken Hosts - Auto Sort¶
The Broken Hosts - Auto Sort and Broken Hosts - Auto Sort v5 searches were implemented in order to optimize
the ordering of the Broken Hosts Lookup. Because the lookup is evaluated in a first-match fashion, the ordering of the
lookup is critical to preventing incorrect matches. You can view more information about the
ordering of the lookup in the Saved Searches documentation.
This search modifies the Broken Hosts Lookup in the following ways:
Entries are reordered based on the ordering rules defined in the Saved Searches documentation.
All fields are converted to lower case, as the lookup is case insensitive.
Broken Hosts Alert Search¶
Broken Hosts Alert Search is the recommended way to get started building your own custom
alerting search. This search produces a single output row for each broken item, and ignores the
contact field from the lookup completely. There are no alert actions defined on this search,
so you are free to configure them as needed. A few examples include:
Add an email alert action to send a tuning report to your Splunk admins
Add a webhook alert action to create tickets in your ticketing system
You can also create clones of this search to enable different alerting for different types of data.
For example, you may want to send email notifications to your Windows server admins when a server
stops sending WinEventLog:Security but want to trigger a ticket to your helpdesk when your
anti-virus system stops sending logs. You can even run a version of this search on your
Enterprise Security search head to generate notable events.
Broken Hosts Alert - by contact¶
Broken Hosts Alert - by contact is primarily intended for anyone upgrading from an older
version of Broken Hosts. This search groups the alert lines by the contact field from the
lookup, and each contact will receive one email (the email action is configured by default on this
search). This search also relies on the default_contact macro to populate the contact when
none is defined in the lookup table.
If you’re coming from an older version of Broken Hosts and choose to implement this search, we’d
still recommend you review the new Broken Hosts Alert Search as you may find additional uses
from it that were difficult or impossible in previous versions of the app.
Broken Hosts Alert - Volume Alerting¶
Broken Hosts Alert - Volume Alerting and
Broken Hosts Alert - Volume Alerting with Seasonality are two example searches that can be used
to generate alerts on indexes that may have stopped ingesting data properly while still generating
some amount of logs. Both searches use a combination of standard score (z-score), moving averages,
and percentiles to determine whether or not log volume is anamalously low for that index.
Broken Hosts Alert - Volume Alerting with Seasonality additionally factors in the time of day,
day of the week, and whether the day is a holiday to determine normal logging activity for indexes
whose volume may be sensitive to user activity.
The macro bh_volume_alerting_indexes is used to designate which indexes should be alerted on.
If both Broken Hosts Alert - Volume Alerting and
Broken Hosts Alert - Volume Alerting with Seasonality are needed, a new macro can be created
and used to designate the indexes that should be used for each search.