Recently in EventSentry Category

Even though Microsoft Exchange Server 2010 has already been released, many organizations still use Exchange 2003. In this article I'll explain how to thoroughly monitor the various Internet protocols that Exchange 2003 offers, including SMTP, POP3, IMAP (and NNTP for that matter). The reason why I'll only be looking at Exchange 2003 is because there is a significant difference in architecture between Exchange 2003 and later versions.

It is a common misconception that you can effectively monitor the W3SVC service (commonly referred to as IIS, though IIS encompasses a lot more than just a web server) and other services provided through IIS, such as SMTP and POP3, by simply monitoring their associated service. It's a misconception, because a given IIS-based service may contain multiple instances - most commonly the case with the World Wide Web Service which often hosts multiple independent web sites. The status of these instances can be controlled independently of the hosting service, though that service needs to be running of course.

Don't despair though, most server-based windows applications, fortunately, can be monitored by ensuring that their respective service is - well - running. For example, to ensure that the Apache service is up, you "simply" make sure that the Apache service is running. The same goes for countless other services such as MySQL - even SQL Server (of course you can still detach individual databases in SQL Server).

Exchange 2003, due its partnership with the Internet Information Services 6.0, is different though. Yes, IIS and Exchange 2003 are tightly coupled, and if you intend to have your Exchange Server 2003 communicate with any other server using a standard Internet protocol such as SMTP, then you will need IIS.

The screenshot above shows that the inetinfo.exe process hosts all the major services (bold name), and that each service can host one or more instance. For more details please see http://technet.microsoft.com/en-us/library/bb124674(EXCHG.65).aspx.

The three most common Internet services your Exchange 2003 server is running are probably SMTP, POP3 and IMAP4. While a lot of attention is being paid to the core Exchange services such as

  • Microsoft Exchange Information Store (MSExchangeIS)
  • Microsoft Exchange System Attendant (MSExchangeSA)

The services providing SMTP, POP3 and IMAP4 connectivity are usually similarly important, especially the SMTP service. Looking at the EventSentry service status page immediately reveals that the SMTP, POP3 and IMAP4 services are managed by IIS:

As you can see, IMAP4Svc, POP3Svc and SMTPSvc all use inetinfo.exe (Executable column) for their host process. So why is this important again?

Since all of these services support multiple instances INSIDE the service (inetinfo.exe), the host process will continue to run even when one or more instances inside the service are stopped. Since most installations only have one instance, stopping that one instance inside the service will still leave the service up and running. The effect of course is the same; the service is not available to the end users while the Windows service will happily continue to run.

A screen shot from the System Manager application shows instances listed inside:

As you can see with the IMAP4 protocol, we have two virtual servers setup that are both hosted inside the "Microsoft Exchange IMAP4" service. To stubbornly illustrate my point further I took a screenshot that shows both IMAP4 instances stopped while the service itself is running:

So I think we're all in agreement now that monitoring the POP3, SMTP etc. services in Exchange 2003 is not enough if you want to ensure that these services are actually available. So how do we monitor all of these instances?
The easiest way is actually with a VBScript, which is included below. VBScript works well since the cscript.exe interpreter is readily installed on Windows 2003, so no additional installation of tools is required. The script enumerates all instances of a given protocol, and checks whether they are running or not. If at least one instance is not running, the tool will return 1, thus setting the ERRORLEVEL to 1.

This VBScript can then be embedded into EventSentry, which will then run the script at set intervals using the application scheduler, notifying you via email (with the proper filter setup) when an instance is stopped. There's a screencast for that, you can view it at http://www.eventsentry.com/screencasts/eventsentry-application-scheduler/eventsentry-application-scheduler.htm. It shows you how to create an embedded script and setup EventSentry to notify you when the scripts returns an error. Note that the screencast uses an older version of the script which only monitored web sites (not SMTP, IMAP4, ...), but the process of setting up the script with EventSentry is exactly the same.

You should be able to use the script as-is, just configure which protocols are monitored by adjusting the values in the "Define which protocols to monitor here" section. The script always prints all installed instances and their status, and any stopped instance is prefixed with an asterisk. Below is what an email from EventSentry looks like:

The line with the stopped instance won't be yellow in the actual email, I just added this for readability. The script can also easily be modified to automatically start any stopped instances - simply add the line

Instance.Start

after line 102. This will still trigger an email (or error) to notify you that it was stopped, but a subsequent run of the script at the next monitoring interval should not trigger an error again if the start was successful.

A note of caution here though - I have seen the script hang indefinitely with this line added when an instance that is currently stopped can't be started because it's not configured correctly. Hence, it's not included by default.

Now that EventSentry v2.91 has been released, I'm happy to have the opportunity to blog about our monitoring solution again.

The most significant new feature in EventSentry is the Health Matrix, a new way to see your network status in a space-efficient way. In fact, you can see the overall health status of your entire network on a single screen, even if it consists of hundreds of hosts.

We also made numerous other changes throughout the web reports, and added some exciting new filtering capabilities with our event log filters, as well as improved speed with the event log engine and file checksum generations.

EventSentry v2.91 also includes many minor improvements throughout the application, including service monitoring, process tracking and more. We have also updated EventSentry Light, and a new version will be released in the coming days after we have completed testing.

But now to the new features in version 2.91:

Health Matrix
In the health matrix, each host is displayed as a colored square, circle or rectangle, with the color indicating the overall health of the monitored computer. When all of the monitored components of a host are in an OK status, the color of the square is green. The color will change to orange or red when a problem is detected, depending on the number or severity of the issue.

The health matrix is highly customizable, for example both the size and shape of the icons can be adjusted depending on the size of the network (and your monitor).

Event Log Monitoring
In 2.91, the event log filtering engine was improved, resulting in reduced CPU usage of the event log monitoring component. Since the CPU usage of the EventSentry agent is already quite low, you will most likely only notice this improvement on hosts that generate an extremely large number of events, such as domain controllers.

Also new is the ability to filter events based on insertion strings in addition to just filtering based on the event message text. This means that one can now match individual strings inside event messages against strings, numbers, file checksums and group memberships. If you are not familiar with the term "insertion string", then I highly recommend my previous post about event message files before you read on.

Consider the following hypothetical example: The environment-monitoring component of EventSentry logs event id 10908:

The temperature (78.21 degrees F) has fallen outside the configured range (60F to 76F).

which is defined as:

The temperature (%3 degrees %4) has fallen outside the configured range (%1%4 to %2%4).

This event obviously informs us, that the current temperature has exceeded a set limit. Now let's say that we wanted to get an email when the temperature exceeds the limit, but also send a page when the temperature exceeds 90 degrees.

The new filtering feature allows you to do just that, by using the numerical comparison functionality with insertion strings (of course you would also need to set the hour/day properties). Assuming that you already have a filter in place for regular email notifications, you would simply setup an additional include filter that would evaluate insertion string 3 (%3) and only match if the number is above 90. See the screen shot below for the example. The result is a filter that only matches when then the temperature exceeds 90 degrees.

2.91 also includes two more comparison options, file checksums and group membership. So, if an insertion string represents a filename (e.g. from a security event), then EventSentry can create a SHA checksum from the specified file and compare it with the value that you specified. Another example would be a security event that includes a username in an insertion string, in which case you could setup a filter that would only match if that user is a member of particular group you specify. Both examples are mostly applicable for security events, since those are most likely to contain either filenames or usernames.
Using file checksums, you can be notified whenever a user plays solitaire, even when the user renames the executable.

Simply create a checksum of the file first using shachecksum.exe (included in the free NTToolkit, make sure you account for different OS versions and platforms) and intercept the corresponding 4688 event.

Service Monitoring
Service Monitoring now collects the username as well as the executable of a service. These additional properties are available in the web reports and in events generated, for example when the username of a service changes.

Software Monitoring
Software monitoring has been overhauled in 2.91, and some limitations and bugs have been removed. On Vista, Win2k8 and later, Windows patches are now monitored and included in the software inventory. 64-bit software is now classified as such and searchable, and searching for installed Windows Updated patches has also been simplified.

SNMP Traps
EventSentry can now send version 2c and version 3 traps, previously only version 1 traps were sent by the agent. The SNMP trap daemon was originally set to be released as part of 2.91, but this feature has been pushed back to v2.92.

Web Reporting
We have made a number of improvements in the web reporting to make using our web-based interface easier:

•    Reports are now easily accessible from every page, in addition to the reports page.
•    The database usage page now shows the actual page name in addition to the table name.
•    The dashboard page has been overhauled
•    The network status page can be customized (performance counters & disks)

Miscellaneous Improvements

There have of course been other improvements across the board, such as:

•    Notes can now be applied to computers
•    AD-linked groups can be sorted, and authentication properties can be set globally
•    Hardware monitoring now includes the IP address of an interface
•    Process tracking can capture the command line of a process
•    Logon tracking includes group information
•    File checksum generation has been optimized and will now use fewer CPU resources (affects file monitoring and file access tracking)
•    The minimum database interval for environment monitoring has been reduced to 5 minutes from 15 minutes
•    Software uninstallation events now include the same information as software installation events

Database servers store massive amounts of data, often including sensitive information. It is not uncommon for there to be databases holding millions of rows of data, where a small subset of rows are considered critical or sensitive. This could be anything from a Social Security number to an EventSentry entry of a security event. Being notified when existing data in your database changes is crucial for log data, and can be accomplished by using triggers with Microsoft SQL Server.

For those of you not familiar with triggers, a database trigger executes code in response to events on a table or database. Triggers are essentially hooks into a table, and they usually execute SQL statements as a response to another SQL statement.

Since we love the windows event log, we'll take advantage of SQL Server's ability for triggers to log an event to the event log when a row in a table is modified. This allows us to not only log that activity, but also get notified immediately when suspicious or important activity occurs in the EventSentry database.

In EventSentry, we have a table named ESEventlogMain that stores Windows event information. This table constantly gets new data inserted into it, and it often gets purged as well to manage the size of the database. However, there is no reason this data should ever be modified. If it is, then we know that something is amiss and we want to trigger an event in the event log. It is also useful to know what account made that change.

The first step is to create the message in SQL. You can use this SQL statement to create it:

     sp_addmessage 80000, 10, 'Data Integrity Alert: %s', @with_log = TRUE

The first argument is a unique SQL server message ID that should be 50001 or higher, you can delete it again using sp_dropmessage. The number 10 is the severity level, but you can read more about the different options for sp_addmessage here.

Now we create the trigger that will use this message:

CREATE TRIGGER Trigger_ESEventlogMain_Modified ON
ESEventlogMain
FOR UPDATE
AS

IF UPDATE(eventmessage) OR UPDATE(eventid) OR UPDATE(eventtime) OR UPDATE(eventcomputer)
BEGIN

     DECLARE @Msg VARCHAR(8000)
     DECLARE @EventNumber INT
     DECLARE @EventID INT
     DECLARE @Computer VARCHAR(255)
     DECLARE @EventMessageOld VARCHAR(8000)
     DECLARE @EventMessageNew VARCHAR(8000)

     SET @EventNumber = (SELECT eventnumber from deleted)
     SET @EventID = (SELECT eventid from deleted)
     SET @Computer = (SELECT A.eventcomputer from ESEventlogComputer as A, deleted as B WHERE A.id = B.eventcomputer)
     SET @EventMessageOld = (SELECT eventmessage from deleted)
     SET @EventMessageNew = (SELECT eventmessage from inserted)

     SET @Msg = 'ESEventlogMain modified by ' + CONVERT(VARCHAR(20), USER_NAME(USER_ID())) + ' at ' + CONVERT(VARCHAR(20), GETDATE()) + '. Computer: ' + @Computer + ', Event ID: ' + CONVERT(VARCHAR(8), @EventID) + ', Event Number: ' + CONVERT(VARCHAR(16), @EventNumber) + ', EventMessage (old) =' + @EventMessageOld + ', EventMessage (new) = ' + @EventMessageNew

     RAISERROR( 80000, 10, 1, @Msg)
END

This creates a trigger which will generate an event when the eventmessage column in the ESEventlogMain table is modified. You can remove the "IF UPDATE(eventmessage) ..." clause (as well as the BEGIN & END statements) if you want to be notified of any changes to that table, this might however create some noise since acknowledging events will also perform an UPDATE on this table.

FYI: "deleted" and "inserted" are keywords that refer to either the old record that was updated (=deleted) or the new data (=inserted).

As you can see from the screen shot above, the message text from a logoff event was renamed to "Trigger Test". So now that the event is in the event log, we can set up a filter in EventSentry to alert us: