Event Log Blog

Event 4964: Special Groups Feature for Vista + Windows 2008 Entrepreneurs

2008-05-01T14:04:03Z

There is certainly a lot of talk about the benefits of using Vista, but a lot of administrators and users seem to be avoiding it and instead hold on to Windows XP - which now appears to have a better reputation than ever! Well, here is a small reason to upgrade to Vista or Windows Server 2008.

Microsoft introduced a new event, 4964, called the Special Groups Feature. The purpose of this feature is to log event 4964 to the security event log when a member of a group you specify logs on to a computer.

So let's say you want to know when a member of a local Administrator group logs on to a computer (and with EventSentry you could get an email when that happens for example), then you can accomplish that with the special groups feature.

In order to use this feature you need to do three things:

Determine the SID of the group(s) you want to monitor
Specify the SID(s) of the groups you want to monitor in a registry key
Ensure that you are auditing the Special Logon Feature (enabled by default)

One way to obtain the SID of a group is to use the getsid.exe tool which is part of the Windows XP SP2 Support Tools and other Microsoft Resource Kits. Note that the primary purpose of this tool is to compare the SID of two user accounts (so it requires you to specify two user/group accounts), but you can just enter the same group name twice to get around this. Here is an example output of the tool:

getsid \\mydc "Domain Admins" \\mydc "Domain Admins"

The SID for account BUILTIN\Domain Admins matches account BUILTIN\Domain Admins
The SID for account BUILTIN\Domain Admins is S-1-5-21-9817441204-4587651373-9817264971-512
The SID for account BUILTIN\Domain Admins is S-1-5-21-9817441204-4587651373-9817264971-512

As you can see you need to point to tool to computer where the group exists, in our case I used a domain controller since I want to monitor if somebody from the Domain Admins group logs on to the computer. If you monitor a built-in group (e.g. Administrators) then you will see that the SID is much shorter and the same across all your computers.

Now that we know the SID, we can specify it in the registry. Navigate to key HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Lsa\Audit and create a new String with the name SpecialGroups.

The value for this new string will be the SID of the group you want to monitor, and you can separate multiple SIDs with a semicolon. For example:

S-1-5-32-544;S-1-5-32-123-54-65

You do not have to reboot after making this change, it is effective immediately with the first subsequent login. The event that is being logged will look similar to this (screen shot from the EventSentry Web Reports):

The relevant information is shown in the lower part of the event in the New Logon section. Security ID shows the user that logged on, and Special Groups Assigned shows the group the account is a member of (of course this group has to be specified in the registry).

Voila. This feature probably makes most sense on critical servers, though I would recommend enabling it on all workstations as well since you probably want to know if a member of the local Administrators group logs on. But of course this also means that you need to be running Vista on your network :-).

Since this feature needs to be activated using the registry, you can use AutoAdministrator to push this registry change to multiple computers. AutoAdministrator has actually been rewritten from scratch and we will be releasing a new version 2.0 very soon.

Event Log Message Files (The description for Event ID ... cannot be found)

2008-04-13T19:00:00Z

Anybody who has used the built-in event viewer that comes with Windows more than once, has probably seen the message “The description for Event ID ( 50 ) in Source ( SomeService ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer.” when viewing certain events. This message occurs more often when viewing events on a remote event log, but it appears often enough on the local machine as well.

I will explain this dubious error message here, but before I do I will explain how messages are in fact logged to the event log. After reading this you should have a much clearer picture about how applications log to the event log and how you go about troubleshooting this “error”.

The framework that Microsoft created for the event log, back in the NT 3.51 days, was actually quite sophisticated in many ways – especially when compared with the more simplistic Syslog capabilities (though Syslog still has some unique features).

A key feature of event logging in Windows is the fact that an application, at least when using the event log framework in the way it was intended to be used, will never actually directly write the actual message to the event log – instead it will log only the event source and event id, along with some properties such as category and insertion strings. The framework also supports multiple languages, so if you open an event on a French Windows, then the event will display in French (of course assuming that the message file from the vendor supports that) instead of English.

Let’s look at an example – using EventSentry - to understand this better. When EventSentry detects a service status change, it will log the event 11000 to the event log that reads something like this:

The service Print Spooler (Spooler) changed its status from RUNNING to STOPPED.

When EventSentry logs this event to the event log, you would expect that the application does (in a simplified manner) something like this:

LogToEventLog(“EventSentry”, 101000, “The service Print Spooler (Spooler) changed its status from RUNNING to STOPPED.”);

However, this is NOT the case. The application logging to the event log never actually logs the message to the event log, instead the application would log something similar to this:

LogToEventLog(“EventSentry”, 101000, “RUNNING”, “STOPPED”);

(Note that the above example is for illustration purposes only, the actual code is somewhat more complicated)

So, our actual string from the event message is nowhere to be found, and that’s because the string is embedded in what is referred to as the “Event Message File”. The event message file contains a list of all events that an application could potentially log to the event log. Here is what an event message file looks like before it is compiled:

MessageId=10100
SymbolicName=EVENTSENTRY_SVC_STATUSCHANGE
Language=English
The status for service %1 (%2) changed from %3 to %4.
.
Language=German
Der Dienststatus von Dienst %1 (%2) aenderte sich von %3 auf %4.
.

Notice the numbers contained in the string that start with the percentage sign. These are placeholders for so-called insertion strings, and they make it possible to make the event log message dynamic, since an application developer can’t possible account for all imaginable error message or information that might be accumulated during the runtime of the application. For example, an application might log the name of a file that is being monitored to the event log, clearly this can’t be embedded into the event message file.

Instead, the application can insert strings (hence, insertion strings) into the event message during run time. Those strings are then stored in the actual event log, along with all the other static properties of event, such as the event id and the event source.

Event message files are usually DLL files, but event resources can also be embedded in executables – as is the case in EventSentry, where all events are contained in the eventsentry_svc.exe file. This is generally a good idea, since it reduces the number of files that have to be shipped with the software and it also prevents you from “losing” the message DLL.

You can browse through all embedded events in a message file by using the event message browser that is included in the free NTToolkit which you can download here. Simply launch the application, select an event log (e.g. Application), select an event source (e.g. EventSentry), and browse through all the registered event messages, sorted by the ID.

So now that we know how Windows handles event messages internally, we can go back to the original problem: “The description for Event ID ( 50 ) in Source ( SomeService ) cannot be found.”. The Windows Event Viewer logs this message for one of the following reasons:

    * No message file is registered for the source (e.g. SomeService)
    * The registered message file does not exist or cannot be accessed
    * The specified event id is not included in the message file

If the message file is not registered, then this is probably because the application wasn’t installed correctly, or because it has already been uninstalled by the time you are trying to view the event message. For example, if the event message was logged before the application was uninstalled, but you are viewing the event after the application was uninstalled, then you will see this message.

If the event you are trying to view is important, then you can try to fix the problem yourself by either fixing the registry entry or locating the missing event message file.

The registry location depends on only two factors: The event log [EVENTLOG] the event was logged to as well as the event source [EVENTSOURCE].

HKLM\System\CurrentControlSet\Services\Eventlog\[EVENTLOG]\[EVENTSOURCE]

(Replace [EVENTLOG] and [EVENTSOURCE] with the respective values, and view/add/edit the value EventMessageFile. This is the value that points to the message file)

If this value doesn’t exist, then you can add it as either a REG_SZ or a REG_EXPAND_SZ value. You can specify multiple message files with a semicolon.

If the message file specified in the value doesn’t exist, then you can simply copy it into the appropriate location – assuming you can get a hold of it that is :-). Oracle is notorious for not including the message file, in particular with the Express Edition.

A final note on message files for those of you haven’t had enough yet: You can use message files not only to translate event messages, but also for categories, GUIDs and more. Some of the values you might find (mostly in the security event log) are CategoryMessageFile, GuidMessageFile and ParameterMessageFile.

Well, this article turned out a lot longer than I had anticipated, but hopefully you will have a better understanding as to why this message is logged and what you can do about it.

Showing Server Uptime with uptime.exe

2008-04-01T17:50:20Z

It's been almost 15 years since Microsoft released the first NT-based operating system, Windows NT 3.1, on July 27th 1993. So it came as a bit of a surprise to me that not even the brand-new Windows 2008 ships with an easy way to show the current uptime of the OS.

Linux/Unix users are probably quite familiar with the convenient uptime command, which shows how long the OS has been running and also includes a current load average.

Windows still doesn't ship with such a tool (I will refrain from posting sarcastic assumptions as to why they might not want to do that) which makes it difficult for any SysAdmin to quickly determine how long a machine has been up and running. One can of course dig through the System Event Log to find the 6009 event or create a script, but I'd hardly call that convenient.

That's why, a while back , we developed the free uptime.exe application which is included in our free NTToolkit. Simply run uptime.exe and it will show you the uptime of the system you are logged in as, and keep counting until you abort with CTRL+C:

Uptime: 11 days, 4 hours, 33 minutes, 4 seconds

Uptime.exe also accepts the /onetime parameter which just displays the current uptime and returns, and you can also display the uptime in seconds with the /secs command line switch. This might be useful if you want to use uptime.exe in batch files for example.

You can download uptime.exe from http://www.netikus.net/products_downloads.html, and if you choose the version without the installer then you don't even have to log in. The setup version of the NTToolkit allows you to extract the MSI however, which you could automatically deploy to all of your servers. You could then take advantage of all the tools in the NTToolkit without having to download or install anything.

The upcoming 2.90 release of EventSentry will also be able to track the uptime of all monitored servers, so that you can easily view and compare the uptime of one or more servers through our web reporting interface.

Are you looking for a small tool that would make your life as a SysAdmin easier? Just send an email to suggestions {{AT}} netikus [[DOT]] net.

Operational Event 567? Maybe sometimes.

2008-03-18T02:54:38Z

In my previous post I explained when and why 560 events are logged, and roughly explained why they are only of limited usefulness since they only log what a user could have done, not what they actually did.

Starting with Windows XP and Windows Server 2003, Microsoft introduced the so-called operational event 567. This event is supposed to enhance the auditing experience by not only logging what a user could do, but instead what he actually did! Does this sound too good to be true? Well, I guess that's because it almost is.

According to Eric Fitzgerald's Object Access Auditing Overview (Eric is the former head of the Windows Auditing Team), the 567 event should be logged in between the 560 (open handle) and 562 (close handle) events.

As such, the idea behind the 567 is simple:

1. User opens text file backup.cmd with a text editor. Event 560 is logged which includes all the rights the user will have to the document backup.cmd.

2. The user hits the save button, and Windows logs event 567, most likely including the WRITE_DATA access mask which indicates that data was written to the file. Note that the 567 event will not, unlike the 560 event, include the file name in the message text, but instead just the value of the handle that was included in the previously logged 560 event. As such, to make use of that event, you will have to go back to the 560 event to figure out which file was affected by the subsequently logged 567 event.

3. When the user closes the file, event 562 is logged. This event also only includes the value of the handle which was returned by the previously logged 560 event.

Well, you can imagine that I got pretty excited after doing all the research and was ready to see that 567 event in action on our production and test network. So I enabled auditing on a folder and started creating files, modifying files and so forth. Events 560 and 562 were logged just fine and as expected, but I had difficulties seeing a 567 event - it just wasn't there. Since there is no option to turn event 567 on or off, I wasn't exactly sure what I was doing wrong. I was ready to give up - after all I was quite tired that night, but after playing around more, trying different operating systems, different auditing options, logging on locally etc. I finally saw the 567 event. Hooray!

All seemed well until the next morning, when I tried to continue last night's work and got stuck again. No 567 event. I then remember Eric's blog entry, where he pointed out that event 567, due to a bug, wasn't logged when files were accessed through a file share unless you had WinXP SP2 or Win2k3 SP1. Surely this couldn't be a problem in my network, since I was running SP2 on XP and Windows Server 2003. Well, it was the best hint I had to work with, and so I compared local and remote file access auditing to see if it would make a difference.

Bingo!

As it turned out, event 567 was only logged when I accessed files locally, that is if the file that was audited resided on the same machine that I had logged on to. As soon as I accessed a file through a file share (as most people do), event 567 was not logged.

I was still confused though, after all I had read about the 567 event not only in Microsoft's documentation and blog, but also at other trustworthy sources and still thought that maybe something was off in my environment. Maybe I was missing a hotfix or some other secret ingredient that would prevent my server from generating the highly desired 567 event.

So expanded my tests to another test network, another production network, a SBS 2003 network and so on and so on. The results were always the same, event 567 was not logged when I accessed the audited file through a file share.

Since I ran out of options and the existence vs. non-existence of the 567 affected development of a new EventSentry feature, I opened up a support call with Microsoft's Enterprise Support. After an hour of mostly hold time and an actually helpful engineer, it turns out that event 567 is indeed only logged sometimes. The engineer didn't want to be too specific, but the bottom line was that one should not except the 567 event to always be logged when a 560/562 event pair was logged. Asking why that was the case, I was told that implementing event 567 "correctly" would have required a kernel change which was not an option. So there you have it.

I stick to my "research" however - event 567 is indeed logged as long as you are accessing the audited files locally, and not through a network share. Otherwise you will have make do with the 560/562 events.

Of course you can also upgrade to Vista or Windows Server 2008, which log event 4663 (= 567 + 4096) regardless of whether you access the file locally or remotely. This event also includes the full filename and path, so collecting the related 4656 and 4658 events is not necessary. I have verified it with both, and it works very well indeed.

Thankfully, EventSentry 2.90 (when released) will take most of that burden of you and perform some additional work for you to give a crisp idea of who is modifying/creating/deleting which file at what time.

Enjoy!

Tracking Objects with 560 and 562 Object Access events

2008-03-15T02:29:45Z

One of the upcoming features in the 2.90 release of EventSentry is file object tracking, which will - as the name implies - track file access!

EventSentry already tracks process activity by intercepting and analyzing the 592 and 593 security events that are generated when a process starts or exits respectively; we also track logons and logoffs by intercepting and analyzing the various logon (e.g. 528) and logoff events. Tracking object access turns out to be a bit more involved as process and logon tracking, since Windows 2003 and earlier don't actually log when an object is modified, but instead log when an object handle is being returned to the caller. I would like to mention here that object auditing has been drastically improved in Vista and later, but more on that next week.

But before I explain the 560, 562 and the problematic 567 events, let's make sure we have everything setup for auditing to work.

1. Make sure that "Audit Object Access" is active on the machine where the files will be accessed. In most cases this will be your file server, and you will probably want to configure this with a group policy object and apply this setting to all machines from which you plan on collecting object audit events.

2. Once auditing is enabled on the machine, you will have to tell Windows which files you effectively want to audit, since generating an audit event for every single file by default would fill up your security event log quicker than you could get a cup of coffee. To audit a folder, bring up the security properties of the folder, click advanced and select the "Auditing" tab. Here you will specify which accesses and users will be audited, and I recommend that you always use Everyone when adding an audit entry to ensure that all object access is audited. I also recommend only auditing the access type you really care about. since 560 events can quickly fill up your event log (and consequently any consolidated database you might have) and there is no reason to monitor accesses you're not concerned with (e.g. ReadAttributes).

Now to get back to the 560 and 562 events, this is better explained with an example. In Windows, when you need to read or write to a file, you usually call the CreateFile() API function which will return a handle to the object (=file in this case) you are about to access. When calling CreateFile(), you tell Windows which access to the file you need. For example, when you simply need to read from a file then you can pass GENERIC_READ (or the more specific FILE_READ_DATA) for the dwDesiredAccess parameter.

Assuming that you are allowed READ access to the file, Windows will return a handle to the requested file (that you can now use in subsequent ReadFile() operations). And this is exactly where Windows logs the 560 Audit Success event (assuming of course the access type and user match the auditing enries), essentially documenting that an object handle was returned. While this all sounds nice and dandy, the problem with the 560 event is that it doesn't actually tell you what the caller ended up doing with that handle. Even if the caller where to close the handle right away with CloseHandle(), the 560 event would have still been logged - even if the caller never actually accessed the file.

The same holds true for potential write access to a file. If I access a file with the GENERIC_WRITE access right, then Windows will log a 560 event that looks similar to this:

Object Open:
Object Server: Security
Object Type: File
Object Name: E:\Folder\Customers\Sheet.xls
Handle ID: 20084
Operation ID: {0,93244500}
Process ID: 4
Image File Name:
Primary User Name: DC1$
Primary Domain: ESDOMAIN
Primary Logon ID: (0x0,0x3E7)
Client User Name: support.engineer
Client Domain: ESDOMAIN
Client Logon ID: (0x0,0x58C5419)
Accesses: READ_CONTROL
ReadData (or ListDirectory)
WriteData (or AddFile)
AppendData (or AddSubdirectory or CreatePipeInstance)
ReadEA
WriteEA
ReadAttributes
WriteAttributes

At first glance, one would assume that support.engineer wrote to the file, after all WriteData is included in the listed Accesses. This is far from accurate however, since the user could have closed the file right-away again (without ever reading or writing data from/to it) and the event would have still been logged in exactly the same manner.

This means that unless you manually verify some properties of the file, for example the access stamps, size or checksum, the 560 events only tell you what a user could have done, not what they actually did.

When the calling process is done working with the file, it will call CloseHandle() to close the handle it had previously opened. As such, a 560 event is always followed by a 562 event that includes the same handle ID as the original 560 event.

At some point during the Windows XP development, Microsoft seems to have realized that the 560 events are limited in their usefulness (at least for authorized access), and introduced the 567 event, also called an "operational event". The purpose of the 567 event is not to log when a handle is returned, but instead when a file is actually being accessed - much more useful - at least in theory. So even though the 567 event was created to solve the problems of the 560 event, it does so only under limited circumstances.

But since I already wrote more on this subject than most people probably want to read, I will explain the 567 event in all detail in my next post this weekend.

Server Virtualization: Physical to Virtual Migration

2008-02-17T18:53:05Z

Despite being around for quite some time now (VMWare released its first version of VMWare Workstation in 2001), server virtualization seems to be all the rage these days. It seems that not a day goes by without seeing virtualization mentioned in a newsletter, magazine or web site.

The first virtualization software product I used was indeed VMWare Workstation, some time around 2002. I used it mainly for testing and development, and the fact that it supported most major operating systems (e.g. Linux) was very helpful. A couple of years later we got our first dedicated server where we ran the VMWare GSX Server (now called VMWare Server). At that time I also evaluated Microsoft Virtual Server, but found it to be inferior from both a management and performance aspect, and it also didn't support Non-Microsoft Operating Systems. Microsoft Virtual Server has improved since then however and is a viable alternative today.

As virtualization has become more and more important for our business, we have since migrated to the VMWare ESX Server, which supplies its own Linux-based operating system and thus offers a slightly better performance among other benefits. I know this is starting to sound a lot like a VMWare advertisement, so I'll try to get to the point. Don't get me wrong, we've definitely found problems in VMWare's products over the years, but its proven to be a stable and reliable platform overall. For example, installing and upgrading ESX Server has thus far not caused us any problems or difficulties. Read Michael's review of VMWare Server 2 however, he doesn't appear to have liked the latest release of VMWare Server too much.

One thing that is often overlooked in my opinion however is the ability to migrate physical machines to virtual machines using the free VMWare Converter. Using the converter, you can migrate a physical into a virtual machine, which can run on any of VMWare's products. How many IT departments are running servers that are barely used anymore, yet cannot be turned off because a handful of users are occasionally accessing the server to access old data? Those machines usually don't require fast hardware and might even be running on systems that are no longer supported by the manufacturer. Systems like these are ideal candidates for virtualization.

Here are just some of the benefits you get by moving a legacy or underutilized machine to a virtual server:

If you retire (=recycle) the original hardware, you save money on power by requiring less A/C and power consumption in your data center.
You can cancel any maintenance agreements on the hardware if it is retired.
You might speed up the application if the server hosting the virtual machines is more powerful than the original box the software was running on.
If the migration fails or causes unexpected problems, then you have nothing to fear since the original server won't be modified.
The migration is done remotely, so you don't even have to physically log on to the computer being migrated.
Virtual machines can be suspended, thus saving RAM on the host machine while suspended.

We performed a similar migration a couple of months ago when we switched to a new support ticketing system. Since we didn't migrate any data from the previous system to our new system, we wanted to have the ability to login and search tickets periodically - so shutting of and formatting the server was not an option. Of course, keeping the server running 24/7 seemed like waste as well - especially when we wouldn't need to access the machine more than once a week. Hence, a migration to a virtual machines seemed like the best option and the server lives on ESX Server since. The physical server was initially just turned off for a few weeks, but has since found new use for a different project.

So if you're planning to move to virtualization or have already begun, don't just think about new machines but also consider "virtualizing" existing physical machines.

And remember that machines running inside VMWare or Microsoft Virtual Server can be monitored by EventSentry just like a physical machine can. :-)

1983: Coleco Adam

2008-02-03T04:10:50Z

If you're past 30 then you've probably heard about the Commodore C64, the Amiga, the IBM XT and so forth. Well, another lesser known computer that was released around the same time IBM released their "IBM Personal Computer XT" was the Coleco Adam.

Why is this funny?

To find out why this is beyond funny, you will have to read the Wikipedia article about the Coleco Adam, in particular the "Problems" section. We found the 1st and 3rd problem most amusing.

The tale of the dying capacitors

2008-01-18T16:37:36Z

We were recently helping out a company in the same building as ours with a server issue they were having. They noticed it had rebooted out of nowhere a couple times within a week. None of the event logs showed anything, no crash dump file, pretty much no trace software wise. Luckily EventSentry sent us a 6009 event from the system log, letting us know that the server had rebooted. Knowing when this event occurs is great, especially on nights or weekends when users may not notice.

I was 99% sure it was a hardware issue since it was out of the blue with no recent hardware or software changes. We ran some basic diagnostics, including the ones from Dell. Everything kept coming back clean. After contacting Dell, they recommended re-seating all the RAM, the CPU's, VRM's etc. They have had problems in the past with CPU's coming out of the sockets from the heatsink compound drying up and causing the same issue. I should have instantly noticed the problem then, but we will get to that.

None of this was helping and the reboots were becoming more and more frequent. The server was not under warranty so Dell couldn't help out much more than that. I was actually amazed how helpful they were at all since it wasn't. Their last suggestion was to start disabling hardware until we got to the root of the problem.

I went into device manager and disabled anything I could. The system became much more stable, although useless without the devices. I started enabling hardware again one at a time. After I enabled the built in NIC, the computer crashed. We threw in a PCI network card, and disabled the onboard NIC in the BIOS. The server booted up and all was great. For about 3 days...

The crashes started again, this time Windows couldn't even finish loading before it would reboot. We opened the server again and this time I instantly saw what was wrong. I had seen this in a workstation before so I couldn't believe I missed it. Almost all the capacitors on the board were bulging at the top.

This has become so common lately, I highly recommend looking for that right away on any critical server you have. There were even a few motherboard makers sued over this.

Some makers, like Gigabyte, are using solid state capacitors instead of the cheaper, more common electrolytic ones for some of their boards. I'm sure it costs them a little more, but for reliability I think it is completely worth it.

We ordered a new motherboard for the server, and sure enough it had a completely different brand of capacitors. Once we swapped it out and booted it up, the server has been running smooth. An extra $5 for some quality capacitors would have probably prevented the whole situation.

Here are some pictures of what to look for:

Taken from http://img.photobucket.com/albums/v711/whurd/Bad.jpg

The tops should be completely flat. If there is any bulging at all, it is most likely on its way out. The picture below shows leaking capacitors, also not a good thing.

Taken from http://macmedics.com/images/imac-logicaboard-with-leaking-capacitors.jpg

So, next time one of your servers starts acting up out of the blue, without any recent hardware or software changes, take a close look at those capacitors :-)

Vista/Win2k8 Event Log Changes #2: .evtx Format

2007-12-21T18:54:17Z

In my previous post I already mentioned that Vista and Windows Server 2008 introduced many changes to the Windows Event Log, and the event log backup files with the familiar .evt extension are no exception. If you backup event logs in the .evt format and plan on moving to Vista and/or Windows 2008 then you should make yourself familiar with the basic changes and the "new" EVTX format.

The new event viewer on Vista and Win2k8 supports exporting an event log in either the EVTX, XML, TXT or CSV format. If you select the EVTX format then you will not be able to import/load this file on a Pre-Vista/Win2k8 machine, the old event viewer does not understand the new EVTX format.

So far so good, this is to be expected. Like I mentioned in my previous post, Vista and later also still provide the legacy event log APIs so that applications that were developed for Windows 2003 and earlier are still able to access and backup the event log. The next paragraphs get a bit confusing, so only read on if you are interested in more details ;-).

Windows 2003 and earlier provide two API calls to backup and/or clear the event log: ClearEventLog() and BackupEventLog(). If you use any of these functions to backup an event log on Vista and later, then you are still able to create a .evt file. I would expect that this file could be opened on any computer that understands the EVT format, however this is not the case. Even when you export an event log using the aforementioned legacy API calls, the resulting file can still only be opened with the new event viewer on Vista or later. I will refer to event log backup files that were created on Vista and later with the legacy API calls as the new EVT format from now on.

This becomes more clear when you compare the contents of the new EVT format with the EVTX format. While the two files are different for the exact same event log backup - the overall structure are quite similar. You can also rename a file with the new EVT format to the EVTX extension and the new event viewer will open this file correctly. The format of an EVT file on the other hand is very different to that of an EVTX file.

So the bottom line is that you can, in theory, create three types of event log backup files:

1. EVT Format
These files are created on Windows Server 2003 and earlier. Vista and later refer to these files as "Classic Event Log Files", and you can open and read EVT files on any NT-based OS including Vista and later.

2. EVT Format (when created on Vista and later)
These files can only be created on Vista and later by using the legacy API calls ClearEventLog() and BackupEventLog(). It is important to point out that even though these files have the .evt extension, they unfortunately cannot be read on Windows Server 2003 or earlier and the format of this file is similar to the new EVTX format.

3. EVTX Format
These files can only be created and viewed on Vista and later.

Note on EventSentry: If you are backing up event logs with EventSentry v2.72, v2.80 or v2.81 on Vista or Windows 2008, then EventSentry will create EVT files (#2) that can only be viewed on Vista or later. We are switching to the native EVTX format for event log backups with the upcoming v2.90 release of EventSentry.

Plink - or - Issuing SSH Commands on Demand

2007-12-16T19:59:55Z

We have a Linux server running Samba on our network which we use mostly to store ISO images which can be mounted and served on-demand through Samba.

I was looking for a way to issue commands on the Linux machine through SSH yesterday when the Winbind daemon (which is part of Samba and ensures that Linux users are authenticated against our domain controller) on the machine was acting up again. Every time we reboot our Windows 2003 domain controller (which is fortunately not very often but security updates usually require this), the Winbind daemon starts logging a particular error message every 5 minutes to the Syslog daemon which in turn is forwarded to EventSentry by the Linux Syslog daemon.

Since warnings and errors are forwarded to me via email, getting this particular error message every 5 minutes starts getting old after about half an hour - especially when I'm out of the office and get them on my phone. Logging on to the Linux box and restarting the Winbind daemon however solves the problem - and this is what I have been doing for a long time now. Well, until recently.

I thought to myself that if there were a utility that could issue commands through SSH from a Windows box, then I could configure EventSentry to automatically restart the Winbind daemon as soon as the Syslog packet containing the error message is received.

I have been using the free SSH-Client PuTTY for quite some time now, but didn't know that it "included" Plink, a SSH utility that allows you to issue commands through the SSH tunnel and even see the output from the remote command. Perfect!

Setting up EventSentry to automatically restart windbind using plink is a straight-forward 3-step process, assuming you already have the Syslog Daemon in EventSentry up and running:

1. Create a batch file that issues the command you need to run. The batch file I created looks like this:

C:\Batch\plink.exe root@mylinuxhost -pw SecretPass "/etc/init.d/winbind restart"

Make sure you run the script once from the command-line to ensure that it is working.

2. In EventSentry, create a process action that references the above script. You do this by right-clicking the Actions container and selecting Add Action. Then just select the Process tab and point to the batch file you just created.

3. Under the Event Log Packages container, add a filter in an existing package or create a new package. The filter will match the Syslog event that you want to trigger our script. The event source for that filter will always be Application, and the event id should be 9999. Since we don't want the process to be triggered every time a Syslog event comes in, we will also specify the text from the Syslog event - *winbindd*: cli_nt_setup_creds: request challenge failed* in my case. Then just select the process action you created in step 2 and you are all set.

There are a couple of things I need to point out of course. First, make sure that the batch file is secure as it contains the username and password to your Linux host - the appropriate NTFS permission might be enough in most cases. If you cannot keep it secure then you should create a user on the Linux box that is just used for the purpose of issuing particular commands through SSH. Second, make sure that plink.exe is present on the host where the EventSentry Syslog daemon is running, as the file will be executed on that host.

Plink of course is a great utility for automation in any case, regardless of whether you use EventSentry to consolidate Syslog messages. I hope this helps automate some tasks in Windows/Linux environments.

Vista Event Log Changes

2007-12-10T19:59:00Z

As you may already know, Microsoft significantly changed the Windows event log in Windows Vista. I always found the Windows event log to be a very well designed logging infrastructure, at least compared to the logging facilities that are available in other major network operating systems. The Windows event log however hasn't changed much since it was originally included in Windows NT. It has actually been 13 1/2 years since the core event log service and event viewer underwent a major improvement - other than updating security event ids to accommodate new events related to various security components - Windows NT 3.1 was first released in 1993. I have never actually seen the event viewer in Windows NT 3.1, but Windows NT 3.51's event viewer for example was not too much different than Windows 2003's.

So it appears that Microsoft finally realized that a good system can be improved (especially with compliance becoming more and more important over the last years), and so the event log subsystem, including the event viewer, appear to have been rewritten completely in Windows Vista and of course the upcoming Windows Server 2008.

As we are continuing to improve Vista support in EventSentry, I will cover the changes that I believe are relevant to IT professionals that need to manage their event logs. Just as a side note, EventSentry already monitors the Vista event log since the end of 2006, however ES 2.81 currently accesses the Vista event log through the legacy API that Vista (fortunately) still provides to pre-Vista event log software.

Before I dig into the technical details about the new event log, I need to point out that Microsoft made a large amount of changes to the event log, and didn't leave a stone unturned. While the overall logic is the same (you have event logs and events :-) ), a lot has changed under the hood.

While this will affect IT professionals that need to manage event logs (since they need to make sure that their software works with Vista & Windows 2008), it will affect software developers even more. While I like a lot of the changes that were introduced, and we all know improvements were overdue for a long time, I personally feel that the new event log has been over-engineered. A lot of the features that were added are a bit of overkill and accessing, especially writing to, the event log is significantly more involved with the new version (at least if you take advantage of the new XML functionality). I think it would have been better to gradually introduce improvements over the last 10 years, rather than ignoring the event log for a long time and then introduce a myriad of new functionality to it - some of which has yet to make sense (I will prove my point below with future posts).

In any case, I will get off my soapbox now and focus on the relevant changes that were introduced.

Keywords
One of the new fields added to the properties of an event is called "Keywords". I find the most interesting thing about this field that security events now have their severity stored in the Keywords field instead of the Type field (Type was renamed to Level in Vista and later). As you know, events in Windows Server 2003 and earlier used to have their severity stored in the Type property of an event (Information, Warning, Error, Audit Success, Audit Failure), but in Vista and later the severity of security events (Audit Success, Audit Failure) have been moved to the new Keywords field.

This of course leaves the question what the Level is set to for audit events. Well, the answer is Information. All Audit Success and Audit Failure events have their main severity stored in the Keywords field, whereas the Level field is always set to Information. An Audit Failure event that is informational, yeah - that makes a lot of sense!

So in theory it would be possible to have an Audit Failure event logged with a level of Information/Warning/Error, but I am not sure how useful this would be. After all, an Audit Failure is an Audit Failure.

Why was this changed? I am not sure. After asking the head of the Windows Auditing Team at Microsoft I received an explanation that, unfortunately, failed to eliminate my confusion. The original Type field could obviously accommodate the two attributes (since had always been there), and there would have been room for even more. There was some consensus between the two of us that the keywords field, at least in combination with the security events, was maybe not implemented in the best way.

In EventSentry we currently ignore the Keywords field and merge it with the original Type field, so that you can search across Pre-Vista machines and Vista machines using the same field name.

So this is it for now, we will cover a lot more about the new Windows Event Log here in the future. As always, let us know if you have any questions or feedback.

Who Is In My Server Room?

2007-11-13T14:00:00Z

As some of you already know, EventSentry allows you to use different environment sensors to be alerted about changes in your server room. One of these happens to be a motion sensor.

It is great to be alerted when somebody is moving around in there, but it would also be helpful to know who it is. We picked up an Axis 207 network enabled camera from Axis Communications so we can take a peak in there though any available web browser. This works great as long as we are near a computer at the time we get the motion alerts from EventSentry, but not very useful if we aren't.

Luckily, our Axis camera has a pretty good API that you can access. It has the ability to grab a .jpg image by going to a URL (http://cameraIP/jpg/image.jpg). I needed a way to attach this .jpg to an email so that not only am I alerted, but I also have an image of who or what caused it.

There may be other cameras out there that can do this as well. If you know of one please post it in the comments section.

I came up with a batch file that uses some free utilities to accomplish this task. For good measure, I also decided to allow you to grab a series of pictures, put them to a web site directory, thumbnail them, and finally create an HTML page that displays them.

This could probably have been done easier using Perl or another scripting language, but I had already started with a batch file and wanted to just finish it! Feel free to come up with a better way.

The tools needed are included in this zip file:

gethttp.exe - Taken from our free NTToolkit, used to grab the image from the camera
sleep.exe - Also taken from NTToolkit. Allows you to put pauses in your script
blat.exe - Blat is a great command line utility that allows you to send emails
printf.exe - Taken from the GNU tools for Windows. A lot more flexibility than using ECHO
convert.exe - Command line utility from ImageMagick. Used to create the thumbnails.

The zip file also contains the actual script used named "getimages.cmd". You will need to change some of the settings inside of it to get started. Most are self-explanatory and include:

cameraIP - IP address of the camera
binPath - Path to the needed utilities above
imagePath - Where you want the images stored
numImages - The number of images you want to capture each time
timePause - Miliseconds to wait between images
netLocation - URL to your web server hosting the images
eMail - Email address you want the alerts sent to. Comma separate for multiple people.
eSender - Address email comes from
subj - The subject for the email
server - Your SMTP server

Now to make it run when EventSentry detects motion. To do this, create a new action in EventSentry. I named mine "Motion Alert". Go to the "Process" tab at the top and put in the path to the "getimage.cmd".

Next, we will need an event filter to trigger the action. Here are the settings you need:

Event Log: Application
Type: Error
Source: EventSentry
Category: Environment Sensors
Event ID: 10912

That is it, from now on you should know who is setting off your motion sensor.

You can download the entire package from here.

If you have any comments or suggestions, we would love to hear them.

Automatically shutting down workstations

2007-11-08T19:59:34Z

There has been a lot of talk lately about "green" computing, it seems like every other IT magazine covered this topic over the last few months.

Since power isn't free, companies are fortunately interested in saving power for the sake of saving money, if not for the environment as well.

A lot of attention has been given to the power consumption of data centers and how to save money with blade servers and newer processors, but I feel that not enough attention is being paid to workstations, of which there are many more than servers.

Now there are a lot of power-save options available for workstations and software to centrally manage those settings, but how many users are not shutting down their workstations when they leave for the day?

I don't have the resources to calculate how much power is being wasted every day by computers that are running over night across the US - when they really don't have to be - but I imagine it's a good amount.

True, there are some advantages to keeping computers running over night. For example, you might have software and patches pushed to workstations after hours (and we can get around that as well) and it's always nice to come into the office in the morning without having to wait for the computer to boot up.

If you're in charge of managing workstations then I recommend that you take a look to see how many workstations are running after most of the employees have left for the day. If you have a lot of idle PCs sitting on the floor sucking power then please read on. ;-)

Using third-party tools we can automatically schedule a workstation shutdown using the shutdown.exe utility that ships with Windows XP and Vista. The tool is flexible enough to give users a grace period where they can abort the shutdown. And if you have a software management solution in place that pushes updates to your workstations, then you should be able to adjust the schedule so that the shutdown occurs after the deployment of any packages you might have. You can also limit the shutdown to particular weekdays, for example Mon-Thu only.

If you are using EventSentry to monitor not only your servers but also workstations, then you are lucky and setting this sort of scheme up shouldn't take longer than 5 minutes.

EventSentry includes the Application Scheduler feature, which allows you to schedule tasks (similar to the task scheduler in Windows) on one or more computers. Simply create a new system health package, add an application scheduler object and create a new schedule. The command line you want to run can be similar to this:

shutdown.exe -s -f -c "To conserve power, this computer will now be shut down. To abort, click START - RUN and enter SHUTDOWN -A" -t 300

This will give the user 5 minutes before the computer will actually be shutdown. Then, assign the package to your workstations and you are set to go.

If you are not using EventSentry then you can also write a batch script and schedule the script to run from the server as well, the shutdown.exe tool supports shutting down remote computers as well.

If you are still scared about shutting computers down, then take a look at the WakeOnLan feature from the EventSentry Web Reports Hardware Inventory. If the web reports are installed on the same collision domain than your workstations, then you can wake up computers (that support the WakeOnLan feature) with the click of a button from there. Or, you can use wakeonlan.exe from the free NTToolkit to accomplish the same thing from the command-line. You could even write a batch script to wake computers up every morning!

I hope this gives you some ideas on how you can save power, save money and conserve the environment with little effort.

P.S.: Don't forget to tell your fellow coworkers that you will be shutting down their computers at night so that those hard-working individuals won't be caught by surprise!

Setting Service permissions with subinacl.exe

2007-11-02T20:49:22Z

I recently stumbled across a lesser known Microsoft utility (again) called subinacl.exe that you should take a look at if you haven't already done so. It can be downloaded for free from Microsoft.

The tool is incredibly versatile and lets you change permissions of various system objects, such as files, printers, shares, services, registry keys and more from the command line.

I came across it because we needed a way to change the permission of the EventSentry service to allow a particular user account to read the current service status. So I'm only going to cover the service aspect of the tool in this post.

So how is this useful? Imagine you have a junior admin that you want to allow to manage a particular service on one or more of your servers. You don't want the guy to be a local admin or be able to control all services but instead only be able to control one (or more) particular service.

In this case Windows doesn't actually offer any native way of doing this without using a third party tool - with the exception of using group policy.

So let's say you have user "Johnny" and you want Johnny to be able to stop and start the World Wide Web Publishing service. Simply run the following subinacl.exe command:

subinacl /service W3SVC /GRANT=YOURDOMAIN\Johnny=TO

Obviously you will want to replace YOURDOMAIN with the name of your domain. The TO at the end are the identifiers that tell subinacl which actions you actually want grant to Johnny. T is used for "Start Service" and O is for "Stop Service". The complete list is here:

   F : Full Control
   R : Generic Read
   W : Generic Write
   X : Generic eXecute
   L : Read controL
   Q : Query Service Configuration
   S : Query Service Status
   E : Enumerate Dependent Services
   C : Service Change Configuration
   T : Start Service
   O : Stop Service
   P : Pause/Continue Service
   I : Interrogate Service 
   U : Service User-Defined Control Commands

So after running the command, Johnny will be able to stop and start the service without having any other permissions on the system.

But don't stop there. Run subinacl.exe /help to see all the other options that are available to you. Of course you can also run the tool remotely by specifying the remote computer name.

You should also check out the MS KB article 288129 that has information on how to accomplish the same thing with group policies and security templates. This might be a better way especially if you have a large number of servers you want to apply this to.

Hope this is useful!

Inauguration

2007-11-02T19:56:05Z

When Ryan (one of our web developers) first talked about starting an official company blog, I wasn't too amused by the idea. Don't get me wrong, I thought that having a blog would be great! But the thought of creating yet more content on a regular basis just didn't seem to appealing - after all I keep myself pretty busy with lots of other things in relation with the EventSentry project, not even to mention my involvement with all the other content we have on our web sites (knowledge base, documentation, www.myeventlog.com, etc.).

Well, this was many months ago. I'm not sure how and why, but somehow I slowly started liking the idea of starting a blog where we could share ideas, useful tips & tricks, news about our product and personal thoughts. So after dismissing the original idea it got to a point where I was actually begging Ryan to setup the blog on our web site after I reserved the www.eventlogblog.com domain this week. I even have topics lined up already in my ToDo list!

So here it is: The Event Log Blog.

So, stop by and let us know what you think - suggestions for topics and comments are - as always - very welcome.