Event Log Blog

Windows Server has long provided admins the ability to create a software RAID, enabling redundant disks without a (potentially expensive) hardware RAID controller. If you are already using Windows Server 2008's software RAID capabilities, and think that Windows will somehow notify you when a disk in an array fails, then you can skip to "Just say something!" below.

Background
Creating a RAID can all be done from Disk Management view in the Computer Management console, without any scripting or command-line tools.

Software RAIDs are not as powerful and fast as their hardware counterparts, but are nevertheless a good way to enable disk redundancy. Software RAIDs make sense in a variety of scenarios:

•    When you are on a budget and don't want to spend a few hundred $$ on a hardware RAID controller
•    When you need to enable redundancy on a server that wasn't originally designed with redundancy in mind (as if that would ever happen!)
•    When you need to add redundancy to a server without reinstalling the OS or restoring from backup

Windows Server lets you do all this, and it's included with the OS - so why not take advantage of it? The last point is often overlooked I think - you can literally just add a hard disk to any non-redundant Windows server and create a mirror - with less than dozen clicks!

Since this article is starting to sound like a software raid promotion, and for the sake of completeness, I am listing SOME of the advantages of a hardware RAID here as well:

•    Faster performance due to dedicated hardware, including cache
•    More RAID levels than most software RAIDs
•    Hot-plug replacement of failed disks
•    Ability to select a hot spare disk
•    Better monitoring capabilities (though this article will alleviate this somewhat)

But despite being far from perfect, software RAIDs do have their time and place.

Just Say Something Please!
Unfortunately, despite all the positive things about software RAID, there is a major pitfall on Windows 2008: The OS will not tell you when the RAID has failed. If the RAID is in a degraded state (usually because a hard disk is dead) then you will not know unless you navigate to the Disk Management view. The event logs are quiet, there are no notifications (e.g. tray), and even WMI is silent as a grave. Nothing. Nada. Nix.

What's peculiar is that this is a step back from Windows 2003, where RAID problems were actually logged to the System event log with the dmboot and dmio event sources. What gives?

Even though a discussion on why that is (or is not) seems justified, I will focus on the solution instead.

The Solution
Fortunately, there is a way to be notified when a RAID is "broken", thanks in part to the diskpart.exe tool (which is part of Windows) and EventSentry. With a few small steps we'll be able to log an event to the event log when a drive in a software RAID fails, and send an alert via email or other notification methods.

Diskpart is pretty much the command-line interface to the Disk Management MMC snap-in, which allows you to everything the MMC snap-in does - and much more! One of the things you can do with the tool is to review the status of all (logical) drives. Since we're interested as to whether a particular RAID-enabled logical drive is "Healthy", we'll be looking at logical drives.

So how can we turn diskpart's output into an email (or other) alert? Simple: We use EventSentry's application scheduler to run diskpart.exe on a regular basis (and since the tool doesn't stress the system it can be run as often as every minute) and generate an alert. The sequence looks like this:

•    EventSentry runs our VBScript (which in turn runs diskpart) and captures the output
•    When a problem is detected, EventSentry logs an error event 10200 to the application event log, including output from step 1.
•    An event log filter looks for a 10200 error event, possibly looking at the event message as well (for custom routing).

Diskpart
Diskpart's output is pretty straightforward. If you just run diskpart and execute the "list volume" command, you will see output similar to this:

Volume ###  Ltr  Label        Fs     Type        Size     Status     Info   
----------  ---  -----------  -----  ----------  -------  ---------  --------
Volume 0         System Rese  NTFS   Simple       100 MB  Healthy    System  
Volume 1     C                NTFS   Mirror       141 GB  Healthy    Boot    
Volume 2     D   System Rese  NTFS   Simple       100 MB  Healthy           











Notice the "Status" column, which indicates that our "BOOT" volume is feeling dandy right now.  However, when a disk fails, the status is updated and reads "Failed Rd" instead:

Volume ###  Ltr  Label        Fs     Type        Size     Status     Info   
----------  ---  -----------  -----  ----------  -------  ---------  --------
Volume 0         System Rese  NTFS   Simple       100 MB  Healthy    System  
Volume 1     C                NTFS   Mirror       141 GB  Failed Rd  Boot    
Volume 2     D   System Rese  NTFS   Simple       100 MB  Healthy           

Technically, scripting diskpart is a bit cumbersome, as the creators of the tool want you to specify any commands to pass to diskpart in a text file, and in turn specify that text file with the /s parameter. This makes sense, since diskpart can automate partitioning, which can certainly result in a dozen or so commands.

For our purposes however it's overkill, so we can trick diskpart by running a single command:

echo list volume | diskpart

which will yield the same results as above, without the need of an "instruction" file.

The easy way out
The quickest way (though per usual not the most elegant) to get RAID notifications is to create a batch file (e.g. list_raid.cmd) with the content shown earlier

echo list volume | diskpart

and execute the script on a regular basis (e.g. every minute) which will result in the output of the diskpart command being logged to the event log as event 10200.

Then, you can create an include filter in an event log package, which will look for the following string:

*DISKPART*Status*Failed Rd*

If your EventSentry configuration is already setup to email you all new errors then you don't even have to setup an event log filter - just creating the script and scheduling it will do the trick.

But surely you will want to know how this can be accomplished in a more elegant fashion? Yes? Excellent, here it is.

A Better Solution
One problem with the "easy way out" is that it will not detect all Non-Ok RAID statuses, such as:

•    At Risk
•    Rebuild

Furthermore, the output can be rather verbose, and will include any logical drive, include CD-ROMs, removable disks and others.

It is for this reason we have created a VBScript, which will parse the output of the diskpart command with a regular expression, and provide the following:

•    A filtered output, showing only drives in a software raid
•    Formatted output, showing only relevant drive parameters
•    Detecting any Non-Healthy RAID

Alas, an example output of the script is as follows:

Status of Mirror C: (Boot) is "Healthy"

Much nicer, isn't it? If a problem is detected, then output will be more verbose:

**WARNING** Status of Mirror C: (Boot) is "Failed Rd"
   
WARNING: One or more redundant drives are not in a "Healthy" state!

The VBScript will look at the actual "Status" column and report any status that is not "Healthy", a more accurate way to verify the status of the RAID.

Since the script has a dynamic ERRORLEVEL, it's not necessary to evaluate the script output - simply evaluating the return code is sufficient.

Implementation
Let's leave the theory behind us and implement the solution, which requires only three steps:

1.    Create an embedded script (we will include this script with v2.93 by default) through the Tools -> Embedded Scripts option, based on the VBScript below. Select "cscript.exe" as the interpreter. Embedded scripts are elegant because they are automatically included in the EventSentry configuration - no need to manage the scripts outside EventSentry.

2.    Create a new System Health package and add the "Application Scheduler" object to it. Alternatively you can also add the Application Scheduler object to an existing system health package. Either way, schedule the script with a recurring schedule.

Note that commands starting with the @ symbol are embedded scripts. The "Log application return code 0 to event log ..." option is not selected here, since the script runs every minute and would generate 1440 entries per day. You may want to enable this option first to ensure that your configuration is working, or if you don't mind having that many entries in your application log. It's mainly a matter of preference.

3.    This step is optional if you already have a filter in place which forwards Errors to a notification. Otherwise, create an event log filter which looks for the following properties:

  Log: Application
  Severity: Error
  Source: EventSentry
  ID: 10200
  Text (optional): "WARNING: One or more redundant drives*"

The VBScript

Most of the time I work on a Lenovo laptop running Windows 7, and I'm overall quite happy with the laptop (especially after the mainboard was replaced and it stopped randomly rebooting). However, a minor nuance had been bugging me for a while: If I plugged my computer into a LAN (I have a docking station at work and at home) while a wireless signal was also available (and configured on the laptop), Windows 7 would keep both connections active.


1. The Problem

So I'd have my laptop in the docking station, connected to a 1Gb Ethernet network, and yet the laptop would also be connected to a WiFi network. While not a big deal per se, it does have a few advantages to automatically disable the WiFi connection when already connected to Ethernet:

• Avoid potential connectivity issues
• Increase security by not transmitting data via Wifi when not necessary
• Increase battery life when connected to a LAN
• Because you can!
  

Now, Lenovo equips most (if not all) laptops with a software called "Access Connections", a pretty nifty and free tool! One of the things it can do, is disable a Wireless adapter when the computer is connected to Ethernet. However, it never re-enables it when you disconnect from the wired network (at least I haven't found a way), and besides not everybody has Lenovo laptop.

So how could I disable the WiFi connection automatically when I connected the laptop to an Ethernet, yet automatically re-enable it when there is no Ethernet connection?


2. The Research

After some intense brainstorming, I remembered two things:

1. Most Ethernet NIC drivers log event to the System event log when a network port is connected/disconnected.
2. A while back, I used the netsh command to configure DNS servers from the command line. Maybe one could toggle the state of network adapters with this tool as well?

3. Evidence Gathering

The first one was easy, a quick look at the system event log revealed the following event:

A similar event is logged when the "network link" has connected. The event shown here is specific to the driver of my laptop's network card (an Intel(R) 82577LM adapter), but most newer drivers will log events when a cable is disconnected or the link is otherwise lost. If you are already running EventSentry with its hardware inventory feature enabled, then you can obtain the name of the network adapter from any monitored host on the network through the hardware inventory page, an example is shown below.

Coming up with a way to enable and disable a particular network connection with netsh.exe was a bit more challenging, but I eventually cracked the cryptic command line parameters of netsh.exe.

Enable a connection
netsh interface set interface "Wireless Network Connection" ENABLED

Disable a connection
netsh interface set interface "Wireless Network Connection" DISABLED

And yes, you do need to specify the word "interface" twice. If you do find yourself wanting to automate network adapter settings with scripts and/or the command line frequently, then you should check out this link.


4. The Solution

So now that we have all the ingredients, let's take a look at the recipe. In order to accomplish the automatic interface toggling, we need to create:

• an embedded script called wifi_enable.cmd, using the command line from above
• an embedded script called wifi_disable.cmd, again using the command line from above
• a process action "Wifi Enable", referencing the above wifi_enable.cmd embedded script
• a process action "Wifi Disable", referencing the above wifi_disable.cmd embedded script
• an event log filter for event source "e1kexpress" and event id 27, triggering the "Wifi Enable" action
• an event log filter for event source "e1kexpress" and event id 32, triggering the "Wifi Disable" action
  

A couple of clarifications: First, you do not need to use embedded scripts, you can create the scripts in the file system too and then point the process action to those files. I prefer embedded scripts since I don't have to worry about maintaining the script, as it gets distributed to remote hosts automatically when needed. Second, the event source and event id will depend on the network card you have installed on your network, the above example will only work with Lenovo T410 laptops.

So what happens is pretty straightforward: When I connect my laptop to a LAN, the Intel NIC driver writes event id 32 with the event source e1kexpress to the system event log. EventSentry intercepts the event and triggers the Wifi Disable action, which in turns runs the netsh.exe process, disabling the WiFi connection.


5. Setting it up in the management console

Embedded Scripts
You can manage embedded scripts via Tools -> Embedded Scripts. Click "New", specify a descriptive name (e.g. wifi_enable.cmd) and paste the command line netsh interface set interface "Wireless Network Connection" ENABLED into the script content window. Then, do the same for the wifi_disable.cmd script, but this time use the netsh interface set interface "Wireless Network Connection" DISABLED command line. You can leave the interpreter empty as long as you give the filename the .cmd extension.

Actions

I recommend enabling both "Event Log Options", as this will help with troubleshooting. Now we just need the event log filters, and we are all set.

• Process Action
  
• Event Log Filters
  
• Embedded scripts
  

Passwords are everywhere. You use them to log on to your network, login to business applications and Facebook, check your personal email, and more.

I'll be rethinking passwords in this blog, and what you can do to make authenticating with passwords more secure.

As it turns out, the British comedian Nick Helm won an award in Edinburgh for the funniest joke, just one day before I posted this article. He won for the joke: "I needed a password eight characters long so I picked Snow White and the Seven Dwarves."

Of course, passwords have been around for a while, even though more advanced ways to authenticate like fingerprint readers and biometric scans exist today. Still, passwords prevail as the primary method to authenticate for the majority of networks and computer systems. One-time pads like RSA's SecurID are another secure alternative, but any system can be exploited as recent events have shown. Even fingerprint readers can be fooled: either with brainpower or through more "traditional" methods.

Password Cracking goes Mainstream
By 1999, Windows NT 4.0 started gaining a lot of traction in networks across the globe. Of course we all know that with popularity comes quantity, and with quantity comes increased exposure. As more and more networks were using Windows NT (to authenticate among other things), a new piece of software called l0phtcrack (it had a GUI!) was gaining popularity. What l0phtcrack could do - and quite easily I might add - was download all password hashes from the Windows NT user database, and then run both brute-force and dictionary attacks on those hashes.

If a user chose a password that was in the English dictionary, l0phtcrack could often crack it within seconds. If the password was a bit more complicated, it would take a couple of days. Due to the way Windows NT stored password hashes (in the name of compatibility with LanManager), passwords with 7 or fewer characters were particularly easy to crack. And, as CPUs grew stronger and faster, the time required to run those brute-force attacks kept getting shorter. Of course this general mechanism is and was not restricted to Windows NT and l0phtcrack; you could do the same thing with any password hash. For example, I used a Perl script (utilizing a dictionary text file) back in 2001 against hashes obtained from our NIS system to show the UNIX admins that the NIS installation was, politely speaking, insecure.

So choosing a password that is in any dictionary is clearly not a good idea (and really shouldn't be allowed when setting the password) since a dictionary attack can be fast. An easy way to prevent against a simple dictionary attack is to require users to choose an additional non-letter character. Since words in dictionaries usually don't contain characters other than letters, this is certainly a step in the right direction.

Secure Password for Dummies
Technically, adding a single non-letter character to an English word would indeed prevent a dictionary attack. Yes, a bad one! A persistent and motivated attacker (and most attackers are persistent and motivated) could modify their dictionary attack, and automatically prepend and append numbers to dictionary words, so that a password like "house7" or even "house1!" could still be found. This may sound like a lot of work, after all this would increase the time a dictionary attack takes around 20-fold "0house, 1house ... house0, house1, ....house9". True, but dictionary attacks are so fast that this technique would still be preferable to a brute-force attack. An attacker would still prefer a 4-hour dictionary attack over a 2-month brute-force attack (I made those numbers up, but the idea is that dictionary attacks are a lot faster). It also turns out that users tend to use the same numbers / special characters in their passwords, e.g. "1!", "99", "123" and so forth. Even worse, there appear to be a set of "favorite" passwords: http://www.schneier.com/blog/archives/2006/12/realworld_passw.html.

The obvious way to protect against a dictionary attack is to not use words from a dictionary in the first place. Indeed, many authentication systems require the use of letters with uppercase/lowercase, numbers as well as special characters. A requirement like this will surely protect us against even the most sophisticated dictionary attacks. Mission accomplished. Easy!

Not so fast. Attackers still have a few more options at their disposal. The attacker can:

1. Look for software vulnerabilities so that they can inject their own malware that would give them access to the server/workstation/network. They would then simply create a new user, reset the password of an existing user, or - if possible - just download whatever data they need. Event Log Monitoring can help here, since you can get notified when a new user is created/deleted or a password is changed (you could setup a filter to email you when a user password is changed between 11pm and 6am for example).
2. Employ social engineering techniques to get access to the password, either through physical access, a phone call or something similar. A combination of (1) & (2) is most common, as an attacker will send a malformed PDF (or similar) to the target, which will then implant some Trojan horse.
3. Use a brute-force attack to guess the password, either against an offline database (if the attacker was lucky enough to obtain one), or run the attack directly against the login system (a web site, a Windows domain, etc.). Owasp has a good article about brute-force attacks against web sites, which can be very susceptible to these types of attacks.

Brute-force attacks generally only work in two cases:

• One has access to an encrypted password database.
• The system one is trying log on to does not employ an account-lockout technique, so that a brute-force attack can be aimed directly at a logon portal.

Most network systems do employ an account lockout mechanism, and I highly recommend you enable this on systems which support it. Many systems, in particular web sites, do not support this functionality, however, so brute-force attacks are still a real risk.

When enabling account lockout, it's important to keep your end users in mind. Your users will ultimately need to log on to a network in order to do their work, and if the system locks them out every time they type in a wrong password twice, then your support team will spend a lot of time unlocking user accounts, and your users (depending on how calm they are) will be more or less annoyed. A log monitoring solution like EventSentry can email you when an account lockout occurs on a system (e.g. on Windows through the event log, on other devices through Syslog).

A patient system administrator on the phone troubleshooting a logon problem
So, when we brute-force a password, we try every combination regardless of dictionary and such. We start at "a" and make our way to, say, "ZZZZZZZZZZZ". Consequently, when the required password length is short (say 7 characters), a brute-force attack will be faster than when the required length is large (say 15 characters).

All this begs the ultimate question: Is it better to use a short complex password like C0mP1eX! (8 characters), or a long more simple password like ClimbingUpATree (15 characters)? Time to bring out the calculator. In order to come up with a conclusion, we'll create three password policies: One that requires complex but shorter passwords, and two with longer but less complex passwords.

Password Policy 1: "Complex Is Best"

Minimum length: 8 characters
Required character groups: One lowercase letter, one uppercase letter, one number, one special character out of: !@#$%^&*()_+[]{}
Possible passwords: 1,370,114,370,683,140 (yes, that's one quadrillion)

That's mighty complex, but a password like C0mp1ex! would be valid.

Password Policy 2: "A little long is enough"

Minimum length: 10 characters
Required character groups: One lowercase letter, one uppercase letter, one number. Special characters are allowed but not required.
Possible passwords: 839,299,365,868,340,000 (that's 839 quadrillion and a little bit)

Of course there would be even more possible passwords if a user decides to include a special character in their password (after all the policy only specifies the minimum requirement, and we wouldn't dare prohibit additional complex characters now, would we?). This policy is 612 times more complex than the previous policy, even though it only requires two more characters. It's flaw, however, is that a user could potentially use insecure passwords like Gardenhose1 which could be guessed with a sophisticated dictionary attack.

Password Policy 3: "The longer, the better"

Minimum length: 15 characters
Required character groups: One lowercase letter, one uppercase letter
Possible passwords: 54,960,434,128,018,700,000,000,000 (that's 54 septillions, 960 sextillions, 434 quintillions - you get the idea)

Phew, you need a lot of GPUs and a time machine to brute-force a password from that selection - and that's without even requiring a user to include a number! This policy is 40 billion (40,117,105,202 to be exact) times more complex than the first policy. And long passwords are not hard to come up with - just use a simple sentence like "Idontlikepasswords55" is a pretty long password (20 characters) and not that hard to remember at all.

A compromise?
As you can see, length trumps complexity in most cases, but as is often the case in computer security, things aren't always as simple as they seem. The numbers are correct, but a longer password without complex requirements might, as mentioned before, encourage a user to choose a password that could be guessed with a sophisticated dictionary attack.

For example, "Gardenhose1" would match the 2^nd policy's requirements but not be very secure. Users also tend to use family names, user names and the like in their passwords. A smart attacker could leverage this and adapt their dictionary attack accordingly. So if "Jean Reno" was to use "JeanReno1948" as his password, then this would still not be as secure as assumed - despite the 12-character length.

We can see that every additional character in the length of a password increases the possible combinations exponentially, more so than a few additional special characters. Still, not requiring special characters at all might allow the end user to pick passwords that are in a dictionary. Requiring extremely complicated passwords, on the other hand, will make it difficult for many users to remember them, and your end users might resort to writing their passwords down on a post-it, the bottom of their keyboard, or come up with "secure" passwords like ASDFasdf1!. Yes, annoyed users can be very creative. Put yourself in the shoes of somebody who is not familiar with security and needs to choose a password, would you voluntarily choose something like T3a#fE@8 ?

You do not want your users doing this
Password expiration policies, while certainly important, can also result in a backlash from your users. Users can get very annoyed with overly ambitious expiration policies, and fight back with number schemes or passwords stored on post-it notes (see http://open.salon.com/blog/unemployedmarx/2011/02/17/changing_passwords if you don't believe me). The result is the opposite: weaker passwords and less security. How about a "secure" 9-letter password that even exceeds our first policy?

• Cats2010!
• Cats2011!

or

• Q1cats2011!
• Q2cats2011!

This is heaven for an attacker: as long as the user sticks to the same pattern, the attacker will always know the password - even if the user changes it every 3 months. See "Changing Passwords" (http://www.schneier.com/blog/archives/2010/11/changing_passwo.html) for a more thorough discussion on this topic.

So what's the solution? As often, probably a little bit of everything. Dictionary passwords need to be avoided like the plague, so we'll never get around requiring some complexity. Complexity alone can be misleading though, so a minimum length of 12 characters seems like a good baseline. In addition, enable account lock-out techniques and set a (reasonable) maximum password age. A pretty good password policy would look like this:

• 18 characters minimum
• Lowercase, uppercase & numbers
• 180-day password age
• No part of first name, last name, username, etc. allowed in password
  

Also, don't forget to educate your users, so that people know why and how. Tell them that corporate espionage is a real threat, and suggest the use of a sentence for a password. Of course there will always be naysayers, but the majority of your user base should understand this.

Abuse

I have seen web sites (e.g. banking) require me to use a complex password, yet require that it shall be no longer than 8 characters! Whatever the reason behind something like that, it's far from secure and counterproductive. Even if I'd want to choose a strong 14-letter password I couldn't, and I would have to settle for something less secure.

Password Reuse
Another often overlooked risk is the reuse of passwords. Nowadays, people are required to use passwords at a multitude of web sites and systems. Some of those web sites store confidential information (SSN, credit card), but many don't.

The more often one uses the same password, the higher the risk that it is compromised. As such, your password - if used at more than one place - is only as strong as its weakest link. Don't use the same password that you use for your banking web site on your photo-sharing site!

I personally don't care too much if some cracker hacks the photo-sharing site I use, and downloads (and cracks) all the passwords. But I do care if my password to my banking web site is compromised. An attacker may not be able to easily guess a password at bankofamerica.com, but if I use the same password as my photo-sharing site, then I'm just asking for trouble. Recycling and reuse are a good thing - but not with passwords.

I hope this longer than expected article inspired you to review your corporate password policy, and maybe even your personal password habits. If you made it this far then I have included some relevant links regarding ... well ... passwords!

Nick Helm's password joke:
http://www.bbc.co.uk/news/uk-scotland-14646532

Interesting Statistics:
http://www.passwordresearch.com/stats/statindex.html

A Strong Password Isn't the Strongest Security:
http://www.nytimes.com/2010/09/05/business/05digi.html