Security, et al

Originally published at Lumension.com
http://blog.lumension.com/4804/chances-are-someone-is-trying-to-steal-your-organizations-information/

I'm happy to let you know that with the recent release of LOGbinder SP 3.0 I've updated our Recommended Alerts and Reports for SharePoint (LOGbinder SP) which you can find at http://www.logbinder.com/products/logbindersp/resources/reports.aspx.

Updates include

• coverage of new events and features in LOGbinder SP 3.0.
• new recommended alert rules
• important notes and explanations regarding "insertion strings", how date/time works in LOGbinder SP and information about the 2 possible "event log sources" that LOGbinder SP events can bear.

This free resource is valuable for anyone looking for tips and recommendations on creating reports or alert rules for SharePoint audit events.

I have 2 rules of thumb when it comes to audit logging.  First, if it has a log I recommend enabling it – as simple as that.  The only exceptions are prohibitively verbose logging options associated with debugging or the surprisingly rare case where logging has an impact on performance.  Second, if you can collect the log and archive it with your log management/SIEM solution, do it - even if you don’t set up any alert rules or reports. 

I’ve repeatedly seen the value following these 2 rules for several reasons.  First, you really never know what audit log data will be valuable in a forensics situation and there is no way to reconstruct log data after the fact.  Enable auditing beforehand or forever hold your peace.  The most obscure or seemingly unimportant systems and applications may end up being crucial to determining the extent or root vector of an intrusion or critical to building your case in insider investigations.  One time I was called upon to help document a case of inappropriate behavior by a user and the client program for a popular Internet provider who at the time provided a log which proved very helpful in showing the user accessed certain forums at specific times linked to the actions being investigated.

Security events that you don’t necessarily monitor and respond to may also be valuable in providing documentation that you performed certain actions when necessary as part of a compliance related matter.  For instance in my standard audit recommendations for Active Directory I suggest auditing and archiving events related to the disabling and deletion of accounts and group membership removals.  There is typically no need to review these events but when the auditor comes around you can use them to document that you removed access in response to changes in employee status and role.

Finally, audit logs even if not reviewed can serve as a powerful deterrent against malicious actions and policy violations by end users and administrators alike.  If they know that their actions are being recorded and can be looked up and used against them in the future they will be much less likely to take chances.  But that’s only true if the audit logs are collected an archived to a system separate from the people being monitored.  Otherwise, especially in the case of administrators, logs a vulnerable to deletion or tampering.

What are some common logs that you should consider enabling and if possible adding to your log management process for collection and archival besides the obvious candidates like your domain controller security logs?  First and foremost, I strongly recommend you enable appropriate auditing on member servers and at least securely archiving them with your DC logs.  Many of the most critical security events on your network are only caught by your member servers.  This would include attempts to break into member servers with local accounts, security configuration changes, programs executed and files accessed.  Then like I stressed in last month’s article there is a wealth of security information in your workstation security logs.  I understand that you have a lot of workstations and it may be a challenge to collect and archive that many logs and that amount of data, but at least enable auditing on those workstations.  Set maximum log size to 300MB and chances are the audit data will be there when you need it even if you aren’t centrally managing them.

Besides the Windows security logs of all your systems, what are other good candidates for logging?  I highly recommend enabling DHCP server logging because many other security logs list the IP address but no other information to identity to identify the client.  If you have your DHCP logs you can look up that IP address and date and time against the lease events to figure out the MAC address and likely the computer name of the system that had that IP address at the time the event was recorded.  If you use RRAS (Routing and Remote Access Server) be aware that it has client authentication and connecting logging capabilities which record important events not sent to the Windows security log.  The same goes for IIS.  IIS can log every incoming request to any web based application you host including URL, verb, result code, client IP address and more.  This is critical for tracking down attacks against your website and other web based applications.  More and more companies are beginning to store and process security and compliance critical information in SharePoint and SharePoint too has an audit log capability.  Exchange 2010 has a new non-owner mailbox activity log and a new administrator audit log.  VMWare has an audit capability that is critical to ensuring accountability over your virtualization infrastructure.  SQL Server 2008 has a new audit log too.  Even Microsoft Dynamics CRM 2010 has an audit capability. 

 It seems like every software vendor is seeing the need to provide audit trail capability which is good news for information security but it also means we need to learn how to at least turn on these new audit features and ensure we don’t inadvertently consume too much storage or other resources.   And remember, there’s good reason to enable auditing even if you aren’t regularly reviewing or monitoring the logs.  If at all possible get these logs out of the systems and applications where they are generated and into your separate and protected log management/SIEM solution so that the integrity of these audit trails in ensured.  Follow these steps and chances are you will score points in the future when you can explain who did what and when they did it!

Originally published at Lumension.com
http://blog.lumension.com/4675/the-year-i-started-being-afraid/

I’ve been in IT since I was a kid. I was a real, stereotypical nerd. While other computer nerds were learning to program games, I turned up my nose at their childish efforts and learned database programming because at 12 I actually wanted to write accounting software. I know, I know, weird. Anyway I say this to underline the fact I’ve been in technology since PC’s first came out and business technology at that.

When the Internet and Windows NT got big in the nineties I switched from development to security – so I’ve been there. But except for a brief period when I got my first constant connection to the Internet, I haven’t been afraid. I’ve respected the risks of the Internet and the danger from the bad guys on it but I’ve never been paranoid like many of my other colleagues in infosec. I’ve always taken sensible measures, ran AV most of the time, kept my attack surface small and monitored my logs. And I’ve only had 2 security incidents over that time. I got the “Ethan Frome” Word virus which was harmless and after that my Windows 2000 based IIS web site suffered SQL injection once which broke some drop down lists on the site.

Then Things Started to Change

So, I’ve always felt pretty safe with my informed, common sense approach. But last year that started to change. Part of it is because my business is growing. More people are on my network. More endpoints connect to my network. There’s more to protect. Part of it is the accelerating sophistication of bad buy technology. Malware is getting more sophisticated and beginning to outpace signature based detection. The bad guys’ work in content related vulnerabilities is outflanking us by going beyond OS and penetrating us via PDFs, JPGs, Flash files, ad nauseam. But the biggest part I think is that that bad guy of the last few years is a new and different bad guy.

It used to be loosely organized nihilistic antisocial kids defacing web sites sometimes for ostensibly social causes but sometimes for the pure nihilism of it. Or, in other instances, it was “security researchers” trying to make a name for themselves or their companies. And the scenarios I would describe in my security and audit classes where just that – theoretical scenarios about would could happen in theory. But when pressed for real examples and anecdotes, I usually came up short. It was more like “this is what could happen if we were in the middle of a Mission Impossible movie.”

But today, Mission Impossible scenarios are happening all the time. The biggest, most respected name in strong 2-factor authentication gets hacked. Then a major defense contractor apparently gets hacked as the fruits of the first attack.

The bad guys are now financially and politically driven. What are more powerful motivators than that? What can gather greater resources and stimulus than money and power? Religion is the only thing I can think of. But money and power are more than enough for now.

So I’m now taking information security more seriously for my business than I ever have before. And while it’s tempting to think about my little datacenters out there exposed to the Internet 24/7 or the data in the few cloud services we use, what keeps me awake at night – well, I’m not really at the point of losing sleep, but let’s say if I did wake up at night and start thinking about the security of my business – is endpoints.

Time to Worry About Your Endpoints

Endpoints worry me. There’s so many. They are so exposed. Endpoints process so much content directly from the Internet. And so intimately – a file server or a SharePoint site may store files from the Internet but it’s on the endpoint where they are actually parsed and rendered. (To be accurate, and while not extremely common, SharePoint servers are getting pretty intimate with content today given how Visio, Access and Excel is actually parsed, rendered and manipulated by SharePoint, within SharePoint itself.) And the bad guys know this. The endpoint is the initial target of APTs.

At least I realize this. Too many folks in management, IT, infosec and Internal Audit, still have the mainframe philosophy superimposed on servers: “all my data is on my server so I need to protect my server. Endpoints aren’t that important because that isn’t where the data resides.”

I’m preaching to the converted when I say “Wrong”, right? Let’s just put aside the reality there is confidential, sensitive data out there on nearly every laptop, workstation and smartphone. Let’s just assume for a minute that no important data is on a given endpoint.

Doesn’t matter. That laptop or other endpoint is still part of your trusted computing base and if it gets compromised, you’re in trouble. After all, I don’t think anyone believes the seed codes for SecurID tokens or anything else proprietary about SecurID technology was on the endpoint initially hacked at RSA when that poor employee opened the intriguing email about next year’s recruiting plan. But that’s where it started and we all know where it ended.

The security of your endpoints – of all our endpoints – is more than important – it’s critical. And I’m going to put some real effort into endpoint security this year. I hope you do too.

Originally published at Lumension.com
http://blog.lumension.com/4675/the-year-i-started-being-afraid/

I never have to convince people that monitoring Domain Controller security logs is important.  With member servers on the other hand I sometimes have to explain why those security logs are important but that’s not difficult because folks, and management in particular, understand that member servers is “where our data is”.  But when it comes to workstations I often face an uphill battle helping people understand why workstation security logs are so important. 

Frequently I hear things like “but we have a policy that forbids storing confidential information locally”.  Be that as it may workstations and especially laptops always have sensitive information on them – there’s no way to prevent it.  Besides applications like Outlook, Offline Files and SharePoint workspace that cache server information locally, there’s also your page file which at any time can contain content from any document or other information you are working with.

But even if there were no confidential information on your workstations, their security logs are still very important to your enterprise audit trail, forensics and for detecting advanced persistent threats.  There’s a wealth of audit trail information logged by workstations that you can’t find anywhere else and consequently a host of questions that can only be answered by workstation logs.

First of all, if you care about when a user logged off, you can only find this information in the workstation’s security log.  Domain controllers audit your initial authentication when you logon to your workstation but essentially forget about you after that.  You can’t figure out logoff times based on shared folder connections because Windows does not keep network logons open to file servers between file accesses.  So, when you logout from Windows, the only computer on the network that logs this is your workstation. 

And what about logon failures?  Yes, domain controllers do log authentication failures but the events logged are tied to Kerberos and the error codes are based on RFC 1510’s failure codes.  Kerberos failure codes are not as granular and do not map directly to all the reasons a logon can fail in Windows.  Therefore some authentication failure codes provided in event IDs 4768 and 4771 can mean any one of several possible reasons.  For instance, failure code 0x12 which Kerberos defines as “Clients credentials have been revoked” can mean that the logon failed due to the account being disabled, locked out or outside of authorized logon hours.

With today’s focus by bad guys on the endpoint you also need to know if someone is trying to break into to a workstation. If the attacker is trying to break into the workstation with a domain account you can find evidence of this in your domain controller security logs by looking for the same to events mentioned above.  However workstations also have local accounts and these are big targets for attackers since local accounts are often poorly secured and tend to fly under the radar in terms of security monitoring.  When you attempt to logon to a workstation with a local account, this activity is not logged on the domain controller.  Since the authentication is being handled locally the event is also logged locally in the form of event ID 4776.  Don’t let the description of this event throw you because it says “The domain controller failed to validate the credentials for an account”.  The event should say “the system” instead of “domain controller” because this event is logged on all types of computers.

You can track what files a user accesses on file servers with file system auditing on those servers but to have an audit trail of what programs the user is executing, that too is only logged on workstation security logs. In forensic investigations I’ve found that knowing what programs a user ran and for how long was crucial to documenting what occurred and making our case.  Furthermore, many of the advanced persistent threat (APT) attacks being waged depend on malicious executables run on end user workstations.  So you can’t afford not to have a record of what’s running on your endpoints.  Unfortunately, the Process Tracking (aka Detailed Tracking) category only logs programs run on the local computer therefore no information about end user desktop program usage is available in domain controller or member server security logs. The only process events logged on servers are the actual server programs that execute there.

As you can see there’s a lot of critical audit trail information only available if you enable auditing on your workstations.  Of course enabling auditing on workstations is one thing while collecting logs from thousands of additional computers is another.  In addition there are some very important workstation security events which Windows auditing does not record.  For instance, Windows does not audit when you connect or disconnect devices or removable storage like flash drives.  Much less does it record what files you transfer to or from the removable storage.  Nor does Windows audit the installation of software.

 In a webinar I will present later this year in cooperation with Prism Microsystems I’d dive more deeply into these issues and how to address them.  t’s important that we all educate decision makers about why endpoint security and in this case audit logs from endpoints is so important.  We have to get beyond the mainframe inspired mindset that security only matters on the centralized systems where critical data resides.  Workstations are really just as important as any other component of a secure network.  If an attacker can compromise the workstation of a user with access to critical information the attacker can impersonate that user and access any information or applications that user has access to on the network.  Even workstations of users without access to sensitive resources are important because attackers, especially in APT, scenarios are happy to start with any endpoint as beach head and attack other systems from there.  Moreover workstations are arguably the most vulnerable components of your network since they process so much content from the Internet connected with web browsing and email, because they come into contact with potentially infected files on removable storage and because they connect to other insecure networks like Wi-Fi hotspots.  Keep an eye and be sure to register for this event and invite your manager.

This month I’m going to zero in on a specific area of audit logging that creates no end of confusion, and that is the difference between 2 categories in the Windows security log: Account Logon events and Logon/Logoff events.  These 2 categories are related but distinct and their names contribute to the confusion.  Let’s start by calling Account Logon events what they really are – authentication events.  I wish Microsoft had named the Account Logon category Authentication; it would have helped a lot. The way I remember it is, “the A in Account Logon is for Authentication.”

That being said, what is the difference between authentication and logon?  In Windows, when you access the computer in front of you or any other Windows computer on the network you must first authenticate and then you obtain a logon session on that computer.  Authentication is a point in time event.  A logon session has a beginning and end.  So are authentication events simply a duplicate of logon events?  No, and the reason is that authentication may take place on a different computer than the one you are logging into. 

Workstation Logons

Let’s start with the simplest case.  You are logging onto at the console (aka “interactive logon”) of a standalone workstation – meaning it is not a member of any domain – in fact let’s say it’s not even connected to a network.  The only type of account you can logon with in this case is a local user account defined in Computer Management \ Local Users and Groups.  You don’t hear the term much anymore but local accounts are the same thing as a SAM account.  In this case both the authentication and logon occur on the very same computer because we simply logged on to the local computer using a local account.  Therefore you will see both an Account Logon event (680/4776) and a Logon/Logoff (528/4624) event in its security log.  Note in this article I’ll always note the 3 digit Windows 2003 event followed by the 4 digit Windows 2008 event.

Now let’s make this workstation a member of a domain.  At this point it’s possible to authenticate to this computer using a local account or a domain account – or, for that matter, a domain account from any domain that this domain trusts.  We’ve already discussed what happens when you logon with a local account so let’s logon with a domain account.  Since the user specifies a domain account, the local workstation can’t perform the authentication because the account (and its password hash) isn’t stored locally.  So the workstation must request authentication from a domain controller via Kerberos.  So this time, the authentication event (672/4768) is logged on which ever domain controller handles the authentication request from the workstation.  Once the domain controller tells the workstation that the user is authentic the workstation proceeds with creating the logon session and a records a logon event (528/4624) in its security log. 

What if we logon to the workstation with an account from a trusted domain?  In that case one of the domain controllers in the trusted domain will handle the authentication and log 672/4768 there, with the workstation logging 528/4624 the same as above. 

In all such “interactive logons”, when you finally logoff, the workstation will record a “logoff initiated” event (551/4647) followed by the actual logoff event (538/4634).  You can correlate logon and logoff events by Logon ID which is a hexadecimal code that identifies that particular logon session.

Accessing Member Servers

After logging on to a workstation you normally re-connect to shared folders on a file server.  What gets logged in this case?  Remember, whenever you access a Windows computer you must obtain a logon session – in this case a “network logon” session.  Intuitively you might assume that the logon session begins when you connect to the share and then ends when you disconnect from it – usually when logging off your local workstation.  Unfortunately though, Windows servers only keep network logon sessions alive for as long as you have a file open on the server.  This accounts for why you see repeated logon/logoff events on Windows file servers by the same user throughout the course of the day.  With network logons, Windows 2003 logs 540 instead of 528 while Windows 2008 logs 4624 for all types of logons. 

When you logon at the console of the server the events logged are the same as those with interactive logons at the workstation as described above.  More often though, you logon to a member server via Remote Desktop.  In this case the same 528/4624 event is logged but the logon type indicates a “remote interactive” (aka Remote Desktop) logon.  I’ll explain logon types next.

Something we often need to know, when looking at logon events is what type of logon are we dealing with?  Is this an interactive logon at the console of the sever indicating the user was physically present or is it a remote desktop logon.  For that matter the logon could be associated with a service starting or a scheduled task kicking off.  In all such cases you will need to look at the Logon Type specified in the logon event 528/540/4624.  A full list of Logon Types is provided at the provided links for those events but in short:

Events at the Domain Controller

When you logon to your workstation or access a shared folder on a file server, you are not “logging onto the domain”.  There’s no such concept.  Each Windows computer is responsible for maintaining its own set of active logon sessions and there is no central entity aware of all everyone who is logged on somewhere in the domain.  After servicing an authentication request, the domain controller forgets about you; it doesn’t know how you were logging (console, remote desktop, network, etc) nor when you logoff. 

“But wait,” you might say “I see logon events on the domain controller, not just authentication events, whenever a user logs on to their workstation” and then periodically after that.   OK, that’s true: on domain controllers you often see one or more logon/logoff pairs immediately following authentication events for the same user.  But these logon/logoff events are generated by the group policy client on the local computer retrieving the applicable group policy objects from the domain controller so that policy can be applied for that user.  Then approximately every 90 minutes Windows refreshes group policy and you see a network logon and logoff on the domain controller again.  These network logon/logoff events don’t tell you much, I regard them usually as noise.  In forensic situations they could give you a rough idea of how long the user was logged on since as long as the user remains logged on group policy will refresh about every 90 minutes.  The other possibly useful thing about these group policy related logon/logoff pairs is that they help you infer that the preceding authentication events for the same user were in conjunction with an interactive or remote desktop logon as opposed to a service or scheduled task logon. 

What about the other service ticket related events seen on the domain controller?  That requires an in-depth discussion of Kerberos and how it’s implemented which is beyond the scope of this article.  Basically, after your initial authentication to the domain controller which logs log 672/4768 you also obtain a service ticket (673, 4769) for every computer you logon to including your workstation, the domain controller itself for the purpose of group policy and any member servers such as in connection with shared folder access.  Then as computers remain up and running and users remain logged on, tickets expire and have to be renewed which all generate further Account Logon events on the domain controller.

The Facts: Good, Bad and Ugly

Both the Account Logon and Logon/Logoff categories provide needed information and neither can replace the other – you need both.  And, ideally, you really need both categories of events from all your computers – including your workstations.  Here’s a number of important facts you need to understand, and… well… accept. 

1.       To determine definitely how a user logged on you have find the logon event on the computer where the account logged on.  You can only make some tenuous inferences about logon type by looking at the domain controller and that requires analyzing multiple events. 

2.       To determine when a user logged off you have to go to the workstation and find the “user initiated logoff” event (551/4647).

3.       To correlate authentication events on a domain controller with the corresponding logon events on a workstation or member server there is no “hard’ correlation code shared between the events.  Folks at Microsoft have suggested the Logon GUID field in these events would provide that but my research and experiments indicate that unfortunately the GUIDs are either not supplied or do not match.  So to make that correlation you basically have to dead reckon based on time, computer names and user account names. 

4.       Account Logon events on domain controllers are great because they allow you to see all authentication activity (successful or failed) for all domain accounts.  Remember that you need to analyze the security logs of all your domain controllers – security logs are not replicated between DCs. 

5.       Account Logon events on workstations and member servers are great because they allow you to easily pick out use of or attacks against local accounts on those computers.  You should be interested in that because using local accounts is bad practice and bad guys know they tend to be more vulnerable than domain accounts.  But, you don’t have to use Account Logon to detect logon attempts on local accounts; you can use Logon/Logoff events if you know what you are doing.  When viewing a Logon/Logoff event compare the domain name in the event details to the computer name that generated the event; if they match you are looking at a local account logon attempt – otherwise the domain name field with reflect some domain.  So can you survive with only enabling Logon/Logoff events on member servers and workstations?  I suppose so.

6.       Logon/Logoff events are a huge source of noise on domain controllers because every computer and every user must frequently refresh group policy.  If you disable this category on domain controllers what will you lose?  You will lose some visibility into logons at the domain controller itself such as when an admin logs on at the console, via remote desktop or a service or scheduled task starts up.  In all cases Account Logon events will still be logged but see points 1 and 2 above.

7.       Basically, successful network logon and logoff events are noise on domain controllers and member servers because of the amount logged.  Unfortunately you can’t just disable successful network logon/logoff events without also losing other logon/logoff events for interactive, remote desktop, etc.  My old adage is: “you can’t configure the noise out of the Windows security log” – that’s the job of your log management / SIEM solution. 

So that’s the difference between Account Logon (i.e. authentication) and Logon/Logoff events.  All things considered, I’d like to see both categories enabled on all computers ideally.  I haven’t seen these events create a noticeable impact on the server but the amount of log data might exceed your log management / SIEM solution’s current capacity.  If you can’t afford to collect workstation logs, I still suggest enabling these 2 categories on workstations and letting the log automatically wrap after reaching 100MB or so.  Chances are the data will be there if you need it for forensic purposes.

Dynamics CRM 2011 keeps us sane here at Monterey Technology Group, Inc as we manage a wide array of product and service offerings with a handful of people.  But CRM is missing some key features that seem like no brainers.  Thankfully I've found solutions to each (been using them since CRM 4.0) and thought I'd share them. 

1. No way to print a quote to PDF and email it from within CRM in one step - crazy I know!  Solution: ePDF from http://downloads.mycrmgroup.com/.  Incredibly easy to install!  Great support!

2. Moving invoices from CRM to QuickBooks.  OK this isn't a missing feature but definitely a needed integration link.  Solution: Inogics Inolink http://www.inogic.com/integration_quickbooks.htm.  Involved install process and you will probably need support but they are responsive and it does work.

3. Converting incoming emails to CRM Queues to Cases - crazy I know!  For this I use c360's EmailToCase.  Least favorite solution and company out of the 3 but it gets the job done and their support staff do respond.

I think all of these support CRM Online in addition to on-premise.

Was just messing with BitLocker today.  I enabled BitLocker on a Win7 computer that is a member of a domain but before configuring group policy to require BitLocker recovery keys to be backed up to AD before locking the drive.

So I enabled the "Store BitLocker recovery information in Active Directory Domain Services" policy.  I forced a group policy refresh on that PC.   Then I went to my domain controller (win2008r2) and opened that computer account. First problem: no BitLocker tab on the computer account's properties dialog.  Had to open Server Manager and add the BitLocker Recovery Password Viewer under Add Features, Remote Server Administration Tools, Feature Administration Tools, BitLocker...

OK, after that the BitLocker tab showed up but nothing had been backed up.  If you miss requiring backup to AD when you first enable BitLocker it will never happen unless you explicitly tell Windows to with manage-bde.

So after LOTS of horsing around with manage-bde and figuring out all the really bad documentation errors on Technet and in the command line help I figured out I had to run "manage-bde -protectors -adbackup C: -ID {GUID}".  To figure out the GUID I had to run "manage-bde -protectors -get c:".

Now, the BitLocker tab on this computer's account in AD properly shows the recovery password and it's ID.

I’ve been noticing a lot hype about security risks with the rise of virtualization and much of it vague and short on specifics.  Also much of it seems to assume all the security we’ve built up on a physical server goes out the window when you migrate to being a virtual machine.  And that’s just not true.  A virtual server is the SAME server it was before it was P2V’d from a physical server.  The same authentication, access control, audit, network controls are active as they were before.  The virtual server sits on hosts and SANs in the same datacenter as the physical server did before.  So what has changed?  What are the new risks?

It’s in the virtualization infrastructure layer.  (Given the ubiquity of VMWare and my personal knowledge of that solution I’m going to use it to give various virtualization components real world names.)  How secure is the host, the storage and the host control server.  In VMWare terms I mean how secure are the ESX/ESXi hosts, the SANS and the vCenter servers?  That is the real question here.  Now that we’ve inserted a new layer between the guest OS and the hardware that layer’s immediate components and other components they depend on (the Active Directory forest that the vCenter servers belong to) need the same security controls anything else on the work needs.  Virtualization security is even more critical in some respects because someone who compromises that layer has low level access equivalent to physical access to every guest server and its data.

I have observed in recent audits that some areas of security and control of virtualization components is immature and does reflect consciousness how critical the virtualization layer is to security.  Before I dive into those areas let’s first talk about network security.  In the hype I referred to in the introduction, much is made about network security risks associated with virtualization and I’m not really seeing that.  Most servers are not behind internal firewalls or on heavily restricted network segments in the first place so moving them to an ESX/i host on a virtual switch doesn’t expose the server to new network risks.  For servers that had such controls when they were physical, you can set up the very same thing with virtual switches and firewalls.  If you need to mix external facing and internal VMs on a host you can set up completely different virtual switches and that’s what I IT shops doing.  There are more advanced attack scenarios involving a compromised VM where the attacker breaks out of that VM and into the host and possibly back up into other VMs but right now that is one area where we are ahead of the bad guys in the arms race. That is something to stay vigilant about though.

The other area of network security is the protection of the visualization infrastructure itself.  Unless IT shops totally ignore virtualization best practices they are going to implement multiple network cards on ESX/i hosts that allow you to completely separate guest, live migration (aka vMotion), storage and management (connections from VMWare vCenter and clients to the host) traffic.  In the audits I’ve performed, SANs and the management interfaces on hosts are isolated from rest of the organization’s internal LAN. 

This brings me to the areas of virtualization where I have found serious risks: privileged access, Active Directory and auditing.  First privileged access.  Anyone reading this knows that allowing administrators (especially multiple administrators!) use the built-in root or admin account on operating system is bad practice and risky for all sorts of reasons.  Everyone should have their own account.  The same goes for virtualization hosts only more so.  However in my audits I’ve observed many hosts being managed by the built-in root account shared among multiple administrators.  With a virtualization host, the risk of shared and insecure root accounts is multiplied by number and criticality of all the guest VMs on that host.  There’s no good reason for this bad practice and there’s no need to create additional root accounts on hosts for each admin either.  That’s because best practice is to lock down ESX/i hosts so that even admins don’t directly access them and instead go through the central management server (called vCenter in the case of the VMWare environment).  vCenter doesn’t share the same prevalence of insecure root access because vCenter integrates with Active Directory and allows you to leverage the AD accounts admins already have.  If for some reason a shop can’t follow that important best practice of avoiding direct access to hosts then you can configure the hosts themselves to integrate with Active Directory.

Speaking of Active Directory, I’ve uncovered another prevalent risk associated with the dependence of virtualization infrastructures on AD.  Let me be clear that I think directory integration and unified authentication is definitely the way to go but here are some risk factors people have been missing.  First there’s the fact that I often find virtualization management servers like vCenter to be members of the main AD forest.  Here’s what that means.  A Windows server belonging to a domain is exposed to any all risks in Active Directory and all domain controllers within that forest (remember the security boundary in AD is the forest not the domain).  Building on that, a vCenter server is the “boss” of all the ESX/i hosts connected to it.  Ergo, everyone who has domain admin authority anywhere in the AD forest and anyone who compromises a domain controller in the AD forest can take the vCenter server and from there compromise the virtualization infrastructure and ultimately any guest VM and its data.  A point I have to make, sometimes to the chagrin of some folks in IT, is that any outstanding risks from previous AD audits must be carried forward to the virtualization infrastructure audit too.  For instance, at one bank, there was a problem of a very excessive number of IT folks having domain admin authority to the main AD forest.  That was problem enough as a risk to AD and the Windows systems within that forest.  But with the virtualization management server as a member of that forest, now every virtual machine – even those running Linux and Windows servers in other forests are now accessible to that same excessively large group of AD admins.  Worse, in this organization remote access was widely available with no strong authentication.  So the entire virtualization infrastructure and countless servers running on it were vulnerable to compromise by a successful password based attack against any one of many AD admins. 

The solution?  First, think about the AD forest(s) that hold either your virtualization management servers (e.g. vCenter) or those user accounts with privileged access to the virtualization infrastructure (i.e. users with the Administrator role in vCenter).  Those forests, including each domain and domain controller within them, must be locked down and secure to a level appropriate for the virtualization infrastructure itself.  There’s no avoiding that.  Organizations with outstanding AD security issues with no resolution in site should really look at implementing a small, separate AD forest for providing directory and authentication services to their infrastructure including virtualization, storage and network components.  This small AD forest would be much more locked down and protected and careful thought should be given before implementing synchronization or trust relationships between it and other forests.  This may be at the price of maintaining additional user accounts for infrastructure admins but that is the price of security in this instance.  If trust is implemented it should such that the infrastructure forest is trusted by the other forests not vice versa.   If synchronization is implemented, password changes or other authentication data should not flow from other forests or directories into the infrastructure forest.  

The final risk area I’ve observed with newly virtualized organizations is a lack of auditing and log management for virtualization infrastructure components.  And in this case there’s no easy solution.  Virtualization management servers (e.g. vCenter) and hosts (e.g. ESX/i) can generate audit logs and it is crucial to enable this feature and subsequently collect, archive alert and report on this log data the same as is necessary with any other security critical components on your network.  Virtualization hosts like ESX/i are simple to accommodate since they can send events via syslog.  But management servers like vCenter are more problematic.  vCenter creates a number of text log files named vpxd-1 through 9 but my research has proven them to omit very important data and fail to resolve other key ID codes.  That’s not to say the audit trails aren’t there.  They are but they are trapped inside database tables.  In the case of vCenter the audit trail is stored in VPX_EVENT and VPX_EVENT_ARG tables within the vCenter SQL database.   Incomprehensibly command line interfaces like Get-VIEvent that pull data from these leave out critical event arguments as well as the name or ID of the event itself!  So the final option seems to be direct query of the SQL tables themselves with the necessary resolution of foreign keys to rows in related tables.  This presents an opportunity to log management and SIEM vendors to distinguish themselves with enhanced support for collecting enriched audit trails from virtualization infrastructures.