Security, et al

This month I’m going to zero in on a specific area of audit logging that creates no end of confusion, and that is the difference between 2 categories in the Windows security log: Account Logon events and Logon/Logoff events.  These 2 categories are related but distinct and their names contribute to the confusion.  Let’s start by calling Account Logon events what they really are – authentication events.  I wish Microsoft had named the Account Logon category Authentication; it would have helped a lot. The way I remember it is, “the A in Account Logon is for Authentication.”

That being said, what is the difference between authentication and logon?  In Windows, when you access the computer in front of you or any other Windows computer on the network you must first authenticate and then you obtain a logon session on that computer.  Authentication is a point in time event.  A logon session has a beginning and end.  So are authentication events simply a duplicate of logon events?  No, and the reason is that authentication may take place on a different computer than the one you are logging into. 

Workstation Logons

Let’s start with the simplest case.  You are logging onto at the console (aka “interactive logon”) of a standalone workstation – meaning it is not a member of any domain – in fact let’s say it’s not even connected to a network.  The only type of account you can logon with in this case is a local user account defined in Computer Management \ Local Users and Groups.  You don’t hear the term much anymore but local accounts are the same thing as a SAM account.  In this case both the authentication and logon occur on the very same computer because we simply logged on to the local computer using a local account.  Therefore you will see both an Account Logon event (680/4776) and a Logon/Logoff (528/4624) event in its security log.  Note in this article I’ll always note the 3 digit Windows 2003 event followed by the 4 digit Windows 2008 event.

Now let’s make this workstation a member of a domain.  At this point it’s possible to authenticate to this computer using a local account or a domain account – or, for that matter, a domain account from any domain that this domain trusts.  We’ve already discussed what happens when you logon with a local account so let’s logon with a domain account.  Since the user specifies a domain account, the local workstation can’t perform the authentication because the account (and its password hash) isn’t stored locally.  So the workstation must request authentication from a domain controller via Kerberos.  So this time, the authentication event (672/4768) is logged on which ever domain controller handles the authentication request from the workstation.  Once the domain controller tells the workstation that the user is authentic the workstation proceeds with creating the logon session and a records a logon event (528/4624) in its security log. 

What if we logon to the workstation with an account from a trusted domain?  In that case one of the domain controllers in the trusted domain will handle the authentication and log 672/4768 there, with the workstation logging 528/4624 the same as above. 

In all such “interactive logons”, when you finally logoff, the workstation will record a “logoff initiated” event (551/4647) followed by the actual logoff event (538/4634).  You can correlate logon and logoff events by Logon ID which is a hexadecimal code that identifies that particular logon session.

Accessing Member Servers

After logging on to a workstation you normally re-connect to shared folders on a file server.  What gets logged in this case?  Remember, whenever you access a Windows computer you must obtain a logon session – in this case a “network logon” session.  Intuitively you might assume that the logon session begins when you connect to the share and then ends when you disconnect from it – usually when logging off your local workstation.  Unfortunately though, Windows servers only keep network logon sessions alive for as long as you have a file open on the server.  This accounts for why you see repeated logon/logoff events on Windows file servers by the same user throughout the course of the day.  With network logons, Windows 2003 logs 540 instead of 528 while Windows 2008 logs 4624 for all types of logons. 

When you logon at the console of the server the events logged are the same as those with interactive logons at the workstation as described above.  More often though, you logon to a member server via Remote Desktop.  In this case the same 528/4624 event is logged but the logon type indicates a “remote interactive” (aka Remote Desktop) logon.  I’ll explain logon types next.

Something we often need to know, when looking at logon events is what type of logon are we dealing with?  Is this an interactive logon at the console of the sever indicating the user was physically present or is it a remote desktop logon.  For that matter the logon could be associated with a service starting or a scheduled task kicking off.  In all such cases you will need to look at the Logon Type specified in the logon event 528/540/4624.  A full list of Logon Types is provided at the provided links for those events but in short:

Events at the Domain Controller

When you logon to your workstation or access a shared folder on a file server, you are not “logging onto the domain”.  There’s no such concept.  Each Windows computer is responsible for maintaining its own set of active logon sessions and there is no central entity aware of all everyone who is logged on somewhere in the domain.  After servicing an authentication request, the domain controller forgets about you; it doesn’t know how you were logging (console, remote desktop, network, etc) nor when you logoff. 

“But wait,” you might say “I see logon events on the domain controller, not just authentication events, whenever a user logs on to their workstation” and then periodically after that.   OK, that’s true: on domain controllers you often see one or more logon/logoff pairs immediately following authentication events for the same user.  But these logon/logoff events are generated by the group policy client on the local computer retrieving the applicable group policy objects from the domain controller so that policy can be applied for that user.  Then approximately every 90 minutes Windows refreshes group policy and you see a network logon and logoff on the domain controller again.  These network logon/logoff events don’t tell you much, I regard them usually as noise.  In forensic situations they could give you a rough idea of how long the user was logged on since as long as the user remains logged on group policy will refresh about every 90 minutes.  The other possibly useful thing about these group policy related logon/logoff pairs is that they help you infer that the preceding authentication events for the same user were in conjunction with an interactive or remote desktop logon as opposed to a service or scheduled task logon. 

What about the other service ticket related events seen on the domain controller?  That requires an in-depth discussion of Kerberos and how it’s implemented which is beyond the scope of this article.  Basically, after your initial authentication to the domain controller which logs log 672/4768 you also obtain a service ticket (673, 4769) for every computer you logon to including your workstation, the domain controller itself for the purpose of group policy and any member servers such as in connection with shared folder access.  Then as computers remain up and running and users remain logged on, tickets expire and have to be renewed which all generate further Account Logon events on the domain controller.

The Facts: Good, Bad and Ugly

Both the Account Logon and Logon/Logoff categories provide needed information and neither can replace the other – you need both.  And, ideally, you really need both categories of events from all your computers – including your workstations.  Here’s a number of important facts you need to understand, and… well… accept. 

1.       To determine definitely how a user logged on you have find the logon event on the computer where the account logged on.  You can only make some tenuous inferences about logon type by looking at the domain controller and that requires analyzing multiple events. 

2.       To determine when a user logged off you have to go to the workstation and find the “user initiated logoff” event (551/4647).

3.       To correlate authentication events on a domain controller with the corresponding logon events on a workstation or member server there is no “hard’ correlation code shared between the events.  Folks at Microsoft have suggested the Logon GUID field in these events would provide that but my research and experiments indicate that unfortunately the GUIDs are either not supplied or do not match.  So to make that correlation you basically have to dead reckon based on time, computer names and user account names. 

4.       Account Logon events on domain controllers are great because they allow you to see all authentication activity (successful or failed) for all domain accounts.  Remember that you need to analyze the security logs of all your domain controllers – security logs are not replicated between DCs. 

5.       Account Logon events on workstations and member servers are great because they allow you to easily pick out use of or attacks against local accounts on those computers.  You should be interested in that because using local accounts is bad practice and bad guys know they tend to be more vulnerable than domain accounts.  But, you don’t have to use Account Logon to detect logon attempts on local accounts; you can use Logon/Logoff events if you know what you are doing.  When viewing a Logon/Logoff event compare the domain name in the event details to the computer name that generated the event; if they match you are looking at a local account logon attempt – otherwise the domain name field with reflect some domain.  So can you survive with only enabling Logon/Logoff events on member servers and workstations?  I suppose so.

6.       Logon/Logoff events are a huge source of noise on domain controllers because every computer and every user must frequently refresh group policy.  If you disable this category on domain controllers what will you lose?  You will lose some visibility into logons at the domain controller itself such as when an admin logs on at the console, via remote desktop or a service or scheduled task starts up.  In all cases Account Logon events will still be logged but see points 1 and 2 above.

7.       Basically, successful network logon and logoff events are noise on domain controllers and member servers because of the amount logged.  Unfortunately you can’t just disable successful network logon/logoff events without also losing other logon/logoff events for interactive, remote desktop, etc.  My old adage is: “you can’t configure the noise out of the Windows security log” – that’s the job of your log management / SIEM solution. 

So that’s the difference between Account Logon (i.e. authentication) and Logon/Logoff events.  All things considered, I’d like to see both categories enabled on all computers ideally.  I haven’t seen these events create a noticeable impact on the server but the amount of log data might exceed your log management / SIEM solution’s current capacity.  If you can’t afford to collect workstation logs, I still suggest enabling these 2 categories on workstations and letting the log automatically wrap after reaching 100MB or so.  Chances are the data will be there if you need it for forensic purposes.

Dynamics CRM 2011 keeps us sane here at Monterey Technology Group, Inc as we manage a wide array of product and service offerings with a handful of people.  But CRM is missing some key features that seem like no brainers.  Thankfully I've found solutions to each (been using them since CRM 4.0) and thought I'd share them. 

1. No way to print a quote to PDF and email it from within CRM in one step - crazy I know!  Solution: ePDF from http://downloads.mycrmgroup.com/.  Incredibly easy to install!  Great support!

2. Moving invoices from CRM to QuickBooks.  OK this isn't a missing feature but definitely a needed integration link.  Solution: Inogics Inolink http://www.inogic.com/integration_quickbooks.htm.  Involved install process and you will probably need support but they are responsive and it does work.

3. Converting incoming emails to CRM Queues to Cases - crazy I know!  For this I use c360's EmailToCase.  Least favorite solution and company out of the 3 but it gets the job done and their support staff do respond.

I think all of these support CRM Online in addition to on-premise.

Was just messing with BitLocker today.  I enabled BitLocker on a Win7 computer that is a member of a domain but before configuring group policy to require BitLocker recovery keys to be backed up to AD before locking the drive.

So I enabled the "Store BitLocker recovery information in Active Directory Domain Services" policy.  I forced a group policy refresh on that PC.   Then I went to my domain controller (win2008r2) and opened that computer account. First problem: no BitLocker tab on the computer account's properties dialog.  Had to open Server Manager and add the BitLocker Recovery Password Viewer under Add Features, Remote Server Administration Tools, Feature Administration Tools, BitLocker...

OK, after that the BitLocker tab showed up but nothing had been backed up.  If you miss requiring backup to AD when you first enable BitLocker it will never happen unless you explicitly tell Windows to with manage-bde.

So after LOTS of horsing around with manage-bde and figuring out all the really bad documentation errors on Technet and in the command line help I figured out I had to run "manage-bde -protectors -adbackup C: -ID {GUID}".  To figure out the GUID I had to run "manage-bde -protectors -get c:".

Now, the BitLocker tab on this computer's account in AD properly shows the recovery password and it's ID.

I’ve been noticing a lot hype about security risks with the rise of virtualization and much of it vague and short on specifics.  Also much of it seems to assume all the security we’ve built up on a physical server goes out the window when you migrate to being a virtual machine.  And that’s just not true.  A virtual server is the SAME server it was before it was P2V’d from a physical server.  The same authentication, access control, audit, network controls are active as they were before.  The virtual server sits on hosts and SANs in the same datacenter as the physical server did before.  So what has changed?  What are the new risks?

It’s in the virtualization infrastructure layer.  (Given the ubiquity of VMWare and my personal knowledge of that solution I’m going to use it to give various virtualization components real world names.)  How secure is the host, the storage and the host control server.  In VMWare terms I mean how secure are the ESX/ESXi hosts, the SANS and the vCenter servers?  That is the real question here.  Now that we’ve inserted a new layer between the guest OS and the hardware that layer’s immediate components and other components they depend on (the Active Directory forest that the vCenter servers belong to) need the same security controls anything else on the work needs.  Virtualization security is even more critical in some respects because someone who compromises that layer has low level access equivalent to physical access to every guest server and its data.

I have observed in recent audits that some areas of security and control of virtualization components is immature and does reflect consciousness how critical the virtualization layer is to security.  Before I dive into those areas let’s first talk about network security.  In the hype I referred to in the introduction, much is made about network security risks associated with virtualization and I’m not really seeing that.  Most servers are not behind internal firewalls or on heavily restricted network segments in the first place so moving them to an ESX/i host on a virtual switch doesn’t expose the server to new network risks.  For servers that had such controls when they were physical, you can set up the very same thing with virtual switches and firewalls.  If you need to mix external facing and internal VMs on a host you can set up completely different virtual switches and that’s what I IT shops doing.  There are more advanced attack scenarios involving a compromised VM where the attacker breaks out of that VM and into the host and possibly back up into other VMs but right now that is one area where we are ahead of the bad guys in the arms race. That is something to stay vigilant about though.

The other area of network security is the protection of the visualization infrastructure itself.  Unless IT shops totally ignore virtualization best practices they are going to implement multiple network cards on ESX/i hosts that allow you to completely separate guest, live migration (aka vMotion), storage and management (connections from VMWare vCenter and clients to the host) traffic.  In the audits I’ve performed, SANs and the management interfaces on hosts are isolated from rest of the organization’s internal LAN. 

This brings me to the areas of virtualization where I have found serious risks: privileged access, Active Directory and auditing.  First privileged access.  Anyone reading this knows that allowing administrators (especially multiple administrators!) use the built-in root or admin account on operating system is bad practice and risky for all sorts of reasons.  Everyone should have their own account.  The same goes for virtualization hosts only more so.  However in my audits I’ve observed many hosts being managed by the built-in root account shared among multiple administrators.  With a virtualization host, the risk of shared and insecure root accounts is multiplied by number and criticality of all the guest VMs on that host.  There’s no good reason for this bad practice and there’s no need to create additional root accounts on hosts for each admin either.  That’s because best practice is to lock down ESX/i hosts so that even admins don’t directly access them and instead go through the central management server (called vCenter in the case of the VMWare environment).  vCenter doesn’t share the same prevalence of insecure root access because vCenter integrates with Active Directory and allows you to leverage the AD accounts admins already have.  If for some reason a shop can’t follow that important best practice of avoiding direct access to hosts then you can configure the hosts themselves to integrate with Active Directory.

Speaking of Active Directory, I’ve uncovered another prevalent risk associated with the dependence of virtualization infrastructures on AD.  Let me be clear that I think directory integration and unified authentication is definitely the way to go but here are some risk factors people have been missing.  First there’s the fact that I often find virtualization management servers like vCenter to be members of the main AD forest.  Here’s what that means.  A Windows server belonging to a domain is exposed to any all risks in Active Directory and all domain controllers within that forest (remember the security boundary in AD is the forest not the domain).  Building on that, a vCenter server is the “boss” of all the ESX/i hosts connected to it.  Ergo, everyone who has domain admin authority anywhere in the AD forest and anyone who compromises a domain controller in the AD forest can take the vCenter server and from there compromise the virtualization infrastructure and ultimately any guest VM and its data.  A point I have to make, sometimes to the chagrin of some folks in IT, is that any outstanding risks from previous AD audits must be carried forward to the virtualization infrastructure audit too.  For instance, at one bank, there was a problem of a very excessive number of IT folks having domain admin authority to the main AD forest.  That was problem enough as a risk to AD and the Windows systems within that forest.  But with the virtualization management server as a member of that forest, now every virtual machine – even those running Linux and Windows servers in other forests are now accessible to that same excessively large group of AD admins.  Worse, in this organization remote access was widely available with no strong authentication.  So the entire virtualization infrastructure and countless servers running on it were vulnerable to compromise by a successful password based attack against any one of many AD admins. 

The solution?  First, think about the AD forest(s) that hold either your virtualization management servers (e.g. vCenter) or those user accounts with privileged access to the virtualization infrastructure (i.e. users with the Administrator role in vCenter).  Those forests, including each domain and domain controller within them, must be locked down and secure to a level appropriate for the virtualization infrastructure itself.  There’s no avoiding that.  Organizations with outstanding AD security issues with no resolution in site should really look at implementing a small, separate AD forest for providing directory and authentication services to their infrastructure including virtualization, storage and network components.  This small AD forest would be much more locked down and protected and careful thought should be given before implementing synchronization or trust relationships between it and other forests.  This may be at the price of maintaining additional user accounts for infrastructure admins but that is the price of security in this instance.  If trust is implemented it should such that the infrastructure forest is trusted by the other forests not vice versa.   If synchronization is implemented, password changes or other authentication data should not flow from other forests or directories into the infrastructure forest.  

The final risk area I’ve observed with newly virtualized organizations is a lack of auditing and log management for virtualization infrastructure components.  And in this case there’s no easy solution.  Virtualization management servers (e.g. vCenter) and hosts (e.g. ESX/i) can generate audit logs and it is crucial to enable this feature and subsequently collect, archive alert and report on this log data the same as is necessary with any other security critical components on your network.  Virtualization hosts like ESX/i are simple to accommodate since they can send events via syslog.  But management servers like vCenter are more problematic.  vCenter creates a number of text log files named vpxd-1 through 9 but my research has proven them to omit very important data and fail to resolve other key ID codes.  That’s not to say the audit trails aren’t there.  They are but they are trapped inside database tables.  In the case of vCenter the audit trail is stored in VPX_EVENT and VPX_EVENT_ARG tables within the vCenter SQL database.   Incomprehensibly command line interfaces like Get-VIEvent that pull data from these leave out critical event arguments as well as the name or ID of the event itself!  So the final option seems to be direct query of the SQL tables themselves with the necessary resolution of foreign keys to rows in related tables.  This presents an opportunity to log management and SIEM vendors to distinguish themselves with enhanced support for collecting enriched audit trails from virtualization infrastructures.

One of the next coming of age moments for audit log management is automating more of the review and response tasks associated with security events.  Right now we tend to expect log management SIEM solutions to scour logs, identity high-impact changes or other suspicious activity and simply alert someone.  It’s up to that someone to assess the information, make inquiries or other research and ultimately resolve the matter.

Instead of creating more manual work for infosec staff and administrators wouldn’t it be great if the system could carry out some of that review, inquiry and decision work on its own?

In this article I’ll present several automation opportunities.  In addition to some simple integration with Active Directory or other LDAP services these scenarios would require a degree of scripting and workflow capabilities like those provided by SharePoint or similar technologies.

#1 Verifying New User Accounts

The appearance of a new user account is an important event to the security of a system since it signifies a new way to gain entry to that system’s resources.  Such new accounts need to be reviewed to prevent the unnoticed creation of backdoor accounts by intruders or simply insecure accounts created outside the change management and security controls of the organization.   But it is laborious for an infosecurity person or admin to wade through new user account notifications and potentially difficult for him or her to identify inappropriate accounts – especially at larger organizations with greater turnover.   

How could this process be automated?  Upon encountering a new user account (e.g. event ID ??? in Windows Server 2008 Active Directory) log management solution would compare the new account’s name to the naming convention standards for the organization.  Naming conventions at most organizations at least distinguish between human user accounts and system accounts created for services, applications and batch processes.  Often there are additional naming standards that make it easy to distinguish between variants like employees, contractors, privileged admin accounts. 

Let’s imagine Acme Corporation prefixes employees with e, contractors with c, admins with p and system accounts with s.  When our automated response rule encounters a new user account beginning with x it would recognize this as a non compliant user account and open a ticket with the team responsible for enforcing security standards. On the other hand when seeing an account beginning with e, it would email the manager for the employee based on the “direct reports” or “department” attribute of the new employee account and ask the manager to attest to this account as corresponding to a new hire in his or her department.  Through many audits performed over the years I’ve seen the value of ensuring that each user account can be tied to an employ record in the HR system so our automated response system may also look up the employee in the HR system ensure proper correlation there as well.  If it finds an exception an ticket would be opened with the same IT group responsible for security control compliance to track down where the disconnect occurred.  As you can see though the amount of manual intervention by IT or other departments is greatly reduced yet the amount of verification is much higher and more thorough than normally possible.  Some organizations may be able to take the automation even further by tying out new account provisioning to new hires in the HR system automatically. 

Inquiries regarding new accounts indicated to be contractors might be routed to the appropriate contract management group in purchasing or to the Manager if that attribute for contractors is populated with the employee who is the main liaison for the contractor.  Such contractor accounts would also be checked to ensure that an expiration date was configured of not more than some defined maximum period such as 3 months.  If the quantity and turnover of contractors warrants it, the system could even look up the contractor in the contract management system to correlate the account and expiration date automatically.  If such correlation fails the system would assign a ticket for the with the appropriate IT group.

By insisting that every account possible be tied to an employee or other business object such as a contract it becomes much more difficult to purposefully sidestep security controls or for accounts to slip through the cracks. 

#2 Verifying Group Membership Additions

Second only to new user accounts in security importance is a member being added to a group which with very few exceptions indicates an expansion of access be granted to the new member.  Such events (???) are likewise excellent opportunities for automated response.  The crucial element in automating new group member response is for our system to be able to identify the data “owner” responsible for approving entitlements to the data to which that group grants access.  Assuming our repository of groups is Active Directory, the obvious attribute is the Managed By property which allows you to specify a user account.  By populating each group’s Managed By attribute with the appropriate data owner we make it easy for our automated response system to determine who to contact when new members are added.  In that email notification the data owner would be asked to confirm that he or she had in fact approved this new entitlement by clicking on an appropriate link.  A positive confirmation would close out the matter and document that re-verification had been performed.  A negative response would open a ticket with the information security group to investigate including all relevant information including the user shown by the audit log to have executed the group member addition.  These are the only cases that would require manual intervention by the information security team, yet each and every group member addition would be confirmed.

#3 Following Up On Suspicious Logon Activity

One of the most difficult events to follow up on are failed logons due to a bad password (???) since they are often caused by a user simply entering the password incorrectly.  These events also present a possible opportunity for automated response.  An information security analyst who had reason to suspect an attack on a given account would typically contact the user to determine if they were inadvertently the source of the audit events.  Likewise out automated response system upon observing failed logons may contact the user similar to the processed described above for group member additions.  Of course in most cases some threshold of logon failures should first be exceeded before contacting the user so that users don’t become wearied by incessant inquires each time they mis-enter their password.  Or perhaps this automated response would only be implemented for privileged accounts with administrator or root access.  In any case consideration should be given to the method of communication used to contact the user.  Since email access may be controlled by the very account under attack it may be better to use an “out of band” communication such as sending a text message to the user’s phone again illustrating the important of being able to leverage information from the organizations directory when responding to events. 

IP Geolocation could be leveraged as well to help identify suspicious logon attempts.  When our system observes a logon attempt (successful or failed) from a country or region outside the normal area defined either for the organization as a whole or for the user individually (if physical location data is stored with the user’s directory information) the system could send message to the user asking them to contact information security if they do not recognize the activity.

These are just 3 examples of security events for review and response could be automated.  Of course the ability to script such automated responses, access to the directory service, HR data, IT ticketing and some type of workflow system are important requirements but by careful analysis organizations can identity those operations which are important and/or frequent enough for which such an investment would pay off.  Through such integration and automation you can accomplish increased vigilance while eliminating the manual effort required to follow up on the myriad security events generated every day at any organization. 

Introducing:  LOGbinder SQL - SQL Audit Policy Wizard

Our totally free SQL Audit Policy Wizard steps you through the process of implementing SQL Server 2008 auditing. You can use our recommended baseline audit policy or customize it to fit your requirements.

After selecting your SQL Server and fine tune your desired audit policy, SQL Audit Policy Wizard automatically creates the necessary Server Audit and Server Audit Specification objects on your SQL server and optionally enables them so that auditing begins automatically.

You can also see the actual Transact-SQL generated by the wizard for learning purposes or for further customization. SQL Server 2008 Audit Policy Wizard even allows you to modify existing audit objects.

Get the wizard now, for free - no trialware expiration, etc.

Much of the security and control of an enterprise IT environment rests on Active Directory. It provides authentication and access control for Windows users and applications, as well as for UNIX, Linux and mainframes. Even VPNs, extranets and internal network security technologies all use Active Directory for policy and identity information.

To comply with information security best practices and compliance requirements, Active Directory must be regularly monitored and audited. However, if you've spent any time with the the native Windows security log you know that it provides only limited Active Directory audit capabilities.

Just one example is the fact that while the Windows security can tell you that a Group Policy Object was edited, it cannot tell you which of the nearly 1,000 settings were changed! And don't get me started on the noisy events generated and important information not logged when you try to audit Windows file sytem access.

In this whitepaper sponsored by Quest Software I explore the gaps in native Windows auditing that prevent organizations from achieving full compliance with best practices and security requirements. In fact I identity specific requirements in PCI, SOX and FISMA that make effective audit log management of Active Directory and the larger Windows environment mandatory.

Then I provide a brief tour of Quest's new On Demand Log Management and explain how this unique cloud based solution not only provides extremely easy to deploy core log management functions but also goes much further than typical log management solutions to bridge the gaps in native Windows auditing like the one described above involving group policy.

Read this whitepaper and learn:

• Where are the serious gaps in native Windows auditing?
• Which compliance requirements do these gaps prevent you from meeting?
• How does Quest On Demand Log Management bridge these gaps by replacing certain native audit funcitons?
• How can you deploy log management and achieve compliance with a fraciton of the effort required by traditional software?

Get the paper now:  Click Here

I'm excited to announce that my software company, LOGbinder, has just released LOGbinder SQL as beta.  If you need audit logging for SQL Server you will be interested to know about SQL Server 2008's new audit foundation and how LOGbinder SQL allows you to connect SQL's audit capability to your existing SIEM/log management solution:

Introducing LOGbinder SQL

SQL Server 2008 introduced a totally new audit logging facility, which is critical to enterprises storing sensitive information and/or processing important transactions in today’s demanding compliance environment. SQL Server Audit is flexible in terms of audit policy and comprehensive in relation to the breadth and depth of objects and actions that can be audited. However, the audit data generated by SQL Server needs additional refinement and processing before it can be relied up on as a usable audit trail and be managed by your existing log management/SIEM solution.

The audit records generated by SQL Server audit are cryptic and difficult to understand. Basically, one log record format is used for documenting everything from an insertion on a table to giving a user ownership rights to a database. And while SQL Server can write events to the security log, it uses the same event ID for all events, and the IDs and keywords are not resolved. Thus, it requires in-depth knowledge of the SQL audit model in order to decipher events.

Our LOGbinder SQL agent enriches SQL Server’s cryptic and generic audit messages to produce easy-to-understand audit log events. Similar to LOGbinder SP, these events can be outputted to the Security log a custom Windows event log, where any log management or SIEM solution can collect, alert, report, and analyze. Here is an example of an event:

Raw Audit Event from SQL Server

event_time:2010-09-16 12:35:30.0787755
sequence_number:1
action_id:APRL
succeeded:true
permission_bitmask:0
is_column_permission:false
session_id:54
server_principal_id:260
database_principal_id:1
target_server_principal_id:0
target_database_principal_id:0
object_id:7
class_type:RL
session_server_principal_name: ACMESP\Administrator
server_principal_name: ACMESP\Administrator
server_principal_sid:0
database_principal_name: dbo
target_server_principal_name: ACMESP\Administrator
target_server_principal_sid: 0
target_database_principal_name: public
server_instance_name: SPDEV\SQL08ENT
database_name: AuditTest
schema_name:
object_name: MyAudit
statement: EXEC sp_addrolemember N'MyAudit', N'public'
additional_information:
file_name=c:\sql audits\AuditAll_12633920-
FB34-4FAA-8F96-E9F8FED158A9_0_ 129276798828120000.sqlaudit
audit_file_offset=1536

Same Event After LOGbinder SQL Processing

Event ID: 24020
Add member to database role succeeded
A principal was successfully added to a database role
Action Group: DATABASE_ROLE_MEMBER_CHANGE_GROUP
Occurred: 9/16/2010 12:35:30.0000000 PM
Session ID: 54
User: ACMESP\administrator
Server: SPDEV\SQL08ENT
Database: AuditTest
Member
Name: public
Domain name: n/a
Role
ID: 7
Name: MyAudit
Statement: EXEC sp_addrolemember N'MyAudit', N'public'

*Learn more about LOGbinder SQL and download the beta today! Click Here.

Intrusion detection and compliance are often the focus of log management/SIEM efforts and security logging in general.  But security logs (when managed correctly) are also the only control over rogue admins.  Why do I say that?  Well, once you give someone root or admin authority there is nothing they cannot do.  They can circumvent any access or authorization controls you implement on the system because with admin authority they can either directly change those settings or use hacker tools that leverage their root access to tamper with the internals of the operating system.  Beyond rogue internal admins, the same issues I bring up in this article apply to the risk of outside attackers who gain admin authority (aka root access).

The only control you really have over privileged super users is an audit log that, if properly handled, can serve as a deterrent and possibly detective control against misuse of privileged authority.  But notice I say “if properly handled”.  Those 3 words serve as in important warning that simply enabling auditing and deploying a log management solution may not suffice to provide a solid deterrent to rogue admins. 

To really be a deterrent, the audit log must be protected from deletion or tampering by rogue admins.  First and foremost that means you must move log data as frequently as possible from the system where it is generated to separate secure log repository.

Today’s enterprise log management solutions do a great job of frequent log collection and long term archival.  However, who has privileged access to your log management solution and the systems on which it runs?

A log management process is not an effective control over administrators if those same administrators have privileged access to the log management components.  To be clear I’m not suggesting that administrators be denied access to run reports, configure alerts or research logs.  I’m talking about privileged access to the log management solution that allows someone to disable, erase or otherwise compromise the integrity of the log collection and archival process.

So, to reiterate: a log management solution cannot serve as a deterrent over administrators if those same administrators have privileged access to it at the application level or any of the infrastructure components on which it runs.  That includes:

·         the operating system of the log management solution

·         any database servers it uses

·         the Active Directory forest in which the log manage server resides if a Windows server

·         the NAS or SAN where it stores log data

Moreover if the log management application or any of the components above run inside a virtual machine you should also include:

·         the virtualization host, such as VMWare ESX(i), if it runs inside a virtual machine

·         the virtualization manager, such as VMWare vCenter

·         any of the components listed earlier which are used by or host the virtualization manager

And of course physical access to any of those components would also potentially allow administrators to compromise the integrity of your audit trail.  Wow!  Does that mean your log management solution needs to run on a completely separate infrastructure?  To the extent possible, absolutely.  It certainly speaks to at least consider building your log management components from traditional physical servers and local disk storage.

And remember such separation is a protection against not just internal rogue admins but any outsider that succeeds in obtaining privileged access well.  Any outsider sophisticated enough to achieve that will also make efforts to erase their tracks.

Of course there are always cost/benefit comparisons to be made.  Typically the larger the organization the more important and more practical it is to achieve maximum separation between the log management solution and the environment it monitors. 

Beyond hardware and software separation, you must also consider who will provide the care and feeding for the log management application, database servers, storage, OS and other components.  Larger organizations have a dedicated information security team.  Within that team there should be a group with responsibility for the audit log management process.  For full accountability and separation of duty, that team should have no privileged access to production business systems monitored by the log management process.  Ideally that group would provide the care and feed necessary for all components in the log management solution.  Of course that may be impractical since information security professionals on that team may not be conversant in all the technologies involved.  In that case they should call upon the expertise of IT staff with the required skills as needed and ideally they would supervise the actions taken by such staff to ensure they do not compromise audit log integrity such as through the introduction of backdoors.

There are a host of reasons why even the “supervised access” method described above may not work.  Staff in smaller IT shops have to back each other up and often can’t specialize so the possibility for separation of duty like that described above may not exist.  Or, there may be a dedicated security and compliance officer, but that person may not be technical enough to supervise IT staff when maintenance is required on log management components.  Finally in some organizations there may be great push back to carving out a separate physical hardware environment for audit log management.  

When an in-house physically and logically separate log management system isn’t possible organizations, log management as a service may be an interesting alternative.  With cloud-based log management the entire log management system is at another site under the control of a professional service team.  Furthermore, some services can be set up with role based access control so that the ability to erase audit logs is controlled.  If organizations can overcome the frequent pushback to shipping audit logs to the cloud, full isolation and integrity of log data can be achieved without building a separate log management system and without needing system care and feeding expertise within the team responsible for audit log management.

Whether an organization goes with an in-house audit log management or turns the cloud it should carefully assess whether its choices in architecture and administrative responsibility allow it to rely on the log management system as a deterrent/detective control over rogue admins and as a means of impact analysis and recourse against outside intruders who gain root access.  When the worst happens, audit logs may be all you have.  Are they secure?

Maybe Quiet Riot isn’t your band (OK, the song was originally a 1973 Slade hit but come on, I was only 3) but that line describes the Windows security log so well.  There’s no getting around the fact that there are a lot of useless and inexplicable events in the Security log and the sooner you get comfortable with that the sooner you’ll save your sanity and get on with work.  In this article I’ll look at some common examples or noise events in the security and discuss strategies for dealing with them. 

You need to recognize noise events for what they are for 2 reasons.  First the earlier in the log management process you can discard or ignore noise events the better for performance, bandwidth and storage.  Second, you can waste a lot time investigating seemingly suspicious events that are meaningless. 

One example of high volume noise events are Kerberos service ticket renewal attempts generated on domain controllers.  Kerberos has 2 kinds of tickets – authentication tickets (aka ticket granting tickets) and service tickets.  Authentication tickets (see event IDs 4768, 4771 and 4772 on Windows 2008 and 627, 675 and 676 on Windows 2003) are connected with the actual authentication of the user (or computer) to the domain controller while service tickets vouch for that user’s identity to the member computers the user subsequently accesses on the network. 

When a user remains logged on (or a computer remains up) long enough the service tickets expire and Windows needs to renew them with the domain controller.  A successful service ticket event (event ID 673 on Windows 2003 and 4769 on Windows 2008) is useful in that it provides a record of the workstation and servers a user accessed.  But a successful ticket renewal (event ID 674 on Windows 2003 and 4770 on Windows 2008) denotes nothing of value other than the fact that the user or computer remained logged on or powered up for a long time.  If a user remains logged on (or a computer remains up without being rebooted) for extra-long, the service ticket reaches it renewal lifetime limit and the domain controller finally rejects the renewal request which generates a renewal failure.  There are other scenarios but at the end of day any kind of service ticket renewal (successful or failure) and any kind of service ticket request failure event is essentially noise.  There are some theoretical malicious situations where service ticket events could conceivably be generated but in practice they are very unlikely and there are no criteria you could use to distinguish those events from all the background noise of other service ticket events. 

An example of a seemingly suspicious event that is really just noise is the sometimes frequent occurrences of event ID 537, “Logon failure - The logon attempt failed for other reasons”, where user name is blank.  I frequently hear from concerned admins who are worried they are under some kind of attack.  In this case look the sub status code in the event description and you will probably see 0xC0000133 “The time at the primary domain controller is different from the time at the backup domain controller or member server by too large an amount”.  No security issue here – just a time sync problem on the initiating computer or the domain controller. 

Dealing with the quantity and variety of events you encounter can be overwhelming but here’s a strategy that will keep you on track and prevent you from wasting too much time on noise events.

First, identify what you know to be noise events – like the examples described above.  Configure your log management / SIEM solution to suppress or filter those events out from any alerts you receive or reports that you review on a regular basis.  Make sure you document your justification for filtering those events – that is, why they are considered noise or unimportant. 

Next set up alerts and reports for events you 1) know to exist and 2) consider important enough to generate an alert or show up in a report you review every day.  There may be other events that don’t deserve any kind of response but which you want to keep around for compliance purposes.  If you aren’t already archiving entire logs – noise and all - make sure that your log management process results in these boring but necessary events being archived.

After those steps the only events left over are events you don’t already know about and thus have not classified as noise, important or “boring but necessary”.  Whatever your log management processes and technology there should be a way to view any such “unclassified” events.  Periodically reviewing these events will prompt revisions to your criteria for the classification types described above with the eventual goal of no unclassified events. 

This strategy ensures you don’t indefinitely miss unknown but important events and provides a systematic way for managing noise.  While the Windows security log graciously logs all events from a finite set of IDs which I’ve documented at www.ultimatewindowssecurity.com/securitylog/encyclopedia.aspx, many other logs, like those from Linux and Unix, have no well-defined and bounded event schema.  The strategy described in this article is even more useful with such logs where you never know exactly what’s going to be thrown your way.  And I can tell you that it works because it’s one of the core concepts I use in my log management consulting projects and in my Rosetta Audit Logging Kits.

All things considered you have to get comfortable with noise events in your logs.  Through audit policy refinements you may be able to reduce some of the useless events written but there’s no way to configure the noise completely out of your logs.  Windows audit policy just isn’t granular or flexible enough – even with the new subcategory audit policy structure released in Windows Server 2008.  The same goes for other log sources.  So, identify what you want from the log and don’t get too hung up on everything else.  The security log isn’t like a financial audit trail where every transaction and penny can and should be justifiable.  There are some events in the security log you’ll just never be able to run to earth but don’t worry about it.