Tag Archives: internal audit

Debugging security

There is an interesting analogy between between debugging software and debugging the security of your systems.

As Brian W. Kernighan and Rob Pike wrote in “The Practice of Programming

As personal choice, we tend not to use debuggers beyond getting a stack trace or the value of a variable or two. One reason is that it is easy to get lost in details of complicated data structures and control flow; we find stepping through a program less productive than thinking harder and adding output statements and self-checking code at critical places. Clicking over statements takes longer than scanning the output of judiciously-placed displays. It takes less time to decide where to put print statements than to single-step to the critical section of code, even assuming we know where that is. More important, debugging statements stay with the program; debugging sessions are transient.

In programming, it is faster to examine the contents of a couple of variables than to single-step through entire sections of code.

Collecting security logs is key to information security management not only for understanding what and why an event happened but also in order  to  prove regulatory compliance with regulations such as the HIPAA security rule. The business requirements are that   security logs  should be both relevant and effective.

  1. Relevant content of audit controls:  For example, providing a  detailed trace of an application whenever it elevates privilege in order to execute a system level function.
  2. Effective audit reduction and report generation:  Given the large amount of data that must be analyzed in security  logs, its crucial that critical events are separated from normal traffic and that concise reports can be produced in real-time to help understand  what happened, why it happened and how it was mediated and how to mitigate similar risks in the future.

In security log analysis, it is faster and definitely more effective for a security analyst to examine the contents of a few real time events than to process gigabytes or terabytes of security logs (the equivalent of stepping through or placing watch points in sections of of a sub-modules with  hundreds or thousands of lines of code.

When you have to analyze security logs, it is easy to get lost in details of complicated data and flows of events and find yourself drifting off into all kinds of directions even as the bells go on in the back of your mind that you are chasing ghosts in a futile and time-consuming exercise of investigation and security event debugging.

In order to understand this better, consider another analogy, this time from the world of search engines.

Precision and recall are key to effective security log analysis and effective software debugging.

In pattern recognition and information retrievalprecision is the fraction of retrieved instances that are relevant, while recall is the fraction of relevant instances that are retrieved. Both precision and recall are therefore based on an understanding and measure of relevance. When a program for recognizing the dogs in a scene correctly identifies four of the nine dogs but mistakes three cats for dogs, its precision is 4/7 while its recall is 4/9. When a search engine returns 30 pages only 20 of which were relevant while failing to return 40 additional relevant pages, its precision is 20/30 = 2/3 while its recall is 20/60 = 1/3. See Precision and recall in the Wikipedia.

In other words – it doesn’t really matter if you have to analyze a program with 100,000 lines of code or a log file with a terabyte of data – if you have good precision and good recall.

The problem is however, that the more data you have, the more difficult it is to achieve high precision and recall and that is why real-time events (or  debugging statements) are more effective in day-to-day security operations.

 

Tell your friends and colleagues about us. Thanks!
Share this

10 guidelines for a security audit

What exactly is the role of an information security auditor?  In some cases, such as compliance  by Level 1 and 2 merchants with PCI DSS 2.0,  external audit is a condition to PCI DSS 2.0 compliance.   In the case of ISO 27001, the audit process is a key to achieving ISO 27001 certification (unlike PCI and HIPAA, ISO regards certification, not compliance as the goal).

There is a gap between what the public expects from an auditor and how auditors understand their role.

Auditors look at transactions and controls. They’re not the business owner and the more billable hours, the better.

The “reasonable person” assumes that the role of the security auditor is to uncover vulnerabilities, point out ways to improve security and produce a report that will enable the client to comply with relevant compliance regulation. The “reasonable person” might add an additional requirement of a “get out of jail free card”, namely that the auditor should produce a report that will stand up to legal scrutiny in times of a data security breach.

Auditors don’t give out “get out of jail” cards and audit is not generally part of the business risk management.

The “reasonable person” is a legal fiction of the common law representing an objective standard against which any individual’s conduct can be measured. As noted in the wikipedia article on the reasonable person:

This standard performs a crucial role in determining negligence in both criminal law—that is, criminal negligence—and tort law. The standard also has a presence in contract law, though its use there is substantially different.

Enron, and the resulting Sarbanes-Oxley legislation resulted in significant changes in accounting firms’ behavior,but judging from the 2009 financial crisis from Morgan Stanley to AIG, the regulation has done little to improve our confidence in our auditors. The numbers of data security breaches are an indication that the situation is similar in corporate information security.  We can all have “get out of jail” cards but data security audits do not seem to be mitigating new risk from tablet devices and mobile apps. Neither am I aware of a PCI DSS certified auditor being detained or sued for negligence in data breaches at PCI DSS compliant organizations such as Health Net where 9 data servers that contained sensitive health information went missing from Health Net’s data center in Rancho Cordova, California. The servers contained the personal information of 1.9 million current and former policyholders, compromising their names, addresses, health information, Social Security numbers and financial information.

The security auditor expectation gap has sometimes been depicted by auditor organizations as an issue to be addressed  by educating users to the audit process. This is a response not unlike the notion that security awareness programs are effective data security countermeasures for employees that willfully steal data or bring their personal device to work.

Convenience and greed tend to trump awareness and education in corporate workplaces.

Here are 10 guidelines that I would suggest for client and auditor alike when planning and executing a data security audit engagement:

1. Use an engagement letter every time. Although the SAS 83 regulation makes it clear that an engagement letter must be used, the practical reason is that an engagement letter sets the mutual expectations, reduces risk of litigation and by putting mutual requirements on the table – improves client-auditor relationship.

2.Plan. Plan carefully who needs to be involved, what data needs to be collected and require input from C-level executives to  group leaders and the people who provide customer service and manufacture the product.

3. Make sure the auditor understands the client and the business.  Aside from wasted time, most of the famous frauds happened where the auditors didn’t really understand the business.   Understanding the business will lead to better quality audit engagements and enable the auditor and audit manager to be peers in the boardroom not peons in the hallway.

4. Speak to your predecessor.   Make sure the auditor talks to the people who came before him.  Speak with the people in your organization who did the last data security audit.   Even if they’ve left the company – it is important to understand what they did and what they thought could have been improved.

5. Don’t tread water. It’s not uncommon to spend a lot of time collecting data, auditing procedures and logs and then run out of time and billable hours, missing the big picture which is” how badly the client organization could be damaged if they had a major data security breach”. Looking at the big picture often leads to audit directions that can prevent disasters and  subsequent litigation.

6. Don’t repeat what you did last year.  Renewing a 2,000 hour audit engagement that regurgitates last years security check list will not reduce your threat surface.  The objective is not to work hard, the object is to reduce your value at risk, comply and …. get your “get out of jail card”.

7. Train the client to fish for himself.   This is win-win for the auditor and client. Beyond reducing the amount of work onsite, training client staff to be more self sufficient in the data collection and risk analysis process enables the auditor to better assess client security and risk staff (one of the requirements of a security audit) and improves the quality of data collected since client employees are the closer to actual vulnerabilities and non-compliance areas than any auditor.

As I learned with security audits at telecom service providers and credit card issuers, the customer service teams know where the bodies are buried, not a wet-behind-the-ears auditor from KPMG.

8. Follow up on incomplete or unsatisfactory information.  After a data security breach, there will be litigation.  During litigation, you can always find expert testimony that agrees with your interpretation of information but

The problem is not interpreting the data but acting on unusual or  missing data.  If your ears start twitching, don’t ignore your instincts. Start unraveling the evidence.

9. Document the work you do.  Plan the audit and document the process.  If there is a peer review, you will have the documentation showing the procedures that were done.  Documentation will help you improve the next audit.

10. Spend some time evaluating your client/auditor.   At the end of the engagement, take a few minutes and interview your auditor/client and ask performance review kinds of questions like: What do think your strengths are, what are your weaknesses?  what was succesful in this audit?  what do you consider a failure?   How would you grade yourself on a scale of 10?

Perhaps the biggest mistake we all make is not carefully evaluating the potential we have to meet our goals as audit, risk and security professionals.

A post-audit performance review will help us do it better next time.

Tell your friends and colleagues about us. Thanks!
Share this

Getting managers out of denial

Maybe you have a manager in denial?

I know the feeling -the absence of effective risk controls for data security often start with denial of vulnerabilities. Here is a true story:

…I’m not concerned about data theft. We’ve outsourced our entire IT operation to a big bank’s data center and they’re up to speed on information security. I can always go back to the logs and figure it out if something happens.
Vice President Internal Audit of a private banking institution with $5BN in assets.

Just 2 months later, the “big bank” had a major data theft event. Both banks missed their earnings estimates and took a beating in the market. Today the private institution is trying to break out of their 5 year outsourcing contract.

Moral of the story don’t be a yes-person.

Tell your friends and colleagues about us. Thanks!
Share this