As Brian W. Kernighan and Rob Pike wrote in “The Practice of Programming“

As personal choice, we tend not to use debuggers beyond getting a stack trace or the value of a variable or two. One reason is that it is easy to get lost in details of complicated data structures and control flow; we find stepping through a program less productive than thinking harder and adding output statements and self-checking code at critical places. Clicking over statements takes longer than scanning the output of judiciously-placed displays. It takes less time to decide where to put print statements than to single-step to the critical section of code, even assuming we know where that is. More important, debugging statements stay with the program; debugging sessions are transient.

In programming, it is faster to examine the contents of a couple of variables than to single-step through entire sections of code.

Collecting security logs is key to information security management not only for understanding what and why an event happened but also in order  to  prove regulatory compliance with regulations such as the HIPAA security rule. The business requirements are that   security logs  should be both relevant and effective.

1. Relevant content of audit controls:  For example, providing a  detailed trace of an application whenever it elevates privilege in order to execute a system level function.
2. Effective audit reduction and report generation:  Given the large amount of data that must be analyzed in security  logs, its crucial that critical events are separated from normal traffic and that concise reports can be produced in real-time to help understand  what happened, why it happened and how it was mediated and how to mitigate similar risks in the future.

In security log analysis, it is faster and definitely more effective for a security analyst to examine the contents of a few real time events than to process gigabytes or terabytes of security logs (the equivalent of stepping through or placing watch points in sections of of a sub-modules with  hundreds or thousands of lines of code.

When you have to analyze security logs, it is easy to get lost in details of complicated data and flows of events and find yourself drifting off into all kinds of directions even as the bells go on in the back of your mind that you are chasing ghosts in a futile and time-consuming exercise of investigation and security event debugging.

In order to understand this better, consider another analogy, this time from the world of search engines.

Precision and recall are key to effective security log analysis and effective software debugging.

In pattern recognition and information retrieval, precision is the fraction of retrieved instances that are relevant, while recall is the fraction of relevant instances that are retrieved. Both precision and recall are therefore based on an understanding and measure of relevance. When a program for recognizing the dogs in a scene correctly identifies four of the nine dogs but mistakes three cats for dogs, its precision is 4/7 while its recall is 4/9. When a search engine returns 30 pages only 20 of which were relevant while failing to return 40 additional relevant pages, its precision is 20/30 = 2/3 while its recall is 20/60 = 1/3. See Precision and recall in the Wikipedia.

In other words – it doesn’t really matter if you have to analyze a program with 100,000 lines of code or a log file with a terabyte of data – if you have good precision and good recall.

The problem is however, that the more data you have, the more difficult it is to achieve high precision and recall and that is why real-time events (or  debugging statements) are more effective in day-to-day security operations.

In order to determine how encryption fits into server data protection, consider 4 encryption components on the server side: passwords, tables, partitions and  inter-tier socket communications.

In these 4 components of a application / database server encryption policy, note that some countermeasures are required (for example one-way hashes of passwords, while other such as encrypting specify table columns may or may not be relevant to a particular application).

1. Encrypted password storage

You must encrypt passwords. It’s surprising to me how many Web sites don’t bother encrypting user passwords – See cases Universal Music Portugal where e-mail addresses and clear-text passwords are dumped on Internet.

What is more surprising is the confusion between encryption and hashing.

Don’t use AES for encrypting passwords in your MySQL or Oracle or MS SQL database.  You’ll end up storing the AES key somewhere in the code and an attacker or malicious insider can read the key by opening up one of your application DLLs in Notepad++ and read that key in a jiffy and breach your entire MySQL database with a single SELECT statement.

Database user passwords should be stored as MD5 hashes, so that a user  (such as a DBA) who has been granted SELECT access to the table (typically called ‘users’)  cannot determine the actual password. Make sure that different instances have different salts and include some additional information in the hash.

If you use MD5 encryption for client authentication, make sure that  the client hashes the password with MD5 before sending the data on the network.

2. Encrypt specific database table columns

The PostgreSQL 9.1 pgcrypto module allows certain fields to be stored encrypted. This is especially useful if some of the data is sensitive for example in the case of ePHI where the Web application needs to comply with the CFR 45 Appendix A Security rule. The client software provides the decryption key and the data is decrypted on the server and then sent to the client.  In most cases the client (a database driver in an MVC application such as Ruby on Rails or CakePHP or ASP.NET MVC is also a server side resource and often lives on the same physical server as the database server. This is not a bad thing.

3. Encrypt entire data partitions

Encrypting entire data partitions has its place.

On Linux, encryption can be layered on top of a file system using a ”loopback device”. This allows an entire file system partition to be encrypted on disk, and decrypted by the operating system. Many operating systems support this functionality, including Windows.

Encrypting entire partitions is a security countermeasure for physical attacks, where the entire computer is stolen. Research we did in 2007 indicated that almost 50% of large volume data breaches employed a physical attack vector (stealing a notebook at a hotel checkin desk, hijacking a truck transporting backup tapes to Iron Mountain and smash and grab jobs where thieves know the rent-a-cop walkaround schedule and break in and steal desktop computers.

On the other hand, once the volume is mounted,  the data is visible.

4. Encrypt socket communications between server tiers

SSL has it’s place, although SSL is not a silver bullet countermeasure for Microsoft Windows vulnerabilities and mobile medical devices vulnerabilities as I wrote here, here and here.

SSL connections encrypt all data sent across the network: the password, the queries, and the data returned. In database client-server connections,  relational database systems such as PostgreSQL allow administrators to specify which hosts can use non-encrypted connections (host) and which require SSL-encrypted connections (hostssl). Also, clients can specify that they connect to servers only via SSL. Stunnel or SSH can also be used to encrypt transmissions.

A friend who is a commercial pilot had his bag swiped literally behind his back while waiting on line to check-in to a 4 star Paris hotel. The hotel security cameras show the thief moving quickly behind his back, quietly taking the bag and calmly walking off.

Is your user password 123456?

The Wharton School at UPenn recently posted an article – is your password 123456?

As the article notes – “Hack attacks have recently hit government agencies, news sites and retailers ranging from the U.S. Justice Department and Gawker to Sony and Lockheed Martin, as hackers become more sophisticated in their ability to steal customers’ identities and personal information.”

But, you don’t need sophisticated hack attacks to know that many people use simple minded passwords like 123456 and thieves use simple techniques like grab and run.

So – why don’t we all use strong passwords?

Every Web site and business application you use has a  different algorithm and password policy.  For users, who need to maintain strong passwords using 25 different policies on 25 different systems and web sites, it’s impossible to maintain a strong password policy without making some compromises.

The biggest vulnerability is using your corporate password on an online porn site.  Since adult sites are routinely subject to attack and cheesier, more marginal adult sites – (mind you we’re not talking Penthouse.com or Playboy.com perish the thought) are frequently unwitting malware distribution platforms.

Here are 5 rules for safe password management :

1. Use technical aids to manage your passwords.  Consider using Keepass password management
2. Match password  strength to asset value. In other words – use a complex combination of letters and numbers for online banking and a simple easy to remember password for Superball news.
3. Don’t reuse.   Don’ use the same strong password on more than one sites.
4. Make passwords easy to remember but hard to guess.  Adopt mnemonics – like 4Tshun KukZ that you can remember
5. Maintain physical security of your passwords.  Treat your passwords like you treat the cash in your wallet.  If you have to write passwords down, put them on a piece of paper in your wallet and treat that piece of paper like a $100 bill,  make sure you don’t lose that wallet.

The FBI have been after Mr Dotcom for 8 years. His big problem was not the file sharing but his other criminal activities.  After all, there is infinite demand for file sharing,  Filesonic is cleaning up now that Megaupload went bust and Viacom didn’t go after Erich Schmidt as Viacom lost their billion dollar copyright case to Google 2 years ago.

But really – beyond the consumer appetite for entertainment, and corporate appetite for filing intellectual property and copyright suites, why isn’t Hollywood getting it right when it comes to content protection?  If they were getting it right, Sony-Columbia would be running the file sharing sites, charging $1/movie and $3 for premium content and driving all the file sharing sites out of business.

Instead – the big studios are making the same mistake that corporate America makes when it comes to content protection – ignoring the attacker economics.

After all, the HDCP black-listing scheme defies the laws of physics and reason. For example, you may be a perfectly law-abiding citizen, but if someone in Sofia hacks your model XY500 DVD player, the device key is revoked, and you will never be able to play discs that came out after the date the device was compromised. If a hacker taps into the HDMI / HDCP signal copies a movie enroute to your model TV Set, the HDCP device key can be revoked and your 80 inch TV will never play high-definition again.

Blu-Ray copy protection was broken 5 years this month (January 2007) (Courtesy of muslix64, the same fellow who cracked HD-DVD). Both HD DVD and Blu-ray use HDCP (High-Bandwidth Digital Content Protection) for authentication and content playing, and both use the AACS (Advanced Access Content System) for content encryption. (AACS is the content protection for the video on DVDs and HDCP is the content protection on the HDMI link between the DVD player and the TV). It appears that muslix64 took a snapshot in memory of a running process, then used selective keying – serially trying bytes 1-4, then 2-5, 3-6 etc as the keys until the MPEG frame decrypted. (much faster than a pure brute force attack). If the video player process stores the key in clear text in memory, this type of attack will always work.

Like most flawed encryption schemes, AACS is vulnerable to threats to due a poor software implementation.

” The AACS design prevents legitimate purchasers from playing legitimately purchased content on legitimately purchased machines, and fails to prevent people from ripping the content and sharing it through bittorrent. The DRM people wanted something that could not be done, so unsurprisingly they winded up buying something that does not do it”

James Donald.

Now we understand why BitTorrent is so popular and why

As Ben noted that in the clinical research field, putting together such an offering is not trivial. Open Clinica is the worlds fastest growing clinical trials software with an interesting Open Source business model of community-supported Open Source and revenue from enterprise licensing, cloud services and training.

Software Associates specializes in helping medical device vendors achieve HIPAA compliance and improve the data and software security of their products in hospital and mobile environments.  We have been working with a regulatory affairs consulting client for over 3 years now, using the Open Clinica application for managing  large multi-center, international clinical trials using Rackspace hosting and more recently using Rackspace Cloud.

I can attest that running multi-center clinical trails in the cloud is neither for the faint of heart nor weak of stomach.  Past the security, compliance and regulatory issues – there is also the issue of performance.

Although resources are instantly scalable on-demand in the cloud, resources are not a substitute for secure software that runs fast.

As I noted in a previous essay “The connection between application performance and security in the cloud“, slow applications require more hardware, more database replication, more load-balancing and more firewalls. More is not always better, and more layers of infrastructure increase the threat surface of the application with more attack points on the interfaces and more things that can go wrong during software updates and system maintenance.

If there is a design or implementation flaw in a cloud application for clinical trials management that results in the front-end Web server making 10,000 round trips to the back-end database server to render a matrix of 100 subjects, then throwing more hardware at the application will be a fruitless exercise.

If we do a threat analysis on the system, we can see that our No. 1 attacker is the software itself.

In that case, the application software designers have to go back to the drawing board and redesign the software and get that number down to 1 or 2 round trips.

The effort will be well worth it in your next bill from your cloud service provider.

Studies suggest that 30-50 percent of patients are likely to give up treatments early.  Microsoft Research has developed an innovative, hand-held medical device called Anatonme to help patients understand their issue and complete their treatment plan more often.

We’ve been doing research and development into private, controlled social networking to reinforce private communications between doctor and patient. It’s gratifying to see Microsoft Research doing work in this area.

Private social networking for doctors and patients provides highly effective secure data sharing between doctors and patients. It allows patient-mediated input of data before visits to the office, making the clinical data more accurate and complete and boosting the trust between doctor/healthcare worker and patient.

A private social network has a controlled 1 to N (doctor to patients) topology and physiological and emotional context, unlike Facebook that has a distracting social graph and entertainment context.

A private social network for doctors and patients also provides powerful information exchange and search:

1. Capture critical events on a timeline (for example blood pressure, dizziness etc) that enables the doctor to respond in a timely fashion.
2. Reconciles differences between what the doctor ordered and what the patient did.
3. Granular access control for sharing of data between doctor, patient and referrals.

If you’re interested in hearing more – contact us.

1. A robust, cost-effective security portfolio based on attack analysis  results in robust compliance over time.
2. Executives related well to the concepts of threat modeling / attack analysis. Competing, understanding the value of their assets, taking risks and protecting themselves from attackers is really, at the end of the day why executives get the big bucks.

As I wrote in a previous essay “The valley of death between IT and security“, there is a fundamental disconnect between IT operations (built on maintaining predictable business processes) and security operations (built on mitigating vulnerabilities).

Business executives delegate information systems to IT and information security to security people on the tacit assumption that they are the experts in information systems and security.  This is a necessary but not sufficient condition.

In the current environment of rapidly evolving types of attacks (hacktivisim, nation-state attacks, credit card attacks mounted by organized crime, script kiddies, competitors and malicious insiders and more…), it is essential that IT and security communicate effectively regarding the types of attacks that their organization may face and what is the potential business impact.

If you have any doubt about the importance of IT and security talking to each other, consider that leading up to 9/11, the CIA  had intelligence on Al Qaeda terrorists and the FBI investigated people taking flying lessons, but no one asked the question why Arabs were learning to fly planes but not land them.

With this fundamental disconnect between 2 key maintainers of information protection, it is no wonder that organizations are having difficulty effectively protecting their assets – whether Web site availability for an online business, PHI for a healthcare organization or intellectual property for an advanced technology firm.

IT and security  need a common language to execute their mission, and I submit that building the security portfolio around most likely threat scenarios from an attacker perspective is the best way to cross that valley of death.

There seems to be a tacit assumption with many executives that regulatory compliance is already a common language of security for an organization.  Compliance is a good thing as it drives organizations to take action on vulnerabilities but compliance checklists like PCI DSS 2.0, the HIPAA security rule, NIST 800 etc, are a dangerous replacement for thinking through the most likely threats to your business.  I have written about insecurity by compliance here and here.

Let me illustrate why compliance control policies are not the common language we need.

PCI DSS 2.0 has an obsessive preoccupation with anti-virus.  It does not matter if you have a 16 quad-core Linux database server that is not attached the Internet with no removable device nor Windows connectivity. PCI DSS 2.0 wants you to install ClamAV and open the server up to the Internet for the daily anti-virus signature updates. This is an example of a compliance control policy that is not rooted in a probable threat scenario that creates additional vulnerabilities for the business.

Now, consider some deeper ramifications of compliance control policy-based security.

When a  QSA or HIPAA auditor records an encounter with a customer, he records the planning, penetration testing, controls, and follow-up, not under a threat scenario, but under a control item (like access control). The next auditor that reviews the  compliance posture of the business  needs to read about the planning, testing, controls, and follow-up and then reverse-engineer the process to arrive at which threats are exploiting which vulnerabilities.

Other actors such as government agencies (DHS for example) and security researchers go through the same process. They all have their own methods of churning through the planning, test results, controls, and follow-up, to reverse-engineer the data in order to arrive at which threats are exploiting which vulnerabilities.

This ongoing process of “reverse-engineering” is the root cause for a series of additional problems:

• Lack of overview of the the security threats and vulnerabilities that really count
• No sufficient connection to best practice security controls, no indication on which controls to follow or which have been followed
• No connection between controls and security events, except circumstantial
• No ability to detect and warn for negative interactions between countermeasures (for example – configuring a firewall that blocks Internet access but also blocks operating system updates and enables malicious insiders or outsiders to back-door into the systems from inside the network and compromise  firewalled services).
• No archiving or demoting of less important and solved threat scenarios (since the data models are control based)
• Lack of overview of security status of a particular business, only a series of historical observations disclosed or not disclosed.  Is Bank of America getting better at data security or worse?
• An excess of event data that cannot possibly be read by the security and risk analyst at every encounter
• Confidentiality and privacy borders are hard to define since the border definitions are networks, systems and applications not confidentiality and privacy.

Threat scenarios as an alternative to compliance control policies

When we perform a software security assessment of a medical device or healthcare system, we think in terms of “threat scenarios” or “attack scenarios”, and the result of that thinking manifests itself in planning, penetration testing, security countermeasures, and follow-up for compliance. The threat scenarios are not “one size fits all”.  The threat scenarios for an AIDS testing lab using medical devices that automatically scan and analyze blood samples, or an Army hospital using a networked brain scanning device to diagnose soldiers with head injuries, or an implanted cardiac device with mobile connectivity are all totally different.

We evaluate the medical device or healthcare product from an attacker point of view, then from the management team point of view, and then recommend specific cost-effective, security countermeasures to mitigate the damage from the most likely attacks.

Threat scenarios consider asset values, vulnerabilities, threats and possible security countermeasures. Threat analysis as a methodology does not look for ROI or ROSI (there is no ROI for security anyhow) but considers the best and cheapest way to reduce asset value at risk.

In our experience, building the security portfolio on threat scenarios has 2 clear benefits;

1. A robust, cost-effective security portfolio based on attack analysis  results in robust compliance over time.
2. Executives relate well to the concepts of threat modeling / attack analysis. Competing, understanding the value of their assets, taking risks and protecting themselves from attackers is really, at the end of the day why executives get the big bucks.

What a simple idea. It doesn’t matter how they break into your network or servers – if attackers can’t take out your data, then you’ve mitigated the threat.

Data loss prevention is a category of information security products that has matured from Web / email content filtering products into technologies that can detect unauthorized network transfer of valuable digital assets such as credit cards. This paper reviews the motivation for and the taxonomies of advanced content flow monitoring technologies that are being used to audit network activity and protect data inside the network.

Motivation – why prevent data loss?

The majority of hacker attacks and data loss events are not on the IT infrastructure but on the data itself.  If you have valuable data (credit cards, customer lists, ePHI) then you have to protect it.

Content monitoring has traditionally meant monitoring of employee or student surfing and filtering out objective content such as violence, pornography and drugs. This sort of Web content filtering became “mainstream” with wide-scale deployments in schools and larger businesses by commercial closed source companies such as McAfee and Bluecoat and Open Source products such as Censor Net and Spam Assassin. Similar signature-based technologies are also used to perform intrusion detection and prevention.

However, starting in 2003, a new class of content monitoring products started emerging that is aimed squarely at protecting firms from unauthorized “information leakage”, “data theft” or “data loss” no matter what kind of attack was mounted. Whether the data was stolen by hackers, leaked by malicious insiders or disclosed via a Web application vulnerability, the data is flowing out of the organization. The attack vector in a data loss event is immaterial if we focus on preventing the data loss itself.

The motivation for using data loss prevention products is economic not behavioral; transfer of digital assets  such as credit cards and PHI by trusted insiders or trusted systems can cause much more economic damage than viruses to a business.

Unlike viruses, once a competitor steals data you cannot reformat the hard disk and restore from backup.

Companies often hesitate from publicly reporting data loss events because it damages their corporate brand, gives competitors an advantage and undermines customer trust no matter how much economic damage was actually done.

Who buys DLP (data loss prevention)?

This is an interesting question. On one hand, we understand that protecting intellectual property, commercial assets and compliance-regulated data like ePHI and credit cards is  essentially an issue of  business risk management. On the other hand, companies like Symantec and McAfee and IBM sell security products to IT and information security managers.

IT managers focus on maintaining predictable execution of business processes not dealing with unpredictable, rare, high-impact events like data loss.  Information security managers find DLP technology interesting (and even titillating – since it detects details of employee behavior, good and bad) but an  information security manager who buys Data loss prevention (DLP) technology is essentially admitting that his perimeter security (firewall, IPS) and policies and procedures are inadequate.

While data loss prevention may be a problematic sale for IT and information security staffers, it plays well into the overall risk analysis,  risk management and compliance processes of the business unit.

Data loss prevention for senior executives

There seem to be three schools of thought on this with senior executives:

1. One common approach is to ignore the problem and brush it under the compliance carpet using a line of reasoning that says “If I’m PCI DSS/HIPAA compliant, then I’ve done what needs to be done, and there is no point spending more money on fancy security technologies that will expose even more vulnerabilities”.
2. A second approach is to perform passive data loss detection and monitor flow of data(like email and file transfers) without notifying employees or the whole world. Anomalous detection events can then be used to improve business processes and mitigate system vulnerabilities. The advantage of passive monitoring is that neither employees nor hackers can detect a Layer 2 sniffer device and a sniffer is immune to configuration and operational problems in the network. If it can’t be detected on the network. then this school of thought has plausible deniability.
3. A third approach takes data loss prevention a step beyond security and turns it into a competitive advantage. A smart CEO can use data loss prevention system as a deterrent and as a way of enhancing the brand (“your credit cards are safer with us because even if the Saudi hacker gets past our firewall and into the network, he won’t be able to take the data out”).

A firewall is not enough

Many firms now realize that a firewall is not enough to protect digital assets inside the network and look towards incoming/outgoing content monitoring. This is because: 

1. The firewall might not be properly configured to stop all the suspicious traffic.

2. The firewall doesn’t have the capability to detect all types of content, especially embedded content in tunneled protocols.

3. The major of hacker attacks and data loss events are not on the IT infrastructure but on the data itself.

4. Most hackers do not expect creative defenses so they assume that once they are in, nobody is watching their nasty activities.

5. The firewall itself can be compromised. As we have more and more Day-0 attacks and trusted insider threats, so it is good practice to add additional independent controls.

Detection

Sophisticated incoming and outgoing (data loss prevention or DLP) content monitoring technologies basically use three paradigms for detecting security events

1. AD- Anomaly Detection – describes normal network behavior and flags everything else
2. MD- Misuse Detection – describes attacks and flags them directly
3. BA – Burglar alarm – describes abnormal network behavior (“detection by exception”)

In anomaly detection, new traffic that doesn’t match the model is labeled as suspicious or bad and an alert is generated. The main limitation of anomaly detection is that if it is too conservative, then it will generate too many false positives (a false alarm) and over time the analyst will ignore it. On the other hand, if a tool rapidly adapts the model to evolving traffic change, then too little alerts will be generated and the analyst will again ignore it.

Misuse detection describes attacks and flags them directly, using a database of known attack signatures and constantly tries to match the actual traffic against the database. If there is a match, an alert is generated. The database typically contains rules for:

1. Protocol Stack Verification – RFC’s, ping of death, stealth scanning etc.
2. Application Protocol Verification – WinNuke , invalid packets that cause DNS cache corruption etc.
3. Application Misuse – misuse that causes applications to crash or enables a user to gain super user privileges; typically due to buffer overflows or due to implementation bugs.
4. Intruder detection. Known attacks can be recognized by the effects caused by the attack itself. For example, Back Orifice 2000 sends traffic on default port is 31337
5. Data loss detection – for example by file types, compound regular expressions, linguistic and/or statistical content profiling. Data loss prevention or detection needs to work at a much higher level than intrusion detection – since it needs to understand file formats and analyze the actual content such as Microsoft Office attachments in a Web mail session as opposed to doing simple pattern matching of an http request string.

Using a burglar alarm model, the analyst needs deep understanding of the network and what should not happen with it. He builds rules that model how the monitored network should conceptually work, in order to generate alerts when suspicious traffic is detected. The richer the rules database, the more effective the tool. The advantage of the burglar alarm model is that a good network administrator can leverage his knowledge of servers, segments and clients (for example a Siebel CRM server which is a client to a Oracle database server) in order to focus-in and manage-by-exception.

What about prevention?

Anomaly detection is an excellent way of identifying network vulnerabilities but a customer cannot prevent extrusion events based on general network anomalies such as usage of anonymous ftp. In comparison there is a conceptual problem with misuse detection. If misuse is detected then unless the event can be prevented (either internally with a TCP reset, by notifying the router or firewall) – then the usefulness of the devices is limited to forensics collection.

What about security management?

SIM – or security information management consolidates reporting, analysis, event management and log analysis. There are a number of tools in this category – Netforensics is one. SIM systems do not perform detection or prevention functions – they manage and receive reports from other systems. Checkpoint for example is a vendor that provides this functionality with partnerships.

Summary

There are many novel DLP/data loss prevention products, most provide capabilities far ahead of both business and IT infrastructure management that are only now beginning to look towards content monitoring behind the firewall.

DLP (Data loss prevention) solutions join an array of content and application-security products around the traditional firewall. Customers are already implementing a multitude of network security products for Inbound Web filtering, Anti-virus, Inbound mail filtering and Instant Messaging enforcement along with products for SIM and integrated log analysis.

The industry has reached the point where the need to simplify and reduce IT security implementation and operational costs becomes a major purchasing driver, perhaps more dominant than any single best-of-breed product.

Perhaps data loss prevention needs to become a network security function that is part of the network switching fabric; providing unified network channel and content security.

Software Associates helps healthcare customers design and implement such a unified network channel and enterprise content security solution today enabling customers to easily define policies such as “No Instant Messaging on our network” or “Prevent patient data leaving the company over any channel that is not an authorized SSH client/server”.

For more information contact us.