Every security consultant I run into talks about pen (penetration) testing they do for clients. I have not met a single consulting firm that talks about opening up the production software code and looking for vulnerabilities. The reason is that it is just too damn easy to do a pen test, the customer gets a report, and the consultant gets paid and everybody sleeps well at night.
Let me relate a story. A few years ago, at a previous place of employment (Commerce.net) we provided secure transaction processing services to online merchants. We contracted with one of the top networking consultants to do a pen test - which he did - remotely from his office - and he gave us a report of open ports and services. We asked some questions and after getting some standard answers (I suppose they are standard) we proceded to close down the exposed ports and services.
Our friend the consultant never bothered coming around to the office during the pen test. What about social engineering? What about vulnerability in the source code of the scripts we wrote and ran for secure credit card processing?
About 3 months later, our VP operations got a phone call from a white-hat hacker who had the names and job titles and organizational structure of the entire company. They met for lunch and it turned out that our white-hatted friend had exploited a vulnerability in Windows servers that could never be detected in a remotely executed pen test. I wont go into details in the interest of protecting both parties.
There is a lot of talk about awareness and governance and compliance stuff which I dont understand, but what I do know is that you cannot pen-test security into your systems just the way you cannot test quality into your software.
