Penetration testing is simply fun work. If you like a good mystery novel or a psychological thriller movie, you’ll love penetration testing because you never know what’s around the next corner. Being allowed to try to find the holes in applications and infrastructure is like being a kid in a candy shop. You sit starry eyed looking at all the options wondering where to begin.

One question I routinely get is which type of testing is better, white box or black box. Being the black and white kind of guy I am, that’s easy. Neither is better. Both methods have a specific purpose. If you are considering hiring someone to do testing you probably actually want both and here’s why.

Black box testing is designed to determine the strength of your systems and their ability to withstand attack from external sources with limited knowledge of your architecture. This is very useful when trying to assess risk of attack from the outside. I’d love to say in today’s world most organizations are safe here and black box testing won’t yield much information. In reality, companies are often vulnerable to attacks from any and everyone who simply passes by on the digital highway. Sad…but true.

On the other hand, white box testing assumes that an attacker has some level of knowledge of a given system. It could be a userid, known software versions, offsite backup tape locations, network diagrams, etc. White box testing is ideal for those who need to know how much damage could occur if certain information is known to an attacker.

So trying to say one is better than the other is like trying to say a truck is better than a car. Truth is they both give you basic transportation needs but have different intended purposes. One is more suited to transport cargo and the other people. You have to know your objectives when picking a vehicle to use. When determining which type of testing to use the first step is to identify your objectives for the test. Once you know what you want to accomplish with the testing you should have no problem picking a method. You may even decide to do both.

One thing I've noticed over the past year is that the majority of information on the web relating to digital forensics is geared toward two audiences.  The first is law enforcement and the second are consulting firms.  This makes complete sense.  Law enforcement probably does more evidence collecting and review that anyone else and most companies don't need (or want) a full time forensics team.  So it completely understandable that the bulk of materials cater to these groups.

What concerns me is how underserved those in corporate incident response are by the larger forensics community.  Even if incident response at the corporate level is a small market, it is a market nonetheless.  Many times companies want to add these capabilities to their in-house arsenal but have no idea where to start.  If you are in that camp....keep reading.  This article is just for you.

Companies which are considering adding digital forensic investigation capabilities in-house need to ask themselves several questions upfront.

1. Why do we need this capability?
2. Who's going to provide the services and where will they report?
3. Is the cost to purchase and maintain a lab and provide continuing education worth the expense?

Questions 1 and 2 are typically easy to answer.  Number three is more difficult.  A typical 1 week engagement from one of the well know consulting groups like IBM, Deloitte and others can easily surpass $10,000. Can you provide this in house cheaper? If you plan to do investigations once or twice a year, probably not.  If you plan to do it once or twice a month, then likely so.  Many times though we forget to look at all of the costs associated with standing up a unique environment like a forensic lab.  It shouldn't be hooked to your network which creates management concerns around patching, updating, etc.  The hardware most likely will be vastly different that what you purchase for other needs. If you choose to utilize commercial software packages they must be continually updated and software maintenance is a must.

Once an organization decides to offer these services, some immediate steps need to be taken to ensure uniformity in your investigations.  An incident response plan must be developed which lays out who can request or initiate an investigation, what the grounds for cause are, who can perform the investigation, who sees the results and what actions should or must be taken based on the outcome.  One thing organizations must protect against is the witch hunt mentality.  You've got suspicion someone is up to no good but nothing more than that.  Just a hunch.  I advise my clients never to start an investigation this way.  Make sure there is always a reason to investigate.  Last thing you want is for your employees to feel there is a culture of mistrust and accusations within the organization.  At best your morale will sink to new lows.  At worst you could be sued by employees for harassment, privacy or other accusations.

The next thing to guard against is who can call for an investigation.  I usually like to see a 2nd level manager be in the approval chain.  Let's say Bob is an employee who's manager suspects he's creating fraudulent transactions. If Bob works for Sally and Sally works for Meg then Sally would make the request for investigation and Meg has to approve it.  This again helps reduce the chance for a witch hunt and ensures a level of accountability in the process.

You'll also want to make sure investigations are only completed by those who've had some specialized training in acquiring and handling evidence, digital forensic processes and reporting of results.  I wish I could say you'll never have to worry about your internal investigations ending up in civil or criminal court but I can't.  You should always approach new cases as having the potential to have aspects of civil or criminal law to them.  This will save you a lot of headache later.

In terms of who should have access to the final reports, that will vary by organization.  Needless to say it should be restricted by need to know.  Also remember that many times an initial investigation will lead to one or more follow on investigations and the fewer people who are privy to this fact the better.

So as you begin to build a forensic unit within your organization, here are some things you'll want to consider.

1. Develop an Incident Response Plan prior to doing any investigations
2. Create a dual or dotted line reporting structure to maintain independence
3. Build a self-sustaining lab and staff it appropriately
4. Create a set of criteria for requesting and initiating investigations to ensure objectivity
5. Build a communication framework for investigators to ensure they have support of executives, HR and Legal.
6. Develop a plan for when and how to call for outside help including law enforcement or more experienced investigators.

By following these steps you'll be well on your way to providing these services in house.  I can't promise everything will be picture perfect.  In fact I can almost guarantee at one or more points along this path you'll wonder if this was the right decision.  Adding forensic capabilities to your internal service offering will change the culture at your organization.  Make sure you're leadership, all the way to the top, are on board and understand this decision.

Looking for a job in digital forensics?
If you have experience with digital investigations and are looking for work please send your resume and type of position desired to This email address is being protected from spambots. You need JavaScript enabled to view it.


When someone says they are getting ready to do a gap analysis of your IT systems, what’s the first thing that comes to mind?  Compliance, SOX, HIPAA, log management, change control, identity management?  While these are all components which should be reviewed, they aren’t a starting point.

Process should be the starting point for every review.  You may have a log management system in place however without the process which defines who reviews the logs, what the look for and what actions to take, that technology is useless,  Expensive and pretty to look at, but still useless. 

Companies often start looking at their gaps from a technological perspective.  They try to ensure they’ve got all the latest and greatest tools and applications which are all the rave in industry journals, security blogs or “Best of the Best” competitions.  They need to be placing a higher emphasis on the maturity of their process.  This includes their policy, procedures and overall documentation.

It happens time and time again.  An outage occurs because a simple procedure was not followed.  Perhaps the procedure wasn’t documented or wasn’t documented well.  Perhaps the procedure exists but nobody knows it exists.  These types of issues show a lack of maturing which no amount of technology can fix.  Organizations should begin to place more importance, and thus resources, on improving their process.

It has to be done at some point as technology will only take you so far down the security and compliance path.  In fact, at times technology will mask poor processes which has can cause problems when an organization begins to peel back the layers.

If you are a CIO, CTO, VP, Director or Manager of a technology group I implore you to sit down as a leadership team and review the following 5 items.

1.       Do we have a policy which dictates the design, implementation, management and review of our systems?

2.       Do we have a repository of procedures and associated documentation to carry out this policy?

3.       Are team members well educated in the existence, importance and location of this information?

4.       Is someone responsible for managing this repository and performing periodic reviews as our technology and business process evolves?

5.       Is there a system in place to identify and correct actions not in line with our process?

If more of this activity were to take place I’d venture to bet companies would spend less time trying to fit various security technologies into their organization and more time finding technology which actually complements their existing way of doing business.

If you are a frequent reader you know I try to add postings several times per week.  This past week or so has been tough and my postings have been minimal.  As you can see we've launched a new website and have consolidated my blog into the new design.  I hope this provides a bit more integration and makes it easier to search for relevant content regardless if the content is located in the website articles or my blog postings.  I hope you like the new design and site functionality.  It's certainly an upgrade from our old site.

The Washington Post published a story last week about the rising threat of fraud against small business in the US. (Read) Brian Krebs does a good job of finding some examples of small businesses as well as government agencies like a school district which have been hit with financial fraud.

The FBI has begun to investigate cyber crime rings in Eastern Europe which are targeting US businesses. One of the concerns is the lack of data to support there is a problem. Many companies fear the bad publicity of announcing they are the victim of cyber crime. This creates a big dilemma. First, if not reported as a crime the company has few legal options in trying to recover any loses. Second, crime is investigated based on statistics. If nobody reports cyber crime, law enforcement agencies will never staff those investigative divisions appropriately and the waves will continue to roll.

Mr. Krebs' article included a quote from the controller of a small electronics calibration company in Louisiana. The company lost close to $98,000 in two attacks days apart. There real loss however was the investigation and recovery which is estimated to be 3 times their hard financial loss. That's nearly half a million dollars. This would effectively cripple most small businesses from a cash flow and operations perspective. Many of which might never recover.

If you own a small or medium business and think information security is an expenditure you can't afford, I beg you to reconsider. Not because I want your business, but because I BELIEVE in small business. It's the foundation of our economy. A risk assessment, vulnerability scan and some help with remediation efforts will most likely cost you between $20,000 and $50,000 when using a reputable and experienced consultant. That's no small chunk of change. But when compared with the staggering losses, both soft and hard, which are being felt by others it's really a drop in the bucket.

I can't guarantee you won't be a victim just because you spend some money on security. I can however assure you that you have reduced your risk of being a victim. That's what smart business people do on a daily basis, manage risk.

One of the things which really rubs me the wrong way is how we often interchange terminology even when the terms are truly different. One of these in the security world is calling vulnerability scanning the same thing as penetration testing. They are not the same thing.

Vulnerability scanning is the process by which we examine an application, operating system or database for a weakness which could be exploited. This is typically done using some automated tool. These weaknesses could range from a bug in the code to unvalidated input parameter or configuration errors. Typically vulnerability scanning doesn't include any invasive intrusion techniques because there is a lot of validation which needs to occur to ensure the vulnerability reported is accurate. A scan may reveal a Java vulnerability but after inspection it is only valid on a certain OS and you're something different. And, just because a vulnerability exists doesn't mean it will be attacked.

Penetration testing however is the art of actually exploiting the vulnerabilities which were discovered. I really mean it when I say it's an art. There are tools out there which can automate much of this and anybody can download these and run them against your systems. If you haven't had a breach from an automated tool yet it's probably because of one of the following.

  1. You did, you just don't know it yet.
  2. You're just off the radar and just haven't been discovered by the script kiddies yet.
  3. Your system is fairly well patched and configured so the automated tools don't find much to attach so they move on to an easier target.

If you patch your systems and follow best practices for configuration, you'll eliminate the vast majority of vulnerabilities and therefore limit the effectiveness of the automated penetration testing tools.

But these are not what I'd be worried about. What's going to get you is when your system is architected wrong. Automated tools usually won't discover this. It takes a skilled tester who understands how a fully patched and correctly configured system can be breached because of how it was architected.

If you are considering hiring a vendor or consultant to do security testing for you, drill them on the difference between vulnerability scanning and penetration testing. You might be surprise to find they don't have a clear understanding of the difference which would be a good reason to move on to the next vendor. Also don't be surprised when you find vastly different pricing for "testing" services. Not that the most expensive is necessarily better but you now have the ammunition to go into negotiations with guns a blazin'!

I don't like VPNs. I take that back. I like them a lot, I just don't trust them. Ever had a friend like that? They're fun to be around, are really helpful, always there when you need them, etc. For the most part you're great friends, but…they can't keep their mouth shut. You always have to watch what you say around them because you know it will be repeated. Probably multiple times to multiple parties. That's the view I have of VPNs.

I've been using some sort of VPN for probably a little more than a decade now. Not just remote access but truly secured communication channels. The goal of a VPN was to make location irrelevant in the computing equation. We've done that. You can login to an application or system remotely from just about any device with a processor and operating system, including mobile phones and PDAs.

We've gotten more secure in how we transport the data but for the most part continue to ignore the endpoint. This is my concern.

Read more: VPN: A Necessary Evil

Contact Information

Birmingham Office

205.568.5506


Des Moines Office

515.965.3756


Kansas City Office

913.991.8724