Penetration testing is one of the most common ways for companies to measure their current state of security. Even companies or applications that do not require a penetration test for regulatory reasons rely on them to measure their success or failure. The intent is to hire someone to hack your network or application like “the bad guys” would, and then receive a report indicating weaknesses in the system.

There are multiple ways to measure the value one receives from a penetration test.

• The results, or at least a summary of the assessment, can be provided to clients, vendors, or other 3rd party entities. This is typically done to attest that the company is meeting industry standards when it comes to security. Of course, this view is only from an attack perspective, and doesn’t identify actual security operations within the organization. It is a primal indicator to determine if you are susceptible to security weaknesses that could lead to a breach. This type of testing is often required by these outside entities.
• The results are used internally to help measure the security stance of the organization or application. It is a chance to get verification whether or not the controls put in place are actually effective. It allows the testing of auditing and monitoring controls in real time. It creates a way to determine if patches are applied to the right resources. It helps validate what we should already know. It also helps identify possible items that are not known.
• The results are actionable. The findings can be handed to a developer or administrator to be remediated. This creates the ability to reduce the known risk.

Organizations put a lot of focus on remediating the exact penetration test results. Unfortunately, this leads to a false sense of security. To understand why, you must understand how penetration test reports are consumed. The most important thing to keep in mind is that a penetration test is typically not focused on finding every vulnerability. It isn’t even focused on finding all instances of a specific vulnerability. The focus is identifying weaknesses and risks that are available and determining how much access those items may lead to.

Listen to the podcast of this topic

There are two pieces to a penetration test finding.

• The finding – This is the high level identification of a classification of vulnerability. Cross Site Scripting or SQL injection are common examples of a finding.
• The instance – This is an example of the finding. There can be multiple instances per finding in the report. Cross Site Scripting on the search results page is an example of an instance of the Cross Site Scripting finding.

Due to the nature of a penetration test, remediating the instances provided falls short from a security perspective. Organizations spend so much effort focusing on the wrong information. Rather than focus on the finding, the focus is typically on each instance of a finding. Let’s look at an example.

A penetration test identifies cross-site scripting in the final report. That finding has 2 instances drawn out. The first instance is on the profile page and the second instance is on the documents page.

When the report is provided it is common to see the organization focus on the instances. In the example above, a developer would be assigned to resolve those two instances of cross-site scripting and the finding would be considered closed. By remediating those two items are you sure the issue is really closed? Remember, the goal of the penetration test is not to identify every instance, but to identify the different risks.

Rather than focusing on the instances, it is important to start focusing on the actual finding. Using the same example from above, the organization should create a task to identify why they have cross-site scripting issues within the application and then how they want to proceed to remediate them.

This involves:

• Identifying and understanding the flaw (May indicate needed training)
• Understanding how the application is developed
• Identifying how it should be coded securely
• Going through the application identifying these items to resolve it application wide

By actually analyzing the flaw itself, a much larger impact can be made to the application. Working just the instances identified in the report is like trying to plug randomly identified holes in a sinking ship. Sure, it resolves that one issue, but what is happening with those holes you don’t see letting water in.

If a quick inspection of a ship identified a crack or issue in one location, wouldn’t you want to inspect the rest of the ship making sure that issue isn’t somewhere else?

Penetration tests provide different values, but it is time that the true potential is realized. Don’t stop at just trying to remediate the instances in the penetration test, start looking to enhance your overall security by analyzing the findings.

James Jardine is the CEO and Principal Consultant at Jardine Software Inc. He has over 15 years of combined development and security experience. If you are interested in learning more about Jardine Software, you can reach him at james@jardinesoftware.com or @jardinesoftware on twitter.