pen test

June 3, 2018 by James Jardine

Thinking about starting a bug bounty? Do this first.

Application security has become an important topic within our organizations. We have come to understand that the data that we deem sensitive and critical to our business is made available through these applications. With breaches happening all the time, it is critical to take reasonable steps to help protect that data by ensuring that our applications are implementing strong controls.

Over the years, testing has been the main avenue for “implementing” security into applications. We have seen a shift to the left more recently, leading to doing more throughout the entire development cycle, but we still have a ways to go. I am still a firm believer in embedding security into each of the phases as our main means of securing applications. Testing, however, is still a major component of any security program.

Typically, organizations rely on penetration testing to find the flaws in their applications. This is the de facto standard for understanding your risk. Unfortunately, penetration testing for applications has been watered down from what we think about with network testing. Many of the assessments we call penetration tests these days are just automated scans transposed into a custom report. These types of testing overlook one of the components a penetration test provides, which is the manual testing. Of course, there is much more to a penetration test, but that is not the focus of this post.

Internally, organizations may implement automated tools to help identify security flaws within their applications. These tools are good at finding certain types of flaws, and usually quite quickly. Like many current penetration tests, they lack the manual assessment side.

Not only does manual testing have the ability to find different types of flaws, such as authentication, authorization, CSRF, business logic, etc., it also has the ability to identify flaws that an automation tool overlooks. For example, a tool may not find every instance of cross-site scripting, depending on how that tool analyzes the system. Granted, manual testing is not guaranteed to find every instance either. With each type of testing, there is always a number of issues that will not be identified. The goal is to start reducing these numbers down over time.

Handling the results of all these res ports from the different assessments is critical to how well you start creating more resilient applications. In many organizations, vulnerabilities identified are handled as individual items and patched. In my opinion, the return on investment is when you can analyze these results to review your development process and see what improvements can be made to reduce the chance these types of flaws will be included in the future. Having an expert available to help review the issues and provide insight into how to use that information to improve your process is valuable.

Having a solid application development process in place is important before thinking about implementing a bug bounty program within your organization. If you are not already doing things consistently, there is a better chance the bounty program will fail.

Bug bounty programs have been becoming more prevalent over the last few years. This is especially true for newer technical startups. We have seen much slower adoption with most of the major corporations. There are many reasons for this, which are outside the scope of this post. There have been questions on whether bug bounties can replace penetration testing. The answer is no, because the goal of each of these is different. There are plenty of articles discussing the subject. A bug bounty program has also been seen by many as the evidence to show they are doing application security. Unfortunately, we can’t test ourselves secure. As I stated previously, testing is just a part of our solution for application security.

A key difference between our traditional testing and a bug bounty program is that bug bounties pay by the bug. Our traditional testing is provided at flat fees. For example, that automated tool is a set price for a month or year subscription. A penetration test is a set price per test. A bug bounty is priced per bug, which makes the cost very unpredictable. In addition, if you are not already doing many of the things previously discussed, there could be a lot of bugs to be found, leading to potentially high payouts.

As I have stated before, penetration testing has a different purpose and it can be very expensive. At Jardine Software we offer more budget friendly manual application security testing at a fixed cost. The goal is not necessarily to find every instance of every vulnerability or to exploit vulnerabilities in the way a penetration test would. The focus is on augmenting the automated testing you may already have in place and to provide that missing manual piece. The testing is performed manually by using the application in combination with Burp Suite, to look for weaknesses and provide those in a way that helps prioritize and then remediate them according to your organization’s needs.

The manual application security testing is typically performed over a week to two weeks and includes a broader scope than a typical bug bounty program. The reason for this is that we want to help identify risks that we see based on our years of experience to make you aware. This assessment can then help identify where you may have issues within your application before opening it up for a crowd sourced bounty program where each bug is priced individually.

If you are thinking about implementing a bug bounty program, reach out and lets chat first. Even if you are not considering a bug bounty program, do you have any manual application security testing implemented? We have the expertise to help provide the necessary testing or provide training for your internal teams to start applying manual testing techniques as part of your life cycle.

March 9, 2016 by James Jardine

Getting More Value from a Penetration Test

Penetration testing is one of the most common ways for companies to measure their current state of security. Even companies or applications that do not require a penetration test for regulatory reasons rely on them to measure their success or failure. The intent is to hire someone to hack your network or application like “the bad guys” would, and then receive a report indicating weaknesses in the system.

There are multiple ways to measure the value one receives from a penetration test.

The results, or at least a summary of the assessment, can be provided to clients, vendors, or other 3rd party entities. This is typically done to attest that the company is meeting industry standards when it comes to security. Of course, this view is only from an attack perspective, and doesn’t identify actual security operations within the organization. It is a primal indicator to determine if you are susceptible to security weaknesses that could lead to a breach. This type of testing is often required by these outside entities.
The results are used internally to help measure the security stance of the organization or application. It is a chance to get verification whether or not the controls put in place are actually effective. It allows the testing of auditing and monitoring controls in real time. It creates a way to determine if patches are applied to the right resources. It helps validate what we should already know. It also helps identify possible items that are not known.
The results are actionable. The findings can be handed to a developer or administrator to be remediated. This creates the ability to reduce the known risk.

Organizations put a lot of focus on remediating the exact penetration test results. Unfortunately, this leads to a false sense of security. To understand why, you must understand how penetration test reports are consumed. The most important thing to keep in mind is that a penetration test is typically not focused on finding every vulnerability. It isn’t even focused on finding all instances of a specific vulnerability. The focus is identifying weaknesses and risks that are available and determining how much access those items may lead to.

Listen to the podcast of this topic

There are two pieces to a penetration test finding.

The finding – This is the high level identification of a classification of vulnerability. Cross Site Scripting or SQL injection are common examples of a finding.
The instance – This is an example of the finding. There can be multiple instances per finding in the report. Cross Site Scripting on the search results page is an example of an instance of the Cross Site Scripting finding.

Due to the nature of a penetration test, remediating the instances provided falls short from a security perspective. Organizations spend so much effort focusing on the wrong information. Rather than focus on the finding, the focus is typically on each instance of a finding. Let’s look at an example.

A penetration test identifies cross-site scripting in the final report. That finding has 2 instances drawn out. The first instance is on the profile page and the second instance is on the documents page.

When the report is provided it is common to see the organization focus on the instances. In the example above, a developer would be assigned to resolve those two instances of cross-site scripting and the finding would be considered closed. By remediating those two items are you sure the issue is really closed? Remember, the goal of the penetration test is not to identify every instance, but to identify the different risks.

Rather than focusing on the instances, it is important to start focusing on the actual finding. Using the same example from above, the organization should create a task to identify why they have cross-site scripting issues within the application and then how they want to proceed to remediate them.

This involves:

Identifying and understanding the flaw (May indicate needed training)
Understanding how the application is developed
Identifying how it should be coded securely
Going through the application identifying these items to resolve it application wide

By actually analyzing the flaw itself, a much larger impact can be made to the application. Working just the instances identified in the report is like trying to plug randomly identified holes in a sinking ship. Sure, it resolves that one issue, but what is happening with those holes you don’t see letting water in.

If a quick inspection of a ship identified a crack or issue in one location, wouldn’t you want to inspect the rest of the ship making sure that issue isn’t somewhere else?

Penetration tests provide different values, but it is time that the true potential is realized. Don’t stop at just trying to remediate the instances in the penetration test, start looking to enhance your overall security by analyzing the findings.

James Jardine is the CEO and Principal Consultant at Jardine Software Inc. He has over 15 years of combined development and security experience. If you are interested in learning more about Jardine Software, you can reach him at james@jardinesoftware.com or @jardinesoftware on twitter.

Thinking about starting a bug bounty? Do this first.

Getting More Value from a Penetration Test

Company Profile

Resources

Contact Us

Company Profile

Resources

Engage With Us

Contact Us