This article is provided by Software Secured, an Eden Data penetration testing partner.

With numerous manual penetration testing services out there, all claiming to be the best at finding more vulnerabilities - how do you differentiate between them and decide which is right for your team? How do you know if you need a bug bounty program, a penetration test, or both? In this post,we’ll break down both and describe the advantages and disadvantages with the aim to help you better understand your options.

What is a Bug Bounty Program?

Bug bounties are a way to apply a financial incentive to finding security vulnerabilities. Companies post bounties (ie. financial rewards) for found vulnerabilities at different severity levels. The higher the severity, the higher the price. Freelance ethical hackers can attempt to dive into applications that have posted bounties in an attempt to reap the rewards. Other than receiving financial compensation, hackers may also be motivated through placements on leaderboards to help them build their reputation.

Bug bounty programs are rarely used in isolation. Typically, they are a security measure used in conjunction with penetration testing to ensure improved security resilience overall. 

As an example, Shopify’s program pays out an average of $500-900 per found vulnerability. Payouts for critical vulnerabilities, however, can range from $50,0000 up to $100,000. To date, nearly 1,500 reports have been resolved through the program. 
‍

Responsible Disclosure in Bug Bounty Programs

An important distinction to make about bug bounty programs is that they have an element of responsible disclosure. When companies create programs with pre-set payouts per vulnerability type, it is known that this payment should be made to the hacker when a vulnerability is found. The hacker must responsibly disclose the vulnerability to the company in order to receive the funds. This means they must notify the development team (usually there is an e-mail address listed on the bug bounty site) and offer all known details about the vulnerability. It is best practice to give the development team at least 90 days to patch the vulnerability internally before the hacker can announce it to the world (in some cases, this might be longer or shorter, depending on the response of the development team). 

Hackers that do not responsibly disclose the vulnerability, exploit it, or try to earn money off vulnerabilities at companies that do not have bug bounty programs, are not considered ethical hackers. These are various degrees of malicious intent that are not considered best practices within a bug bounty program.

Pros of a Bug Bounty Program

• You won’t need to allocate a dedicated security person to the job. The hacker who is breaching your application will need to provide you with details on the vulnerability once found. This report can go right to your developers to patch. In some cases, you might be able to work with your ethical hacker on remediation efforts (ie. re-testing the issue after it’s patched) but this is not guaranteed. 
• It’s an always-on type of testing, where as long as your bounty is posted, you are inviting ethical hackers to try and find new vulnerabilities.
• It’s customary that you have at least 90 days after the vulnerability is found to patch it before the hacker can go live with a report about the successful find. You can often work with the hacker(s) to extend this deadline if needed. It’s a great way to help control the PR narrative where possible.

Cons of a Bug Bounty Program

• One of the biggest disadvantages to a bug bounty program is the unknowns of the person who is conducting the testing. You don’t know the experience or reputation of the hacker breaking into your systems. And because of that, it’s hard to identify if the hacker might use this learning opportunity to break into your application for real and cause some serious damage. Likewise, there is a risk they may not actually responsibly disclose the vulnerabilities to you and may use the knowledge to exploit for malicious intent.
• With bug bounty programs, there is very little stability. There is no guarantee that any hacker is going to try to break into your application or that meaningful vulnerabilities will be found. You also don’t have any control over the level of expertise of the hacker who is working in your app.
• If something goes wrong while a hacker is attempting to breach your application, your team has little awareness and time to respond. 
• With little predictability, it makes it difficult for developers to plan out their time. When a critical or high vulnerability comes through the door, your developers will need to set other projects aside to focus on patching the new issue. While it’s good to ensure critical issues get patched soon, this can have an effect on productivity of your team and sprint commitments if you can’t plan in advance.
• There is uncapped gains to be earned by the hackers. Unless you post a cap of the total bounties that you’re willing to disperse, you can find the dues will add up quickly. This can be hard to plan for in your security budgeting, as you won’t know what month the vulnerability will be found.

What is a Penetration Test?

Conducting a penetration test is a planned, comprehensive way to measure your application’s resilience to attack. Like traditional, vetted bug bounty programs, penetration testing relies on ethical hackers to attempt to break into an application and find as many vulnerabilities as possible. A key difference is that penetration testing is done by a vendor, planned in advance, and the penetration testers will try to find as many vulnerabilities in one testing engagement as possible (this can take between 5-20+ days, depending on the scope of the application). Many of the same ethical hacking principles are followed, however it is the planned nature that really differentiates the two services. 

Penetration tests can be conducted on any system, application, network or infrastructure. They can also be conducted in three different types of tests, including:

• White-box penetration testing, where all of the source code is provided so the pentesters can get as deep as possible into the application.
• Black-box penetration testing, which does not provide any access or source code to the pentesters. It is the closest to simulating a real threat from a hacker who does not know anything about the system inside. ‍
• Grey-box penetration testing, which provides the testers with some source code or access, but not all. This enables the pentesters to get into the system quickly and still leverage their creativity to attempt to crack the rest of the system open.

‍

Pros

• The test is planned in advance, meaning your developers have dedicated time to support the test and remediate criticals immediately. You might opt to have one security champion take the lead on managing the relationship with the pentest vendor to make this even more efficient. 
• You are provided with a detailed report. A great report should include replication steps for each vulnerability, evidence of where it exists within the application, details about its impact and severity, and mitigation recommendations. 
• Ideally, your penetration testing vendor will also have a standardized framework for measuring the severity of each vulnerability so you minimize the threat of security theater distracting your development team.
• Your tester is vetted and trusted. When you work with a reputable penetration testing firm, you guarantee that the quality of your report is going to be much higher and there is trust that your vulnerabilities will not be exploited for malicious intent.

Cons

• Testing only occurs for a limited time.. There will not be random tests on the application outside of the agreed testing period. In the same vein, testing will not go outside of the scope of the agreement. This means any other apps or integrations that are not scoped within the pentest will not be looked at by the pentesters. Depending on your needs, you might feel like this could leave you vulnerable in other areas or between tests.
• There is a higher upfront cost when getting a penetration test done. Unlike bug bounties where the fee is paid out at the time of the vulnerability being found, a penetration testing vendor gets paid a larger fee based on the size and scope of the application, and not based on how many vulnerabilities are actually found.

Other Types of Application Security Testing

• Vulnerability scanners
  1. Static pplication security testing (SAST)
  2. Dynamic application security testing (DAST)
  3. Runtime application self protection (RASP)
  4. Interactive application security testing (IAST)

• Red teaming
• Secure code review
• Software composition analysis (SCA)

What Type of Testing Do You Need?

If you’re trying to achieve compliance…

A bug bounty on its own won’t be enough to help you earn compliance. If you’re working towards a SOC 2, ISO 27001, or PCI-DSS certification, you will need a certificate from a vendor that proves your application security has been assessed by an external party. At minimum, a scan from a SAST tool can help you get this. However, a penetration test is still recommended for most compliance frameworks as it goes into more detail, often finds more vulnerabilities (has better coverage over your application), and connects you with an ethical hacker to ask questions during your testing and remediation phases to ensure your team will be well positioned to close any vulnerabilities found.

If you’re trying to scale code deployment…

You can consider pairing bug bounty programs with penetration testing. The bounty program will help cover gaps between your pentests in case any critical or high vulnerabilities arise in new code. If you are notified of any new vulnerabilities between pentests, keep your tester in the loop before the next assessment so they can double-check for those vulnerabilities.

Another alternative is penetration testing as a service (PTaaS). It’s offered as an extended, more comprehensive version of penetration testing and assists with some of the pros of bug bounty programs and cons of annual pentesting. PTaaS usually involves quarterly penetration testing (rather than doing it annually or bi-annually) and includes extra features such as security consulting, unlimited re-testing on patched vulnerabilities, and rotating pentesters to keep up a fresh perspective. Through unlimited re-testing, you can also test new features as they’re released rather than waiting for the next major test. This is a way to experience the benefit of incorporating a bug bounty program into your overall security program without requiring management of multiple vendors or freelance ethical hackers.