Here at NGINX we’re proud that literally millions of organizations around the world trust our software to power their websites and app‑delivery infrastructure. Given how critically our users depend on NGINX, we’re surprised when they admit they haven’t given much thought to the importance of running an up-to-date version that has fixes for the security vulnerabilities – such as Common Vulnerabilities and Exposures (CVEs) – that have become so common on today’s Internet.
Of course it’s not enough to think about guarding against security threats – you need to implement well‑defined processes for tracking vulnerabilities and putting protections in place as soon as they become available. It’s easy to underestimate how much time and effort that can take if you have to do it all yourself, as is the case with open source software. Indeed, making it quick and easy to protect yourself against security threats is an often‑overlooked benefit of commercially supported software like NGINX Plus.
In this blog we explore the challenges of protecting open source software and explain how NGINX Plus makes it so much faster and easier to mitigate CVEs and other security threats.
Dealing with Vulnerabilities in Open Source Software
As much as all developers like to think they write perfect code, no software is free from bugs. Developers hope most bugs are detected during development as part of a rigorous quality compliance process, with just a few surfacing under normal usage over the life of the app. In the worst case, bugs are discovered by malicious users with bad intentions.
The consequences of bugs are compounded greatly when you construct your application stack from multiple open source tools, programming languages, and platforms. You need to review each component separately to validate it as bug‑free. When bugs are discovered in open source software, it’s important to have a defined policy for validating, testing, and (if possible) fixing them.
Your policy for open source software needs to include at least three processes:
- Regular review and testing
- Tracking vulnerabilities and reporting them internally
- Remediating vulnerabilities in a timely manner
Reporting a Vulnerability
When a security vulnerability is discovered, there’s a standard sequence of events for disclosing it. The first step is to submit a detailed report to the author or vendor of the software. The reporter and author together decide how and when to disclose the vulnerability, usually the form of a record in the Common Vulnerabilities and Exposures (CVE) database. By convention, the author has 90 days to resolve the issue before disclosure, but it’s possible to negotiate more time.
NGINX and CVEs
As a provider of open source software and vendor of commercial software, NGINX actively participates in the reporting and remediation process for CVEs and other vulnerabilities that affect NGINX Open Source and NGINX Plus, as well as its other commercial products – which at the time of writing include NGINX Controller [now F5 NGINX Management Suite], NGINX App Protect, and NGINX Ingress Controller – and free and open source offerings, which include NGINX Service Mesh, NGINX Unit, and NGINX Amplify.
Users of NGINX Open Source benefit from the fact that NGINX Plus is based on it, because we are committed to enterprise‑level support and process standards for NGINX Plus customers. These standards include service level agreements (SLAs) with our customers that dictate bug‑fix and testing procedures with which patches must comply, for the core software, dependencies, and supported modules alike. The SLAs help customers achieve compliance with regulations, minimizing the risk of exposure to vulnerability exploits.
An added wrinkle for NGINX Open Source is that many third‑party technologies leverage and embed it in their products. The providers of those technologies have their own processes for disclosing and patching software vulnerabilities when they are discovered. As we discuss in the next section, there is sometimes quite a significant lag between our release of a patch for NGINX Open Source and the release of a corresponding patch for third‑party technologies.
How NGINX Plus Protects Subscribers Against Vulnerabilities
In the introduction we mentioned that an often‑overlooked benefit of NGINX Plus is how it makes it quicker and easier for our customers to protect themselves from vulnerabilities. Dealing with vulnerabilities in open source software can be much more time‑consuming – and therefore expensive – than many people think.
In the following sections we discuss five specific ways that resolving vulnerabilities is faster and easier for NGINX Plus subscribers.
NGINX Proactively Informs NGINX Plus Subscribers About Patches
When we release a security patch for NGINX Open Source and NGINX Plus, we proactively inform NGINX Plus customers about it by email. We also publish a blog announcing the availability of most patches, but we have no way to reach NGINX Open Source users directly. That puts the burden on them to monitor CVEs and other vulnerability databases and to check our blog regularly.
F5 SIRT Provides Real-Time Help to NGINX Plus Subscribers Under Attack
When F5 customers, including NGINX Plus subscribers, fall victim to attacks, the F5 Security Incident Response Team (SIRT) is there to provide real‑time assistance. The SIRT works quickly to effectively mitigate attacks and get you back up and running. They continue to work with you even after the attack stops – they “look beyond the reported incident to reduce the overall harm to your organization, as well as understand, anticipate, and deter future threats”.
NGINX App Protect Steps Up Application Protection
NGINX App Protect is an enterprise‑grade web application firewall (WAF) powered by F5’s 20 years of security experience and deployed as an NGINX Plus dynamic module. It steps up NGINX Plus’s Layer 7 security with application‑specific protection against even more CVEs in your backend application servers. NGINX and F5 are committed to providing signatures related to specific attack campaigns. The result? NGINX App Protect frees NGINX Plus subscribers from building their own signatures and dealing with multiple false positives.
NGINX Plus Supports JWT-Based and OpenID Connect Authentication
Even in the absence of specific vulnerabilities that require patches, sophisticated authentication protocols help prevent bad actors from accessing your apps and infrastructure. In addition to the methods available in NGINX Open Source, NGINX Plus supports authentication based on JSON Web Token (JWT) and OpenID Connect, for web as well as API clients.
NGINX Plus Subscribers Get Patches Immediately
As described in Reporting a Vulnerability, when a vulnerability affects NGINX software, we usually get notified before it is publicly disclosed as a CVE. The advance warning enables us to prepare a patch before the public disclosure, and we generally release the patch on the day of the disclosure (or within a few days in some cases). The fix is available to NGINX Plus customers in the form of patched binaries. It’s available to NGINX Open Source users in the form of corrected source code and patched binaries for supported OSs, but as noted above we have no way to inform NGINX Open Source users directly.
Third‑party technologies that leverage or embed NGINX Open Source might not become aware of the vulnerability before its disclosure. Even if they do, our experience is that their patches can lag behind the NGINX patch by several months. This is understandable, especially for open source software which is often maintained by volunteers, but it leaves your infrastructure unprotected and subject to exploitation by hackers after the vulnerability is publicly disclosed.
As an example, two CVEs about vulnerabilities with HTTP/2 were discovered in fall 2018 (CVE-2018-16843 and CVE-2018-16844). We applied security patches to NGINX Plus R16 and NGINX Open Source 1.15.6 in response. The popular OpenResty software, which builds on NGINX Open Source to provide some of the functionality in NGINX Plus, first issued a release candidate that incorporated these patches on 3 March 2019 – a full 4 months after the patch from NGINX. Solutions built on top of OpenResty then usually take even longer to patch their code base.
Sometimes, the status of a vulnerability isn’t clear, or the software provider doesn’t believe a patch is necessary even if it constitutes a potential attack vector. Such a situation occurred during 2020 with ModSecurity, the most widely used open source WAF software. The team that maintains the OWASP ModSecurity Core Rule Set (CRS) discovered that the way ModSecurity v3 does global matching of regular expressions can result in what the OWASP CRS team considers a denial of service vulnerability (CVE-2020-15598). The organization that maintains ModSecurity itself, Trustwave, disputed that the behavior is a security issue, and declined to issue a patch.
NGINX ModSecurity WAF is a dynamic module for NGINX Plus built on ModSecurity v3. The NGINX team decided that the behavior described in CVE-2020-15598 had enough potential to cause a DoS vulnerability that we issued a patch in September 2020. As we explained in the blog announcing the patch, users of an open source ModSecurity build (which includes NGINX Open Source users) must decide for themselves whether to apply the patch from the OWASP CRS team, or stick with the unpatched software from Trustwave and possibly implement the mitigating changes proposed by Trustwave.
[Editor – NGINX ModSecurity WAF officially went End-of-Sale as of April 1, 2022 and is transitioning to End-of-Life effective March 31, 2024. For more details, see F5 NGINX ModSecurity WAF Is Transitioning to End-of-Life on our blog.]
Conclusion
In today’s ever‑more competitive business landscape, software teams are under pressure to deliver new and updated code as fast as possible to deliver the most innovative services. At the same time, the rise of sophisticated attacks on infrastructure and applications make it vital to track vulnerabilities and mitigate them as soon as possible, a tedious and time‑consuming practice.
An NGINX Plus subscription offloads that burden, enabling application teams to concentrate on their main mission – delivering code faster – while the organization remains protected from security vulnerabilities.
Try out all the advanced features of NGINX Plus for yourself – start your free 30-day trial today or contact us to discuss your use cases.
