AppNexus’ mission is to create a better Internet, by harnessing data and machine learning to power the world’s digital audience platforms. The company’s real-time decisioning platform supports core products that enable its customers to acquire, engage, and monetize their audience.
AppNexus customers buy and sell ad impressions through a real-time bidding (RTB) application. As a web page loads, each available impression on the page triggers an HTTP retrieval to an AppNexus server, which hosts the ads. Each transaction takes mere seconds, but occurs billions of times every day. AppNexus’ worldwide data centers balance the load of incoming traffic based on network proximity to the client.
Among AppNexus’ customers are search engine companies and specialists that aggregate web page impressions from mobile devices. These supply‑side partners (SSPs) also use the AppNexus RTB capabilities, but deliver ads from their own servers instead of having AppNexus host the ad content. The extra step of fetching an ad from the SSP adds time to the process. Although there are far fewer SSPs bidding for ad slots than individual advertisers, which means there are fewer connections, each HTTP session stays open for a much longer time. That makes load balancing and capacity scaling of SSP requests much more challenging than requests for ads hosted by AppNexus.
AppNexus data centers are located around the world, and each data center houses clusters of RTB application servers. Incoming traffic is load balanced first across the data centers, based on network proximity to the client, and then load balanced across the servers in the cluster.
AppNexus’ success and growth soon revealed problems with its F5 BIG-IP hardware load balancers. With more users being served and greater demand on the RTB application, performance and stability issues began to manifest. The hardware didn’t perform up to promised specifications under real-world conditions, and AppNexus experienced high CPU usage and memory leaks.
Thanks to the growing instability of its load balancers, AppNexus had to open 32 different support tickets with F5 in just four years. Critical security vulnerabilities required constant patching. It was clear that the company’s needs had outgrown what F5 could reliably deliver.
“To improve load balancing for our SSP requests, we first tried a traditional hardware appliance,” explained Louis Mamakos, Senior Network Architect for AppNexus. “It improved load balancing, but the hardware costs negatively affected our cost of goods sold. So we started looking for software solutions.”
High-volume traffic requires a high-performance solution. To ensure app availability, AppNexus needed a product that is stable and resilient. Ease of use and operation, a smooth process for managing updates, and minimal impact from migration were other key requirements on AppNexus’ list. Most importantly, the company needed a product that could scale in line with the growth in traffic and users.
With existing F5 BIG-IP hardware no longer keeping up with load-balancing needs, AppNexus began to search for new solutions. The incumbent supplier simply advised them to deploy additional load balancers. This was impractical and expensive, and came with no guarantee of improved performance in the long term. It would also cost a projected $1 million per data center – $3 million in total for the overall deployment.
Of all technology solutions considered by AppNexus, NGINX Plus stood out as the most cost effective, turning that $3 million projection into just $144,000 in final deployment and migration costs. The AppNexus team already had experience with NGINX products for content delivery, so an added benefit was existing operational and technical familiarity.
AppNexus had also shortlisted HAProxy and Apache HTTP Server for reverse proxying, but had concerns about their ability to scale in a multi-threaded environment. Auctions typically run on 12‑core Intel Xeon CPUs. Because AppNexus also offers cloud services hosted on these same systems, taking advantage of their multi-threading capability makes most efficient use of the available computing power.
It soon became apparent that NGINX Plus was the clear choice. “We looked at NGINX because they responded probably to 90% of what we were looking for,” said Ernesto Chaves, AppNexus Network Engineer, speaking at the NGINX annual conference in 2018. “We didn’t want to look for anything else.”
NGINX is the world’s most popular open source web server for high‑traffic websites, and NGINX Plus adds advanced load balancing, session persistence, live activity monitoring, and other features to provide reliability, control, and consistent performance for HTTP applications.
AppNexus initially deployed three instances of NGINX Plus to manage RTB application traffic, combined with a DNS round‑robin load balancer to manage availability of individual software load balancers. NGINX Plus runs on Dell PowerEdge C6220 systems with 12‑core Intel Xeon E5‑2630L CPUs running at 2.0GHz. These systems are configured with dual, bonded 10‑GbE interfaces.
Begun in mid-2015, the migration from F5 to NGINX Plus was complete in February 2018. “We could really see the performance that we didn’t have with the F5s,” said Chaves.
Ensure Frictionless Application Availability
AppNexus performs rolling upgrades on its servers, meaning that at any given time a group of servers might be out of service for maintenance. In the past, requests directed to unavailable servers had to be retried, which increased the risk of missing the ad placement window. With its active health checks capability, NGINX Plus continually checks the health of upstream servers and does not send requests to servers that are offline. When the server is back in service and the health check succeeds, NGINX Plus gradually reintroduces the server into the load-balancing pool. This ensures that even during maintenance customers aren’t missing out on ad-impression opportunities.
Application Delivery Costs Reduced by over 95%
NGINX Plus cut AppNexus’ deployment costs by over 95%, from a projected $3 million to just $144,000.
And the savings didn’t end there. As a load-balancing solution with a very small footprint, NGINX Plus eliminates the need for resource-hungry servers and associated data center costs. Less resources are used for each request, especially for SSPs, keeping the operational cost low.
Thanks to the automation, intelligent metrics, and stability offered by NGINX Plus, the AppNexus team can focus on innovation. With less time consumed in maintenance, troubleshooting, and micromanagement, the AppNexus team is free to focus on development of its core product without having to worry about managing network traffic.
Deep Visibility Accelerates Troubleshooting and Performance Tuning
With thousands of servers generating more than one million data points per minute, ad hoc reporting is impossible. Mamakos said that request logging is impractical because within minutes he’s drowning in log files. NGINX Plus offers automated statistics, fully integrated with AppNexus’ real-time graphing system.
The NGINX Plus API gives AppNexus a full picture of all the metrics it needs. “The detailed statistics that NGINX Plus delivers help us monitor the software and characterize overall performance,” said Mamakos. “We gain visibility into how NGINX Plus sees requests and activity. By knowing how NGINX Plus sees things, we can accelerate troubleshooting, continuously tune performance, and better align our systems to handle whatever load‑balancing needs come along.”
Opening Up New Opportunities
Since initially deploying NGINX Plus in 2014 for one SSP partner, AppNexus has expanded its use to additional partners. Taking advantage of the Lua dynamic module for NGINX Plus, AppNexus embeds custom Lua code in the NGINX Plus configuration to gain even deeper visibility into statistics and to customize functions, such as packet inspection, without having to build special tools.
“As our company and this market mature, we’re increasingly focusing on optimizing and driving efficiency,” concluded Mamakos. “NGINX Plus offers us numerous ways to uncover opportunities and monetize efficiency. It’s really an engine for addressing future customer services and growing our business.”
AppNexus is a technology company that provides trading solutions and powers marketplaces for Internet advertising. Its open, unified, and powerful programmatic platform empowers customers to more effectively buy and sell media, allowing them to innovate, differentiate, and transform their businesses. As the world’s leading independent ad tech company, AppNexus is led by the pioneers of the web’s original ad exchanges. Headquartered in New York City with 23 global offices, AppNexus employs more than 1,000 of the brightest minds in advertising and technology who believe that advertising powers the Internet. For more information, visit www.AppNexus.com.