Solving a Load‑Balancing Issue and Opening the Door to New Opportunities at the Same Time
Internet advertising helps companies to reach their target audiences, creators to get the most from their content, and consumers to get access to online experiences. Placing and presenting the right ads to the right viewer at the right time is incredibly complex. Many ads are dynamically selected for individual users through a real‑time bidding process that occurs within milliseconds behind the scenes. And that bidding runs on AppNexus infrastructure. AppNexus is the world’s leading independent ad tech company. It provides an open, powerful programmatic platform that empowers its customers – including some of the world’s most influential advertising and media companies – to more effectively buy and sell ads. Today, AppNexus relies on NGINX Plus to handle some of its most challenging customer use cases.
AppNexus customers buy and sell ad impressions through a real‑time bidding (RTB) application. As a web page loads, each available impression on the page triggers an HTTP retrieval to an AppNexus server, which hosts the ads. Advertisers bid to have their ad served to the user, and the winner’s ad is delivered to the user’s page. The entire process takes just a few hundred milliseconds, but it occurs billions of times each day.
AppNexus customers also include search engine companies and specialists who aggregate web page impressions from mobile devices. These supply‑side partners (SSPs) also use the AppNexus RTB capabilities, but deliver ads from their own servers instead of having AppNexus host the ad content. The extra step of fetching an ad from the SSP adds time to the process. Although there are far fewer SSPs bidding for ad slots than individual advertisers, which means there are fewer connections, their HTTP sessions stay open much longer. That makes load balancing and capacity scaling of SSP requests much more challenging.
AppNexus data centers are located around the world, and each data center houses clusters of RTB application servers. Incoming traffic is load balanced first across the data centers, based on network proximity to the client, and then load balanced across the servers in the cluster.
“To improve load balancing for our SSP requests, we first tried a traditional hardware appliance,” says Louis Mamakos, Senior Network Architect for AppNexus. “It improved load balancing, but the hardware costs negatively affected our cost of goods sold. So we started looking for software solutions.”
AppNexus considered HAProxy and Apache HTTP Server for reverse proxying but had concerns about their multithreading scaling abilities. Auctions typically run on 12‑core Intel Xeon CPUs. Because AppNexus also offers cloud services hosted on these same systems, taking advantage of their multithreading capability makes most efficient use of the available computing power. AppNexus already used open source NGINX for some content delivery. When the company discovered NGINX Plus, it found its solution for load balancing SSP requests.
NGINX is the world’s most popular open source web server for high‑traffic websites, and NGINX Plus adds advanced load balancing, session persistence, live activity monitoring, and other features to provide reliability, control, and consistent performance for HTTP applications. AppNexus initially deployed three instances of NGINX Plus to manage RTB application traffic, combined with a DNS round‑robin load balancer to manage availability of individual software load balancers. NGINX Plus runs on Dell PowerEdge C6220 systems with 12‑core Intel Xeon E5‑2630L CPUs running at 2.0GHz. These systems are configured with dual, bonded 10‑GbE interfaces.
AppNexus automates configuration of its NGINX Plus solution using a configuration database and Puppet. Automation is critical to the company’s success, enabling it to efficiently scale out while keeping costs under control.
Health Checks Maintain High Availability
One of the main reasons that AppNexus chose NGINX Plus was for its active application health checks. AppNexus performs rolling upgrades on its servers, meaning that at any given time, a group of servers might be out of service for maintenance. In the past, requests directed to unavailable servers had to be retried, which increased the risk of missing the ad placement window. NGINX Plus continually checks the health of upstream servers and does not send requests to servers that are offline. When maintenance is complete and the team places the server back in service, the health check succeeds, and NGINX Plus gradually reintroduces the server into the load‑balanced cluster using a slow‑start feature. NGINX Plus health checks help ensure that customers don’t miss ad impression opportunities when servers are taken down for maintenance.
Detailed Statistics Simplify Operational Insight
With thousands of servers generating more than one million data points per minute, ad hoc reporting is impossible. Mamakos says request logging is impractical because within minutes he’s drowning in log files. Instead, the team relies on automated statistics from NGINX Plus, which are fed into AppNexus’ Graphite real‑time graphing system. From there, other solutions extract and analyze data.
“The detailed statistics that NGINX Plus delivers help us monitor the software and characterize overall performance,” says Mamakos. “We gain visibility into how NGINX Plus sees requests and activity. By knowing how NGINX Plus sees things, we can accelerate troubleshooting, continuously tune performance, and better align our systems to handle whatever load‑balancing needs come along.”
Cost Effectiveness Improves the Bottom Line
Even with ultra‑high transaction volumes, acceptable margins demand cost‑effective solutions. With NGINX Plus, AppNexus gained a highly cost‑effective way to improve SSP request performance and maintain a low cost of goods sold. Automation also saves time and frees the AppNexus team to focus on innovation. As time goes on, AppNexus can scale its capabilities while maximizing its staff’s time.
Since initially deploying NGINX Plus in 2014 for one SSP partner, AppNexus has expanded its use to additional partners. In addition, Lua programming features will allow AppNexus to gain even deeper visibility into statistics and to customize functions, such as packet inspection, without having to build special tools.
“As our company and this market mature, we’re increasingly focusing on optimizing and driving efficiency,” says Mamakos. “NGINX Plus offers us numerous ways to uncover opportunities and monetize efficiency. It’s really an engine for addressing future customer services and growing our business.”
AppNexus is a technology company that provides trading solutions and powers marketplaces for Internet advertising. Its open, unified, and powerful programmatic platform empowers customers to more effectively buy and sell media, allowing them to innovate, differentiate, and transform their businesses. As the world’s leading independent ad tech company, AppNexus is led by the pioneers of the web’s original ad exchanges. Headquartered in New York City with 23 global offices, AppNexus employs more than 1000 of the brightest minds in advertising and technology who believe that advertising powers the Internet. For more information, visit www.AppNexus.com.