Solving a Load‑Balancing Issue and Opening the Door to New Opportunities at the Same Time
Internet advertising helps companies to reach their target audiences, creators to get the most from their content, and consumers to get access to online experiences. Placing and presenting the right ads to the right viewer at the right time is incredibly complex. Many ads are dynamically selected for individual users through a real‑time bidding process that occurs behind the scenes in milliseconds. And that bidding runs on AppNexus infrastructure. AppNexus is the world’s leading independent ad tech company. It provides an open, powerful programmatic platform that empowers its customers – including some of the world’s most influential advertising and media companies – to more effectively buy and sell ads. Today, AppNexus relies on NGINX Plus to handle some of its most challenging customer use cases.
AppNexus customers buy and sell ad impressions through a real‑time bidding (RTB) application. As a web page loads, each available impression on the page triggers an HTTP retrieval to an AppNexus server, which hosts the ads. Advertisers bid to have their ad served to the user, and the winner’s ad is delivered to the user’s page. The entire process takes just a few hundred milliseconds, but it occurs billions of times each day.
AppNexus customers also include search engine companies and specialists who aggregate web page impressions from mobile devices. These supply‑side partners (SSPs) also use the AppNexus RTB capabilities, but instead of AppNexus hosting the ad content, the SSP delivers ads from its own servers. This extra step adds time to the process. There are far fewer SSPs bidding for ad slots than individual advertisers, which means there are fewer connections, but HTTP sessions stay open much longer. This fact makes load balancing and capacity scaling much more challenging for SSP requests.
AppNexus data centers are located around the world, and each data center houses clusters of RTB application servers. Incoming traffic is load balanced first across the data centers, based on network proximity to the client, and then load balanced across the servers in the cluster.
“To improve load balancing for our SSP requests, we first tried a traditional hardware appliance,” said Louis Mamakos, Senior Network Architect for AppNexus. “It improved load balancing, but the hardware costs negatively affected our cost of goods sold. So we started looking for software solutions.”
AppNexus considered HAProxy and Apache reverse proxy but had concerns about their multithreading scaling abilities. Auctions typically run on 12‑core Intel Xeon CPUs. Because AppNexus also offers cloud services from these same systems, taking advantage of the multithreading capability of these systems would make most efficient use of the available computing power. AppNexus already used NGINX open source for some of its content delivery needs. When the company discovered NGINX Plus, it found what it needed to load balance SSP requests.
NGINX is the world’s most popular open source web server for high‑traffic websites, and NGINX Plus adds load balancing, session persistence, monitoring, and other features to provide reliability, control, and consistent performance for HTTP applications. AppNexus initially deployed three instances of NGINX Plus to manage RTB application traffic, combined with a DNS round‑robin load balancer to manage availability of individual software load balancers. NGINX Plus runs on Dell PowerEdge C6220 systems with 12‑core Intel Xeon E5‑2630L CPUs running at 2.0GHz. These systems are configured with dual, bonded 10‑GbE interfaces.
AppNexus automates configuration of its NGINX Plus solution using a configuration database and Puppet. Automation is critical to the company’s success, enabling it to efficiently scale out while keeping costs under control.
> Health Checks Maintain High Availability
One of the main reasons that AppNexus chose NGINX Plus was for its proactive health check features. AppNexus performs rolling upgrades on its servers, meaning that at any given time, a group of servers might be out of service for maintenance. In the past, requests directed to unavailable servers had to be retried, which increased the risk of missing the ad placement window. NGINX Plus continually checks the health of upstream servers and does not send requests to servers that have failed. When maintenance is complete and the team places the server back in service, the health check succeeds, and NGINX Plus gradually reintroduces the server into the load‑balanced cluster using a slow‑start feature. NGINX Plus health checks help ensure that customers don’t miss ad impression opportunities when servers are taken down for maintenance.
> Detailed Statistics Simplify Operational Insight
With thousands of servers generating more than one million data points per minute, ad hoc reporting is impossible. Mamakos said that they could simply turn on request logging, but within minutes would be drowning in log files. Instead, the team relies on automated statistics from NGINX Plus, which are fed into AppNexus’ Graphite real‑time graphing system. From there, other solutions extract and analyze data.
“The detailed statistics that NGINX Plus delivers help us monitor the software and characterize overall performance,” said Mamakos. “We gain visibility into how NGINX Plus see requests and activity. By knowing how NGINX Plus sees things, we can accelerate troubleshooting, continuously tune performance, and better align our systems to handle whatever load‑balancing needs come along.”
> Cost Effectiveness Improves the Bottom Line
Even with ultra‑high transaction volumes, acceptable margins demand cost‑effective solutions. With NGINX Plus, AppNexus gained a highly cost‑effective way to improve SSP request performance and maintain a low cost of goods sold. Automation also saves time and frees the AppNexus team to focus on innovation. As time goes on, AppNexus can scale its capabilities while maximizing its staff’s time.
> Opportunities Abound
Since initially deploying NGINX Plus in 2014 for one SSP partner, AppNexus has expanded its use to additional partners. In addition, Lua programming features will allow AppNexus to gain even deeper visibility into statistics and to customize functions, such as packet inspection, without having to build special tools.
“As our company and this market mature, we’re increasingly focusing on optimizing and driving efficiency,” said Mamakos. “NGINX Plus offers us numerous ways to uncover opportunities and monetize efficiency. It’s really an engine for addressing future customer services and growing our business.”
AppNexus is a technology company that provides trading solutions and powers marketplaces for Internet advertising. Its open, unified, and powerful programmatic platform empowers customers to more effectively buy and sell media, allowing them to innovate, differentiate, and transform their businesses. As the world’s leading independent ad tech company, AppNexus is led by the pioneers of the web’s original ad exchanges. Headquartered in New York City with 23 global offices, AppNexus employs more than 1000 of the brightest minds in advertising and technology who believe that advertising powers the Internet. For more information, follow @AppNexus or visit www.AppNexus.com.