Load Balancing with Confidence Across a Large IT Environment
Locaweb is the largest provider of web hosting and infrastructure services in South America, and is the clear leader in Brazil. The company serves more than 265,000 customers, hosts more than 300,000 sites, and operates 20,000 total servers, both on premises and in the cloud. Although hosting is a large part of Locaweb’s business, the company also offers a full suite of Internet services, such as ecommerce, search marketing, software as a service (SaaS), cloud infrastructure as a service (IaaS), platform as a service (PaaS), and email. As a large email provider, Locaweb supports more than 10 million inbound email addresses, and processes roughly 200 million email messages a day.
Not surprisingly, Locaweb has faced a growing need for load‑balancing solutions over the past few years. The need became acute in 2014 when Locaweb enhanced email security. Locaweb used to make SSL/TLS optional for email, but switched to secure connections for all mail traffic to better meet its own security goals and the needs of its customers. This pleased customers greatly, and resulted in better security overall, but the hardware load balancers couldn’t handle the throughput over SSL/TLS. Locaweb began investigating alternative load balancers.
“We tried other software and hardware load balancers, but in our internal tests we didn’t get the high throughput and performance that we needed, especially with SSL. In addition, hardware licenses are very expensive,” says Danilo Acquaviva, Systems Administrator at Locaweb. “We were already familiar with NGINX as a cache system, proxy server, and web server. We were very interested in trying NGINX Plus for its advanced load‑balancing features.”
NGINX is the world’s most popular open source web server for high‑traffic websites. With NGINX Plus, customers get additional enterprise‑ready features such as HTTP and TCP load balancing, health checks, session persistence, advanced monitoring, and more.
In Locaweb’s internal tests, NGINX Plus clearly outperformed the other load balancers. Now, all email services at Locaweb are running behind NGINX Plus.
The company replaced a cluster of hardware appliances with NGINX Plus load balancers. For its internal email platform, 3 NGINX Plus machines load balance traffic across about 80 backend servers hosting 3.4 million mailboxes. For its large enterprise customers, Locaweb has deployed 2 additional clusters – one with 3 NGINX Plus machines handling 37 servers and 2.2 million mailboxes, and the other with 3 NGINX Plus machines load balancing 60 servers and 800,000 mailboxes. In the second cluster, NGINX Plus is load balancing ActiveSync protocol traffic too.
Further, because all Locaweb websites use the same environment for email, it was an easy decision to switch to NGINX Plus for load balancing all of the websites as well. Locaweb deployed another three NGINX Plus machines for this purpose. All NGINX Plus instances are working as load balancers and some have caching enabled. They are deployed on physical servers, the biggest with 196 GB of RAM, 16 processors, 2 SAS disks in RAID1 (1T), and a 10‑Gbps network interface.
Migrating a large email platform to a new load‑balancing solution is a significant project, and one that Locaweb wanted to implement carefully in order to avoid negatively impacting customers as it deployed the enhancements to its offerings. Senior sysadmins migrated a few domains per week in small steps. The overall migration from hardware load balancers to NGINX Plus took about four months.
NGINX Plus is a complete application delivery platform, optimized for handling high traffic volumes. Its sophisticated event‑driven architecture enables NGINX Plus to scale and handle a very large number of simultaneous connections using a very small memory footprint.
“The biggest benefit for us with NGINX Plus is its amazing performance. We get incredibly high throughput over SSL. So high that it seems unreal. We have 40,000 simultaneous connections in just one portion of our webmail platform alone, and NGINX Plus handles the load with ease. The performance we get with NGINX Plus helps us ensure our customers stay happy with our offerings,” says Acquaviva.
Better Visibility Allows for Rapid Response
Using the NGINX Plus live activity monitoring dashboard, Locaweb’s teams have deep visibility into how their backends and load balancers are working. They can easily see how much traffic each server is handling, what kind of traffic is coming in, where there are failures, and more. This is extremely helpful for system maintenance and for addressing possible incidents. NGINX Plus provides this data in a JSON format, which Locaweb feeds into its own monitoring system.
“The metrics from NGINX Plus help us achieve the uptime and performance we need,” says Claudio Borges da Silva Filho, Technical Leader of the Linux Team. “We developed a system for monitoring using the JSON output from NGINX Plus. If a server stays out of its pool for a specific period of time, we automatically trigger an internal incident report and our staff check the issue. The visibility we get with NGINX Plus is very useful to us.”
Fast and Easy to Deploy New Configurations
NGINX Plus easily integrates into the DevOps culture and workflows at Locaweb. The technical teams are using a DevOps style for deployment to production. They have defined a configuration manager role, which allows specific people to deploy new configurations. With the vast size of their environment, the role of configuration manager is a big deal.
“One small mistake in deployment can result in a big problem over here. We were delighted that NGINX Plus easily integrated into our DevOps environment. NGINX Plus works well with CFEngine, our configuration management system. We can replace variables in the templates and can have the same configurations deployed across the entire environment in no time at all,” says Filho.
More Flexibility Helps Ensure a High‑Quality User Experience
Locaweb uses the on‑the‑fly reconfiguration feature in NGINX Plus to easily add and remove servers from upstream groups. Locaweb’s sysadmins are even building a control panel on top of NGINX Plus to further optimize it for their environment. Locaweb needs to respond quickly if a server crashes, or a hard drive or network interface needs to be replaced. Building on NGINX Plus’s on‑the‑fly reconfiguration capability, Locaweb’s control panel will give its sysadmins control over the specific servers they maintain within the larger environment. This will enable more sysadmins to remove or add servers rapidly without physical access to them, and to respond quickly to potential issues to ensure a high‑quality user experience.
Exceeds Expectations at a Fraction of the Cost
Locaweb found that costs are much lower with NGINX Plus, on several levels. Using NGINX Plus for load balancing replaces hardware appliances and add‑on licenses that cost much more – and still weren’t doing the job. It would take a very large amount of regular servers to get the same performance without load balancing; a very expensive proposition. And Locaweb was already very familiar with the open source NGINX software, which meant they could quickly get up and running with the advanced capabilities of NGINX Plus.
Locaweb plans to migrate even more hardware load balancers to NGINX Plus and use it as its main load‑balancing solution.
Locaweb is the leading provider of hosting infrastructure services in Brazil, according to IDC. With 17 years of experience, about 1,000 employees, and 265,000 customers, Locaweb offers software, platform, infrastructure, and marketing solutions. Its Enterprise Solutions unit is responsible for project delivery, from simple projects to fully customized scopes. With capacity for more than 25,000 servers, according to the Hostmapper report, the company is the leader in the market for .com.br domains. Between 2012 and 2013, Locaweb acquired four new companies: Eventials, Tray, SuperPay, and All In Mail. Locaweb believes that innovation, quality of service, and its highly qualified team are the key factors to achieving success in the market. For more information, visit www.locaweb.com.br.