Globo.com Delivers Live Video Streaming to 500,000+ Viewers with NGINX

Overview

Globo.com is the largest online media group in Latin America. Serving their audience news, entertainment, and sports.

Challenge

In 2012, Globo.com, the number one media and entertainment portal in Brazil, had over a million visitors a day and was experiencing unprecedented growth in its online viewership. As the online branch of the country’s top television brand Grupo Globo, Globo.com is responsible for online distribution of the media company’s news, sports, and entertainment to a massive, and yet still fast‑growing, audience.

Visitors come to Globo.com not just to catch up on news and current events articles, but to enjoy its great video content, with many Globo news reports, TV shows, and sports events broadcast online. These users expect an easy‑to‑use, high‑quality online viewing experience that rivals that of traditional television.

The sheer amount of traffic Globo.com had to handle made delivering live video over the Internet a serious technical challenge. As an example, up to 300,000 simultaneous viewers tuned in online to watch matches live during the 2010 FIFA World Cup. At the time, Globo.com was using a proprietary media server along with Real‑Time Messaging Protocol (RTMP) to stream video from the soccer field to the Web. However, this solution proved less than ideal.

The media server used a specialized port to deliver video, which meant users behind firewalls had trouble viewing the stream. Because it was proprietary, closed‑source software, it was difficult to debug problems or add instrumentation to monitor operations. Further, the video was delivered via Flash, so users with iOS devices were unable to watch the stream.

Performance was also an issue, especially under high load. Because RTMP is a stateful protocol, each viewer had to be served from a single machine per session instead of being able to get video content from multiple servers. Load balancing was limited, failover was challenging, and caching was non‑existent.

The result was that during peak traffic times, clients were kicked off overloaded servers and automatically tried to reconnect to another one. But that server was overloaded as well, causing cascading failure. Many users, once lost, didn’t return.

As developer Juarez Bochi, video platform developer at Globo.com, explains, “We could see the audience and the money flying away through the window.”

Solution

With a growing viewer base and an increasing demand for video on mobile devices, Globo.com decided to switch from the RTMP solution that was bundled with its former media server to HTTP Live Streaming (HLS) delivered with NGINX.

HLS not only added video support for iOS devices, but its stateless HTTP delivery method enabled Globo.com to leverage caching and reap the full benefits of load balancing, with a huge impact on performance and scalability. Using HTTP‑based video delivery and NGINX also made it possible to monitor the video‑delivery architecture.

“NGINX gave us amazing caching. It also gave us easy scalability, because it’s just a matter of spinning up another machine or virtual server,” explains Leandro Moreira, software engineer at Globo.com. “We were able to achieve a better user experience because HLS is better at determining which bitrate stream the user should get. And we also loved the instrumentation that let us see what was going on in the server farm.“

NGINX powers 1 million busiest websites in the world. It’s used by leaders in online video, including Netflix and Hulu, to deliver their content quickly and reliably to the world. With out‑of‑the‑box support for popular streaming protocols like Apple’s HLS, Adobe’s HTTP Dynamic Streaming (HDS), and RTMP, NGINX optimizes the stream for every device and network. The open source NGINX software can be extended to support additional video features; NGINX Plus has advanced video features built in.

In Globo.com’s HLS streaming architecture with NGINX, raw video is converted to HLS and segmented for delivery. Segmentation enables Globo.com to implement a DVR‑type feature that was not possible with the former media server. NGINX and Lua are used in conjunction with a Redis database to generate dynamic playlists. This enables users of Globo.com’s custom video player to watch a segment, such as a home‑team goal, over and over, and skip through other parts of a game, giving them full control over their viewing experience.

NGINX makes the HLS files readily available while providing caching and redundancy. The application is built using microservices, with modules for geolocation, authorization, and authentication (crucial for respecting user and broadcaster rights). Lua, the scripting language that is also a Brazilian product, is used for playlist generation and a queuing system (“waiting room”) for users on congested connections.

NGINX also powers the frontend, delivering the video stream to iOS and Android devices, as well as via a custom web‑based Flash video player for PC users.

Results

Streaming Live Video to 500,000+ Users with Ease

Soccer is popular in Brazil, and with its already huge – and growing – online viewership, Globo.com knew that the 2014 FIFA World Cup would put its video streaming architecture to the test. Requests were load balanced across 80 frontend NGINX nodes in two data centers that cached video segments from 6 backend nodes also running NGINX.

During the 2014 World Cup, Globo.com delivered live video streams of the matches to a audience of over 500,000 viewers at peak times without a hiccup, and could easily have handled more. Video requests reached 125,000 per second and throughput delivered during the game exceeded 640 Gbps.

Video delivery was very efficient – using just 10% CPU, frontend nodes were able to deliver video at 19 Gbps, nearing their network capacity of 20 Gbps per node. It was a huge improvement over the previous architecture. As Bochi explains, “with just 2 NGINX servers, we could easily handle 100,000 users. That’s about what we could get with 50 of our previous media servers.” The system continues in use today.

Deep Visibility for Even More Infrastructure Optimizations

Their previous media server limited Globo.com to checking log files and monitoring resource usage. When Globo.com switched to HLS streaming delivered with NGINX, it gained the ability to drill down much further into its streaming infrastructure.

With NGINX, it was easy for Globo.com to add instrumentation and build a custom dashboard for their server farm, showing everything from CPU and memory utilization to video buffering and cache miss rates.

This information helped Globo.com pinpoint inefficiencies and fine‑tune its delivery. The instrumentation possible with the HLS and NGINX solution directly helped Globo.com reduce its video buffering rate by 600%, as well as optimize its caching mechanisms.

Easy to Add Microservices with NGINX and Lua

Working with NGINX was so easy that by 2013, Globo.com was already happy with the performance and scalability of its video delivery infrastructure and was able to focus on improving the user experience in preparation for the 2014 World Cup.

Besides its core functionality, NGINX has a library of third‑party modules available to extend its capability. The Lua third‑party module tightly integrates with NGINX and provides fast, powerful, asynchronous scripting through an easy‑to‑use language.

Using NGINX and Lua, Globo.com created a microservice that dynamically generates HLS playlists, which supports DVR functionality that lets its viewers rewind and rewatch great plays made on the field.

When local ISP bandwidth bottlenecks created video delivery challenges for its users, Globo.com was able to create another microservice to deal with the issue, using NGINX and Lua to create a “waiting room” when too many users tried to access video over a saturated bandwidth link. This ensured consistent video quality for its viewers and helped Globo.com maintain its reputation as a high‑quality content provider.

About Globo.com

Globo.com is the Internet branch of Organizações Globo, the biggest media conglomerate in Latin America, with offices in Rio de Janeiro and São Paulo. It aggregates the greatest web vertical portals in Brazil, for news (G1), sports (globoesporte.com), videos (GloboPlay), and celebrities (GShow). Besides that, the company serves as a service provider for all other media units of the organization, as well as providing strategic and technology support on Internet issues. It excels at high‑volume web distribution and serves the largest simultaneous video‑streaming audiences in the country. For more information, visit www.globo.com.