Booxware Software Development GmbH is a German-based company that develops and operates gaming applications for Tipico, an international provider of sports betting and casino games. Sports betting is extremely popular, with millions of bets being placed each day through Booxware’s systems. Booxware runs approximately 200 applications on 120 servers to provide high-quality services to sports enthusiasts across Europe.
Sports betting is heavily regulated, and gaming companies like Booxware regularly upgrade their systems to stay in compliance with industry requirements as they evolve to incorporate the latest best practices in software security. For several years, Booxware had successfully used the open source NGINX software for basic load balancing and SSL termination. Previous upgrades to new versions of open source NGINX had all gone smoothly, but the last couple times Booxware attempted an upgrade it ran into a problem with a third-party plug-in for health checks that it had been compiling separately into NGINX.
“Updating open source NGINX is not normally an issue. However the last two times we upgraded NGINX, we noticed an issue with the third-party plug-in we had been compiling. Our load balancers are in a multicore, multiprocessor environment where the processors need to interact with one another to effectively direct traffic. We were seeing a mutex issue with the third-party health check plug-in – for some upstreams it didn’t work and was blocked completely. It was a weird issue, at least for our setup. We either needed to fix the plug-in or find another way to get health checks working as we needed,” explains Sabine Benz-Reichel, Team Lead Operations at Booxware.
Booxware was already familiar with the benefits of open source NGINX, so when they needed to get health checks working quickly in their environment, they turned to NGINX Plus.
“We either had to get someone to fix the third-party plug-in – which would take at least several days and would come with a cost – or we could quickly switch to NGINX Plus, which was the cheaper solution and would allow us to be compliant and secure right away,” says Benz-Reichel.
Since Booxware was already using open source NGINX, upgrading to NGINX Plus was easy. Switching was simply a matter of creating a new folder for their license keys and replacing the NGINX binary with the NGINX Plus binary in their repository. And because Booxware had already been using a deployment script with Ansible, they were able to roll out their NGINX Plus load balancers without any downtime.
Booxware is now running eight instances of NGINX Plus in its data center – six instances for load balancing external traffic, one for load balancing internal traffic between services, and one that acts as a failover, enabling the IT team to repair hardware as needed simply by switching the IP address.
> Reliable, Responsive Health Checks
After switching, Booxware happily discovered that the health checks in NGINX Plus were actually more responsive than the third-party plug-in. When load balancing high-traffic applications, active health checks are critical – NGINX Plus automatically routes traffic away from unhealthy servers to healthy ones, so users don’t see error pages from servers that have failed or been taken down for maintenance.
“Since the change to NGINX Plus, we have no issues anymore with health checks. Even better, the health checks are more responsive than the old ones from the third party, and they are acting far faster,” notes Benz-Reichel.
Active health checks are already included within NGINX Plus, so there’s no need to compile a separate third-party module. This saves time, simplifies upgrades, and gives customers peace of mind knowing that the health checks are fully tested and supported by NGINX engineers.
> Live Activity Monitoring for Enhanced Visibility
With 120 servers delivering 200 applications, visibility into how servers are functioning is an essential part of keeping Booxware running smoothly. The live activity monitoring feature in NGINX Plus reports a much larger number of statistics than the open source NGINX
stub_status module. Key load and performance metrics are displayed on a built-in, easy-to-read dashboard, and are also available through a RESTful JSON interface to easily feed into third-party dashboards and application monitoring tools.
“Our managers quickly know what’s going on with our infrastructure by using the NGINX Plus status page. And I use the JSON REST API to have an overview of our backends through Zabbix. This allows me to see which backends are responding and which are not, without sending the checks multiple times, simply by grabbing the information from the NGINX Plus status page,” explains Benz-Reichel. He adds, “we wanted our load balancers to also indirectly monitor whether or not the network has issues, and the NGINX Plus status page does just that. If we see that some of the servers didn’t come back up after doing maintenance, we have that visibility and can easily troubleshoot the issue.”
> High-Performance Load Balancing for Complex Applications
NGINX Plus provides Booxware with seamless load balancing for its complex gaming application. With users spanning a range of devices and platforms, Booxware’s application has multiple tiers, each with its own separate backend: a web tier, a mobile tier, a “next generation” web and mobile tier, a tablet tier, and a shop tier. Incoming web traffic comes from Booxware’s CDN provider to NGINX Plus load balancers, which perform SSL termination and use DNS records to send the traffic to the appropriate tier backends. Load balancing all this is a hefty challenge but NGINX Plus has worked flawlessly for Booxware.
“We measure our business in bets delivered. At peak times, which for us is on Saturdays, we process 4,500 bets per second, or around 300 megabits per second on each of our NGINX Plus load balancers. We’ve found that NGINX Plus manages all of our traffic easily and efficiently,” says Benz-Reichel.
> Smooth Transition to Microservices
Using NGINX Plus has helped Booxware begin its recent transition from a traditional monolithic application to a more agile, microservice-based architecture. Instead of adding features by updating its main application, Booxware now adds features by creating lightweight microservices, which can be developed independently and load balanced for resiliency. Using intelligent Layer 7 load balancing, NGINX Plus easily routes traffic to the proper application server.
“With NGINX Plus, we’re able to do some magic using regular expressions, where we route our traffic based on the URLs. Using regular expressions, we make sure that traffic handled by a microservice is directed to it, while what is left goes to the main application,” explains Benz-Reichel.
NGINX Plus provides both load balancing and SSL/TLS termination for Booxware’s microservices. Incoming traffic to the internal load balancer is decrypted for routing and Layer 7 load balancing, and then re-encrypted as it is sent to the appropriate microservice, providing robust load balancing and end-to-end security.
Benz-Reichel adds, “It was very nice to be able to migrate from our old system to the new system using NGINX Plus in front. NGINX Plus lets you easily split up parts of your platform that you don’t want to keep in your main application and send it to microservices. This is exactly what we are doing.“
In 2002, a cooperation between bookmakers and software developers set itself the goal of bringing enthusiasm for sport and the excitement of betting even closer together. Out of this cooperation, Booxware was born: dedicated to providing high-quality hardware and software solutions for the sports betting industry from a single source.
Today, Booxware Software Systems and Booxware Software Development are among the pioneers in this specialized field, working both independently and in cooperation with Europe’s leading betting providers and their franchise partners. They are constantly honing new ideas for even better hardware and software modules to deliver uncompromising betting pleasure – for customers and for partners.