When companies don’t want search engines or third parties stealing their Web-based information, they turn to Distil Networks. Distil offers Web-based security products that can prevent bots from scraping text, a popular way for criminals to steal sensitive data, or even entire databases. With so many malicious bots crawling the Web every second of every day, Distil’s full time focus is on keeping them out of customer sites. Andrew Stein, Co-founder and chief scientist of Distil Networks, puts it succinctly: “At Distil Networks, we block bots. We basically stop any form of automation against a Website that the site owners don’t want.”
Distil Networks’ entire business is built around being able to quickly determine whether a Website visitor is a human, or a bit of automated code, like a bot. Humans need to reach their end destination quickly and reliably, while bots need to be bounced off the site, or given a dead-end page with no links to follow.
Distil Networks provides this capability as an on-demand service, the company must quickly handle incoming traffic and properly serve it the content required. This must be done without adding any latency to legitimate user’s load times.
“When these bots come, we have to serve a page back to them quickly. We want to make sure that not only are we serving real people information as quickly as possible, but we want to make sure that whenever we detect a bot or some bad actor against a Website, but that we can also return information to them as quickly as possible just to get them off the site,” said Stein.
Using NGINX as a reverse proxy, Distil Networks is able to bring incoming Web traffic through a single layer where actions can be performed, and HTTP packets can be inspected. With the deeply configurable and extensible inner-workings of NGINX brought to bear upon the problem, Stein said the Distil Networks team was able to utilize their prefered programming languages to build solutions on top.
“For our customers, as much as they want bots to not access their Website, they also couldn’t deal with major slowdowns, so NGINX’s ability to serve pages back fast, and to be extensible enough to allow us to use additional programming languages to help us make those decisions, is invaluable. I don’t think we’d have been able to necessarily do this with any other Web server platform,” said Stein.
Using NGINX as a caching layer in addition to a reverse proxy has also allowed Stein’s team to enhance page loading times for end users. Using a RAM disk or other fast storage medium allows NGINX to quickly serve information from memory, pushing out hundreds of thousands of page requests a second on even a low end server. This means that both successful and failed page access attempts are immediately fulfilled by the same server making the decisions, and without having to touch a hard disk.
Distil Networks’ entire business is built on top of NGINX. As Stein puts it, “I don’t think Distil Networks really happens without NGINX as a backbone layer, just because a lot of the other Web servers we looked at or were introduced to didn’t have either the extensibility, the speed, or the ability to scale that NGINX does. The speed and reliability is kind of the main draw to us.”
About Distil Networks
Distil Networks was founded in April 2011 by Rami Essaid, Engin Akyol, and Andrew Stein. The initial goal of the company was to defend websites, APIs, and mobile apps against the rising threat posed by malicious bots. Today, the company offers an even wider range of security and threat mitigation tools, including systems to prevent Web scraping. Distil Networks is used by such companies as StubHub, Crunchbase, and Doctor.com.