How high online traffic can crash your website

How does high traffic actually crash an ecommerce website or app? Many blogs will tell you that high traffic can crash a site. But if you don’t understand why this happens, how can you expect to avoid the problem? This blog will show you how high traffic causes websites and apps to crash and gives you 4 steps to avoid website failure.

Published: 31. May 2019
waves crashing against shore

Surging online traffic is a dream for any website manager, but especially for ecommerce. More traffic means more customers and more sales.

But it can also be too much of a good thing. You can be a victim of your own success. If you don’t take proactive steps to address website or app performance, peaks in online traffic can cause websites or apps to slow down and ultimately crash.

And let’s be clear—a slowdown is about as bad as a full-on crash. A 2017 study found as little as a 100-millisecond delay decreased conversion rates by 7 percent. Mobile shoppers are especially impatient. According to research by Google, 53% of mobile users entirely abandon websites that take longer than 3 seconds to load.

website broken or just slow meme image

Website failures are disastrous for ecommerce operations. Imagine running a physical store where the front doors haphazardly lock and unlock, the lights flicker on and off, and products on the shelves appear and reappear at random. This is what it’s like for shoppers when an ecommerce site fails. No wonder 79% of shoppers who encounter poor website performance avoid purchasing from that retailer in the future.

To be sure, traffic overload is not the only reason websites fail. Developers introduce code errors, like when a typo at Amazon took out a backbone of the internet. Domain name system (DNS) providers fail, like when a DDoS attack against DNS provider Dyn caused dozens of top websites to go dark. And hosting providers experience downtime, like when roadwork damaged a fiber cable to a server center causing outages at over 2,000 Danish websites on Black Friday 2018.

But crashes due to high web traffic feel the most frustrating. They can stem from doing your job too well. Imagine the frustration of an ecommerce director who crafts the perfect viral campaign, only to see the website buckle under the pressure. There’s nothing more heartbreaking than seeing your website or app fail just when you have the most to gain.      

Most blogs and articles just state traffic-induced crashes as something that can happen. But if you don’t have a deeper understanding, how can you expect to avoid the problem?

How does high traffic actually crash a website or app?

traffic cones on laptop computer

Simply put: Insufficient resources leads to system overload

Website overload affects ecommerce sites in the same way as other websites.

The root cause is a mismatch between traffic levels and website infrastructure capacity over a given timeframe. In other words, system requests—made by visitors browsing your website—exceed the processing capacity resources of your site and any third-party systems involved in the customer journey.

When this happens, your website performance will slow and, for some or all users, crash entirely.

Distribution of traffic is key

Not all requests are equal, though. Some requests are more complex or demanding than others. And the patterns of requests are as important to understand as the amount.

This is why thinking of capacity in terms of concurrent users just doesn’t cut it. Tools like Google Analytics show data on Active users right now, but it only indirectly signals the potential strain on site infrastructure. Let’s take an analogy to explain.

Picture a supermarket. Supermarkets are large and have room for many shoppers if they’re spread through the store. Some are in the frozen goods aisle, some in produce, others at the deli. The main bottleneck exists at the checkout counters, where there are only a set number of cashiers. But only a fraction of the shoppers are there. So the cashiers can keep the lines moving and the shoppers happy.

Now, imagine if everyone in the store headed for the checkout counter at the same time. The number of shoppers in the store hasn’t changed. But suddenly the cashiers are overwhelmed and the customers are frustrated.

Queue man in line at supermarket with frustrated cashier

As this flow-based understanding shows, high online traffic puts pressure on bottlenecks in the system. It is these weak links that serve as limiting factors on website capacity. The recipe for a traffic-induced crash is a high number of visitors within a short timeframe mediated by the distribution of visitors along the customer journey. Your site will handle more visitors if they’re spread out than if they’re clustered in front of key bottlenecks.

Anatomy of a traffic-induced crash

Let’s look at a real-life example to make this more accessible (If you are looking for a technical discussion from a developer’s perspective, check out Website failure under load? This is why).

Meghan Markle, now the Duchess of Sussex, has a knack for causing websites to fail. It’s known as the Meghan Effect.

She has a devoted group of followers who pay close attention to everything she wears. Her fashion choices crashed an online retailer’s site not once, but twice, in 2018.

Meghan Markle and Prince Harry on Christmas 2017

Photo by Mark Jones

Let’s look at a customer journey to uncover just some of the behind-the-scenes legwork the site’s database is doing.

 

Step Nr

End-user action

Request(s)

1

Sees Tweet about dress, clicks on link to home page

Loads images and browser script

2

Types name of dress into search bar

Complex search function must find all items that match search term(s), accounting for misspellings, product categories and features, etc.

3

Browses list of products from search and clicks the dress

Loads images and browser script. Shows up-to-date inventory (size, color) information.

4

Adds dress to cart

Inventory system must update to avoid overpurchasing and stay fully in sync for all other users

5

Proceeds to purchase by creating guest account

Sync dynamically-entered information with membership database

6

Enters shipping information

Send and receive requests from shipping plugins, like address verification and shipping costs calculator

7

Enters payment information

Send dynamically-entered information to payment gateway and await payment confirmation

8

Sees order confirmation

Inventory system updates stock remaining

 

 

It’s evident that there are a lot of requests for the website to handle. What’s more, there are several systems behind the scenes processing all these requests.

wooden filing system representing a database

Each step in the customer journey adds strain to site infrastructure. Eventually bottlenecks appear where processing power is fully extended, and additional requests cannot be served.

Website visitors will experience this in the form of a website that slows down and eventually crashes, returning dreaded HTTP error pages.

In sum, when the overwhelming traffic swamps your infrastructure, the weakest link in the system will set the threshold of the entire system. And the dominoes start to fall.

Your CDN only minimizes load on static pages. Your whole website can’t autoscale because key constituent parts like the master database or payment gateways have limited scalability. Dynamic pages like checkout are extremely database-intensive, so shoppers might see your products but are left unable to pay. Inventory systems drown in so many requests you accidentally oversell. Third-party plugins like payment gateways become weak links that start failing first, out of your control.

All this put together brings your entire site crashing down.

Scenarios where high traffic crashes a website

There are three main types of scenarios where high traffic crashes websites and apps.

1. The unexpected traffic spike

Put another way, there was an abnormally high traffic spike relative to your baseline traffic levels. The Meghan Effect, a feature on primetime TV, bot traffic, viral SoMe posts, or overly successful marketing campaigns can all create unexpected traffic surges.

Unxpected website traffic Google Analytics

Figure 1. Peak from a successful email newsletter campaign. (Source)

2. The major sale

Sales like Black Friday and Singles’ Day are known entities in advance. It’s no surprise that they drive traffic levels up to 30 times higher than a normal day. But even with compressed events like these, precisely predicting the timing and magnitude of online traffic spikes is difficult to nail down.

End-users in queue per minute graph

Figure 2. A smaller customer using Queue-it's virtual waiting room experienced several peaks of end-users in queue per minute during Black Friday 2017, including a 260% jump during the middle of the day.

3. The product launch or collection release

Classic examples are sneaker drops, clothing brand collaborations, or limited-edition collectibles. Given the limited quantity of products available, short, intense spikes in traffic are predictable. These are some of the hardest to deal with. What’s more, collection releases usually lead to huge accumulations of visitors to your website before the products drop. This opens the possibility that your site will crash before the sale even begins.

 

Avoiding website failure

While website crashes are an everyday occurrence, they need not be inevitable for your site.

Here are a few ways that can minimize the risk of high online traffic crashing your site.

1. Optimize performance

Earlier we saw how overwhelming traffic cause overwhelming database requests. So part of your strategy should be identifying heavy database requests and then limiting the number, size, and complexity of those requests. There are many ways to build performance into your website. Some ideas include:

  • Use a content distribution network (CDN), which responds to browser requests for your static pages, like your Contact Us or Privacy Policy pages, freeing up your servers to focus on dynamic content, like your checkout pages.
  • Compress images and upload them at their intended display size. Use lazy loading to load media on demand, rather than all at once. According to Google, visitors perceive sites that load this way as faster, even if the actual load times are comparatively slow.
  • Minimize the use of plugins where possible. They can quickly become out of date, and may not always maintain compatibility with your CMS or ecommerce platform.

2. Downgrade the user experience

If you’re expecting a surge in online traffic, you can identify processes on your website that aren’t critical and either temporarily remove them or scale them back.

For example, in lieu of an advanced, CPU-heavy search function, you can use a simpler search function in order to free up the database for business-critical purposes.

3. Run load tests

Failing to prepare is preparing to fail. It might be a cliché, but it doesn’t make it less true.

Load testing is one of the best preparations you can make for web traffic peaks.

Load testing is a kind of performance testing that involves sending increasing levels of traffic to your site under controlled conditions. The aim is to help you understand how it will perform when real visitors start to hit it in big numbers. Using open-source tools, it can be almost free to run.

Because it’s an iterative process, you must allocate adequate time for load testing. Each test will reveal new bottlenecks that need fixing. Adding new code requires additional load testing. So, while it’s important to get started early, it also doesn’t make sense to test before the customer journey is complete. You might find that you need to have your landing pages and key journeys ready much earlier than expected.

Also, be sure to notify your hosting provider that you’re running the load test. Otherwise, it could look like an unwanted DDoS attack. Many providers consider an unauthorized load test a violation of terms of service.

But this preparation is well worth it. Just like a dress rehearsal is critical to a successful stage performance, load testing in advance pays dividends when it’s time for your big online sale.

4. Manage traffic inflow

You can choose to give access to a certain share of users straight away, and redirect excess users to a virtual waiting room or online queue that doesn’t strain your system.

An online queue has the benefit of managing traffic inflow along the whole user journey, keeping visitor levels where your website or app performs best. This takes the heat off bottlenecks in your own infrastructure, as well as third-party integrations like payment gateways that have capacity limitations out of your control.

Paradoxically, throttling traffic in this way can boost sales by increased conversion rates from tapping into social proof.

Learn how to manage traffic inflow to your retail site