When the load test is "we'll see in production"

There’s a very particular breed of optimist in tech: the person who looks at a chart of concurrent users and tells you, cool as a cucumber, “meh, if it goes badly, we’ll scale”. And by “we’ll scale” they mean restarting the pod by hand, hand shaking on the mouse, while Slack is on fire and the head of sales keeps asking how much money is being lost per minute.

The problem is that when days like Black Friday, Prime Day or the launch of “the new campaign that’s going to revolutionize the industry” come around, no motivational spreadsheet can make up for not having done serious performance testing. Whatever you didn’t test beforehand, production is going to teach you, with interest.

That’s what we’re going to talk about: not defining SLOs, not using real data, not simulating spikes, and believing the ORM is a wizard that turns horrible queries into something optimal. I’ll mix it up with real examples of high-profile outages, and with the guts of it: N+1 queries, locks, a GC gone wild and the rest of the menagerie.

Black Friday: when “we’ll see” makes the front page

Black Friday has become a worldwide stress test… except it runs in production. Year after year it’s the same story: lots of sites go down, and almost never because of functional bugs. They go down because nobody bothered to check how the thing behaves under real traffic.

The people who analyze these outages put it bluntly: nearly all bottlenecks are in the software, not the hardware, so you don’t fix them by throwing more servers at a system that wasn’t built to scale that way. It’s not “some weird bug nobody could have seen coming”.

Over the 2018 Black Friday weekend, for example, the websites of Walmart, J.Crew and Lululemon went down or slowed to a crawl, and there were complaints about Lowe’s and Ulta too.

In J.Crew’s case, according to estimates from LovetheSales.com, the outage cost about $775,000 in sales in the first five hours alone. Walmart’s problems, which started the night before Thanksgiving, hit around 3.6 million shoppers, with roughly $9 million in lost sales. That’s setting money on fire in real time.

Not even the giants are safe. RedLine13 put Macy’s on its list of the biggest load testing blunders of 2016: the website and app of one of the largest department store chains in the world crashed on the most important day of the year, with the campaigns ready and the ads already running. Nobody knows exactly how much money was lost, but we can assume it was enough for more than one person to have a rather… intense conversation.

And the part that really stings is that there was no weird bug. What there was: unrealistic load tests, infrastructure sized by eyeballing it, and code and dependencies that were barely scraping by the rest of the year and sank as soon as the spike hit. If you don’t design your load test, reality will design one for you, far more creative… and far more expensive.

The same movie is starting to play in the Spanish-speaking world: consultancies and performance testing vendors tell real stories of online stores that collapse on campaign day because “they had never been tested with high concurrent traffic”, trusting that the hosting “should hold up”.

In one of these cases, nobody had tested an online store with more than 100 concurrent users before a Black Friday-style event; on the big day, the site stopped responding right at peak demand. You don’t need to look at the error log to know the problem wasn’t some stray NullPointerException. If your only metric is “it works fine with a hundred users”, what you have is wishful thinking.

And yet the pattern keeps repeating: no clear performance targets get defined, real user traffic isn’t simulated, spikes aren’t tested, and everyone trusts that “the cloud scales on its own”.

Spoiler: the cloud does not scale on its own if your code does absurd things.

The original sin: no SLOs and no awkward questions

The root of the problem usually lies long before Black Friday. It starts on the day nobody dares to ask the ugly questions at the beginning of the project. Things like:

• What’s the maximum response time we accept at checkout under normal load?
• And under peak load?
• How many concurrent users do we expect? What’s the extreme scenario?
• What’s an acceptable error rate?

Serious performance testing practices, like the ones described by IBM or AWS, start exactly there: identifying the application’s goals, the metrics that matter, target loads and acceptable thresholds. But a lot of teams would rather talk about new features than about these “boring” details.

The answers to those questions are your SLOs (Service Level Objectives). Without SLOs, anything “works”.

If nobody has said that checkout has to respond in under 500 ms for 95% of requests with 1,000 concurrent users, then a checkout that takes 4 seconds with 300 users is “well, it’s a bit tight, but fine”. Until the traffic spike arrives, ten times as many users show up, and suddenly everything takes more than 10 seconds and you start seeing timeouts, retries and cascading failures.

The irony is that the load testing guides themselves insist on defining response time metrics, resource utilization, maximum capacity and breaking points before you start firing off scripts. But of course, that means sitting down to think, and thinking is less flashy than showing off a Kanban board covered in sticky notes.

A simple rule: if you can’t write your performance targets down on a piece of paper, what you’re doing is praying.

Fake data, fake results

Another classic: testing with toy data. That environment with 300 fake users, 20 orders and three products in the catalog, used to sign off on the performance of a system that will have millions of records in production.

The problem is well documented: many performance pathologies (like N+1 queries) don’t show up until the data volume grows. When you have 10 rows, you can get away with almost any nonsense. When you have 10 million, every dumb query is a ticking time bomb.

Technical articles on N+1 queries make the point: with little data, the degradation is barely noticeable, but as the number of related objects grows, the cost skyrockets and performance tanks. And, of course, you’re not going to see it if your staging environment is empty or filled with irrelevant data that looks nothing like real production patterns.

Serious load testing guides recommend building realistic scenarios based on both user data (browsers, devices, journeys) and system data (TTFB, CPU usage, I/O, etc.), using representative or anonymized production data. But all too often someone fills the database with a hacked-together script that repeats the same name over and over, and the simulation gets a pass.

If your real catalog has thousands of odd combinations, long histories, abandoned carts and dirty data, and your test environment is a “Hello, World” with four squeaky-clean records, your load test is going to lie right to your face.

N+1 queries: the ORM is not on your side

Let’s talk about the classic: N+1 queries. That charming pattern where you run one query to get a list of entities and then, inside a loop, touch a lazy relationship… which fires off a new query for every single item.

The literature on ORMs repeats it ad nauseam: the N+1 problem is a common inefficiency that generates 1 main query plus N additional queries to load the relationships, and it hammers performance.

In articles by people who work with Doctrine, Entity Framework and the like, you see the same pattern: by putting code simplicity first (“the ORM will take care of it”), you end up unleashing a shower of queries that goes unnoticed with small data but, at large volumes, turns the system into an I/O meat grinder.

The problem gets worse when you combine N+1 with peak days. During Black Friday, for instance, the number of concurrent requests goes way up. If every request drags N+1 queries along with it, database resources run out, latency climbs, timeouts kick in and, as a bonus, the connection pool is left gasping for air.

Some consultancies, like Deloitte, have published articles on how to optimize ORMs that flag N+1 as a critical performance problem and explain how uncontrolled lazy loading ends up generating a huge number of connections and queries in scenarios with lots of relationships. Yes, the same Deloitte that had to refund part of its fee to the Australian government over a report with citations made up by AI. Relax: I checked the N+1 part myself.

Other authors, on technical blogs, walk through real cases where this pattern led to near-explosive growth in queries and response times, until it was replaced with well-designed JOIN queries.

Now picture this during a traffic spike: every request multiplies the cost on the database, you saturate it and set off a domino effect that “adding one more node” won’t fix. That’s badly written SQL, and infrastructure won’t fix it.

So how do you fix it? First, catch it in your load tests: measure how many queries each request fires, because that number gives away an N+1 long before latency does. Then use well-thought-out JOINs, fetch only the data you need, or load relationships eagerly, but with some judgment. And above all, rewrite the queries to bring back what you need, not the entire universe. Fixing it after the first load test is way cheaper than finding out with people yelling at you on social media.

Locks everywhere: your transactions wait in line too

Another wonderful source of surprises in production: locks. If you’ve worked with relational databases, you know transactions and locks are a kind of musical chairs: while there are only a few people, nobody fights; when Black Friday arrives and half the country walks in, fists start flying.

Load testing and resilience guides (for example, the AWS Well-Architected ones) recommend validating how your data architecture scales, finding database bottlenecks, and watching metrics like lock times, waits and contention before going to production. But that part usually gets skipped because “it’s fast on my machine”.

The usual anti-pattern is an inventory table with a single row per product, updates wrapped in long transactions with a cheerful SELECT ... FOR UPDATE, and a spike of requests all trying to decrement stock at the same time.

If you haven’t simulated that concurrency, what you’ll find out on campaign day is that requests served per second plummet, transactions start waiting, the API runs out of free threads, and the whole site looks frozen. Even though all it’s doing is waiting. All of this could have been caught by load tests that reproduced real usage patterns and measured lock times and latencies under high concurrency.

Of course, that means knowing what your ORM is really doing and how it translates into SQL. Much more comfortable to assume that thousands of people who want the same deal will never hit the pay button at the same time.

What should have been done comes in three steps. First, load tests that update inventory the way real users do, everyone going for the same deal. Second, measure how long locks last and how long responses take when lots of people show up at once. And third, change the data model, with reservations, queues or events, so that not everyone is fighting over the same row. You can’t have half the country locking the same row and expect things to run smoothly.

GC out of control: when the garbage collector goes on strike

In garbage-collected languages, another source of “we’ll see in production” scares is memory management.

A common scenario: a Java application or similar, with large objects, uncontrolled in-memory caches and unnecessary serialization. Under light load, the JVM handles it fine and the GC barely gets in the way. Under heavy load, memory fills up, pressure builds and the collector hits the “everybody freeze, I’m cleaning up” pause.

What performance testing theory says is clear: you need to monitor memory usage, GC behavior and its pauses, and do it with increasing load, not just toy traffic. Those same sources recommend stress tests that push the system to its limits to see how it behaves, where queues form and which subsystems go down first.

But in a lot of places, performance testing means “we run JMeter/Gatling for 5 minutes, looks fine, next story”. That’s less a test than a massage. Meanwhile, the reality in production is that the GC goes into longer and longer cycles (memory leaks, badly sized caches, structures that never get freed) and, right at peak traffic, GC pauses get so brutal that requests start timing out.

The fun part is that the problem doesn’t show up with 100 users, but it does with 5,000. And if you haven’t tested at several times your expected traffic, you’ll find out when you try to buy from your own store to see if checkout works and the page won’t even load.

Unsimulated spikes: Black Friday isn’t “normal traffic, just a bit higher”

Another mental trap: believing Black Friday is “the usual, times two”.

Performance platforms and e-commerce specialists have been explaining for years that the traffic pattern changes completely on those days: intense bursts in specific windows, campaigns that send thousands of users in at once, a different way of browsing. Companies that analyze traffic for events like Prime Day show how traffic and sales curves have very sharp peaks, way above the average of a normal day.

You can’t simulate that with a test that ramps users up slowly to a “reasonable” level. Black Friday isn’t a ramp, it’s a wall.

Performance testing handbooks talk about load tests with stepped ramps, but also about stress tests and scenarios that push the system beyond normal levels to find its breaking point and see how it behaves when saturated. And for events like Black Friday, they recommend starting as early as possible, simulating realistic spikes and repeating the tests every time a bottleneck gets fixed, because getting ready for that day is an iterative process.

And yet many organizations stop at “we tested with twice the users and it looks stable”. Then the big day comes, they get 5 to 10 times their normal traffic in a short window… and guess what happens.

And it’s not only about volume. The mix of devices, paths and use cases also changes on those days, and if your tests don’t reproduce it, saturation will show up in the service nobody was watching.

I can’t cast the first stone either

Before it sounds like I’m looking down on all this from on high, let me confess something. They say let he who is without sin cast the first stone. Well, I can’t cast that stone, because I’ve done my share of load testing in production myself. Unintentionally, granted, which is how almost all of them happen.

It was on a project where I was the architect. I won’t say which one, but there’s a good chance you know it. We were expecting spikes of hundreds of thousands of users within a ten-second window. We ran load tests, lots of them and very well done, or so we thought. They were the optimistic kind, the ones where users ramp up slowly along a lovely ramp. Exactly what I just told you not to do.

We found bugs, sure. We fixed them, tuned the infrastructure and reached launch day feeling pretty relaxed. And for half a minute, things got completely out of hand.

What we hadn’t accounted for was that our provider’s infrastructure, because of how it was designed, scaled fast, but not as fast as we needed. And to be fair, it scaled beautifully: that day it handled two million requests in one minute. But our wall arrived in ten seconds, and scaling needed a bit longer.

The most painful part, looking back, is that the problem had actually shown up in testing. Right at the start of every test we saw a handful of errors, and then everything went green. Why? Because after scaling out, the infrastructure keeps that extra capacity for a while before shrinking back: that’s the cooldown period.

When we launched the next round, the machines were still there, nice and warm, waiting for us, and we were happy as clams, convinced the problem was solved. We were testing the wall with the infrastructure already warmed up. And on launch day, it showed up cold.

Half a minute doesn’t sound like much. Now picture that half minute with hundreds of thousands of people trying to get in at once, and the whole team in front of the dashboards tearing their hair out, me first.

I learned that one the hard way: you have to test the spike cold. Let the infrastructure go back to its normal size between test runs, and never trust a second round that comes out fine. Your users aren’t going to wait for your system to warm up.

Real cases where the lack of performance testing took its toll

Sometimes, the lack of performance testing shows up directly as the main cause of the disaster.

Take Ello, for example. If the name doesn’t ring a bell, don’t worry, it didn’t for me either until I started researching this article. It was a social network that, in 2014, pitched itself as the anti-Facebook: no ads and no selling anyone’s data. You could only get in by invitation, and at its peak it was getting more than thirty thousand invite requests per hour. It was all the rage for a while. A short while, let’s face it.

It has been documented that its servers couldn’t handle the load because testing hadn’t kept pace with the growth they were expecting. The result: frequent outages, poor performance and a massive exodus of users, largely blamed on not having found the bottlenecks before launch through proper load testing. If your product is a hit and your infrastructure isn’t, users will leave for whichever product works better.

The moral is simple: what was expensive wasn’t running the tests, it was not running them.

What actually works: treating performance as a requirement, not decoration

If you go to the more serious literature (IBM, AWS, performance testing guides, specialized performance blogs), there’s a fairly clear consensus about what you need to do if you don’t want to play Russian roulette in production.

Performance is one more requirement, just as mandatory as the functional ones, with defined targets, and it’s validated continuously. That means, for example:

• Defining metrics and SLOs: response times, throughput, error rates, resource utilization, queues, etc., distinguishing between normal load and expected spikes.
• Building realistic scenarios: with representative data, real browsing patterns and devices/platforms similar to production.
• Running load, stress and resilience tests: starting below usual load levels, ramping up gradually, then going beyond what you expect to find failure points and behavior under saturation.
• Monitoring during the tests: system metrics (CPU, memory, disk, GC), application metrics (latency per endpoint, queues, locks) and user experience metrics (time to first byte, perceived load times).
• Reviewing and optimizing critical code: especially data access through ORMs, eliminating N+1 queries, tuning lazy/eager loading policies and designing queries that fit real use cases.

In practical terms, for an online store a reasonable strategy would start by setting a target for checkout, something like “95% of requests in under half a second, up to so many users”. Then you’d fill staging with real but anonymized data, and write realistic user journeys that browse, search, filter, add to cart and pay. You’d run load tests up to normal traffic plus a margin, and then stress tests at five to ten times the expected traffic for the busiest moments, starting each one with the infrastructure cold.

Meanwhile, you’d keep an eye on CPU, memory, the garbage collector, how many queries each request fires, locks and queues, and with all that you’d fix whatever comes up: N+1 queries, indexes, caches, data models or connection settings. It’s work, sure. But it’s a whole lot less work than explaining yourself after a high-profile failure.

For events like Black Friday, that means testing at serious multiples of expected traffic. Bob Buffone, CTO of Yottaa, summed it up in 2018: a site that hasn’t been load tested at five times its normal traffic will probably fail. I’d aim for five to ten. And you need to repeat the tests regularly instead of relying on “we ran a test last year”. Architectures change, the catalog changes, campaigns change; assuming the results of a test from twelve months ago still hold is optimistic, to say the least.

So, what are you going to do with all this?

By now, you’ve probably already lived through a mini Black Friday on your own system, or you’re about to.

My advice, based on what we know and what’s already been published on the subject, is to stop seeing performance testing as “something that would be nice to do someday if there’s time left over” and make it part of the bare minimum definition of “production-ready”. Just as nobody in their right mind would deploy without halfway serious functional tests, shipping a major launch without reasonable load tests means admitting your strategy is “fingers crossed”.

Everything we’ve covered (N+1 queries, badly managed locks, an out-of-control GC, unsimulated spikes) is a known, documented problem with reasonable solutions, and catching them in time takes just one thing: someone with the bad manners to ask “and how does this behave with ten times the traffic?”.

Ball’s in your court: in the system you work on, are there clear performance SLOs and automated load tests worthy of the name, or does everything still live in the “we’ll see in production” department?

Quick glossary

So nobody has to look up what that N+1 thing was in the middle of Black Friday.

• N+1 queries: a pattern where one query fetches a list and then another query runs for each item to load its relationships; with N items, N+1 queries.
• ORM (Object-Relational Mapping): a library that maps the objects in your code to database tables and writes the SQL for you, for better or for worse.
• Lazy loading: an ORM strategy that doesn’t load a relationship until the code touches it; the usual culprit behind N+1.
• SLO (Service Level Objective): a measurable service quality target, for example “95% of payments in under half a second”.
• Lock: a reservation a transaction holds on a row or table so nobody else can modify it at the same time; under heavy concurrency, queues form.
• SELECT ... FOR UPDATE: a query that reads rows and locks them until the transaction ends.
• GC (garbage collector): the part of the virtual machine (the JVM, in Java) that frees memory that’s no longer in use; some of its phases pause the application.
• Connection pool: a limited set of database connections that the application reuses; when it runs dry, requests wait.
• TTFB (Time To First Byte): how long it takes for the first byte of the response to arrive.
• Load, stress and spike tests: a load test measures behavior under expected traffic; a stress test pushes the system past its limit; and a spike test throws a sudden surge of users at it in a very short time.
• Cooldown: the time autoscaling waits before scaling capacity back down after a spike.

Sources and references

If you’re looking for something to blame when your Black Friday comes around, here’s where to start.

• Load and performance testing to ensure system stability — SQS (in Spanish). A practical guide to structuring load tests and defining stability metrics.
• Don’t fail on Black Friday: why performance testing and monitoring is more critical than ever — Dotcom-Monitor. Why so many systems fail on high-demand days and how proactive monitoring could save them.
• Web stress test: how to prepare for the load — SiEsoluciones IT (in Spanish). An introduction to stress testing and which scenarios to cover before a launch.
• The N+1 query problem — Álvaro Rosado (in Spanish). A clear explanation of the N+1 pattern in ORMs and its impact on performance under load.
• How to optimize ORM performance — Deloitte (in Spanish). N+1, runaway lazy loading and other ORM pathologies from a consulting perspective.
• The N+1 query problem, explained — Los puntos sobre las jotas (in Spanish). A technical walkthrough with concrete examples of how the number of queries piles up, one extra per row.
• Why do so many websites fail on Black Friday? — Capacitas. An analysis of real incidents and the most common failure patterns on peak days.
• Black Friday: why load testing is a must — Gatling / dev.to. The case for load testing before any major campaign, with use cases.
• 5 critical mistakes of skipping performance testing — IzyAcademy (in Spanish). The most expensive mistakes made when someone skips performance testing.
• AWS Well-Architected: resiliency testing and non-functional requirements — Amazon Web Services. A reference framework for validating the resilience and behavior under load of cloud architectures.
• AWS Prescriptive Guidance: load testing foundations — Amazon Web Services. Prescriptive guidance on methodology, test types and success criteria.
• What is performance testing? — IBM. Definition, types and best practices for building performance testing into the development cycle.
• What is load testing? — OpenText. The different kinds of tests (load, stress, spike, soak) and when to use each.
• Load testing: capacity and response time under heavy traffic — FasterCapital (in Spanish). A detailed guide to building realistic load scenarios and measuring results properly.
• Analysis of infrastructure incidents under heavy traffic — YouTube. A video breakdown of real infrastructure saturation cases and which tests would have caught them earlier.
• Amazon Prime Day 2021: traffic and sales trends — Criteo (in Spanish). Traffic and shopping behavior data during Prime Day, with the peaks and patterns that set it apart from a normal day.
• Prime Day sales shrink in the first hours — Investing.com (in Spanish). The impact on sales and stock price when systems can’t keep up with Prime Day demand.
• Can your website handle the traffic volume on Black Friday? — InMotion Hosting. A technical checklist to find out whether a system is ready for a high-traffic event.
• A review of Black Friday load testing — LoadView. A specific methodology for preparing and running load tests aimed at events like Black Friday.
• What is API load testing and why do startups need it? — Ranktracker. Load testing in fast-growth environments, with Ello as an example of a collapse caused by lack of testing.
• How to keep your website from crashing on Black Friday — Kinsta. A preventive guide with real Black Friday outages and the technical measures to avoid them.
• Software performance testing — Wikipedia (in Spanish). A general reference on the types, metrics and methodologies of performance testing.
• Load testing blunders of 2016 — RedLine13. A roundup of real load testing mistakes made by teams in 2016 and what caused them, including Macy’s Black Friday crash.
• Walmart, J.Crew and Lululemon websites down on Black Friday — Fortune. A report on the 2018 Black Friday outages and each company’s explanation.
• J.Crew website tech problems continue — Business Insider. LovetheSales.com’s estimates of the sales J.Crew and Walmart lost.
• Black Friday sales kick off early and websites crash — Business Insider. The first outages of 2018 and Bob Buffone’s (Yottaa) warning about testing at five times normal traffic.
• Say Ello to the anti-Facebook — Engadget. What Ello was, its ad-free pitch and its explosive invite-only growth in 2014.
• Deloitte forced to refund Aussie government — TechRadar. Deloitte’s partial refund to the Australian government over an error-strewn report produced with AI.