Bitware Labs Est. 2022

Bitware Labs/Notebook

Four nginx defaults that quietly cost you


Every one of the four things below was live on bitwarelabs.com until recently. None of them threw an error. None showed up in a log. That is precisely what makes them worth writing down: nginx misconfigurations that fail loudly get fixed on the first afternoon, and the ones that fail quietly stay for years.

1. Every unknown URL returned 200 OK

The catch-all looked reasonable:

location / {
    try_files $uri $uri/ /index.php?$query_string;
}

This is the standard front-controller pattern, copied out of a thousand framework install guides. It is correct for an application that routes its own URLs and answers 404 itself. It is wrong for a site whose index.php is just a page, because the fallback is unconditional: a request for /anything-at-all finds no file, finds no directory, and is handed to index.php, which cheerfully renders the home page and returns 200.

A soft 404 is worse than a broken link. A broken link is a dead end; a soft 404 is an infinite hallway of identical rooms.

Crawlers treat that as an unbounded set of URLs all serving identical content. Worse, it silently swallows real mistakes. Our /sitemap.xml returned 200 for months — with the home page's HTML in the body. Anything fetching it got a well-formed HTTP success containing entirely the wrong document.

The fix has two halves, and the second half is the one people forget:

location / {
    try_files $uri $uri/ @extensionless-php;
}

location @extensionless-php {
    rewrite ^(.*)$ $1.php last;
}

error_page 404 /404.php;

Do not be tempted to write try_files $uri $uri/ $uri.php =404; instead. It looks tidier and it is a source-disclosure bug. try_files processes the file it finds in the current contextlocation /, which serves static files. Your PHP would be sent to the browser as text. rewrite … last restarts location matching, so the request lands in location ~ \.php$ and goes to FastCGI as intended.

And because error_page hands off to PHP, PHP owns the status line. If 404.php does not call http_response_code(404) you have simply relocated the soft 404 rather than fixed it.

2. add_header does not do what you think it does

This is the one that catches careful people. From the nginx documentation, with the emphasis mine:

These directives are inherited from the previous configuration level if and only if there are no add_header directives defined on the current level.

It is not additive. It is all-or-nothing, per level. Define your security headers at server level, then add a single add_header Cache-Control … inside your static-assets location, and every security header vanishes for every image, font and stylesheet on the site. No warning. nginx -t passes.

The always flag does not save you either — that controls whether the header is emitted on error responses, not whether it is inherited.

The workable pattern is a snippet you include at every level that defines any header of its own:

# snippets/headers.conf
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;

# ... and in every location that adds anything:
location ~* \.(png|webp|svg)$ {
    expires 30d;
    add_header Cache-Control "public, max-age=2592000";
    include snippets/headers.conf;   # or these disappear
}

There is a second-order lesson here that applies beyond nginx. If your vhost also reverse-proxies applications you did not write — ours carries several — do not put a Content-Security-Policy or a restrictive Permissions-Policy at server level. You will break someone else's inline scripts or microphone access from three hundred lines away, and the connection between cause and symptom will not be obvious to whoever debugs it. Split the headers: the universally safe ones at server level, the opinionated ones only in the locations serving your own documents.

3. expires max on filenames that change

location ~* \.(css|gif|ico|jpeg|jpg|js|png|svg|woff|woff2)$ {
    expires max;
}

max is ten years. On a hashed build artefact like app.4f3a91.js that is exactly right. On editorial.png it means that anyone who has ever loaded the page will keep the old illustration essentially forever, and you have no way to reach them — not a deploy, not a purge, not a rebuild. The only lever left is renaming the file.

Match the cache lifetime to whether the URL can be versioned, not to the file extension:

  • Fonts — long and immutable. Nobody edits a woff2 in place.
  • CSS and JS — long and immutable if you append a ?v= you actually remember to bump. Otherwise treat them as content.
  • Images and other content — days or weeks. You will want to replace one eventually.

4. The gzip_types line that ships commented out

Debian's stock nginx.conf contains this:

gzip on;

# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_types text/plain text/css application/json application/javascript ...

gzip on is live, so a quick check of any HTML page shows content-encoding: gzip and you conclude compression is working. It is — for text/html, which is the entire default value of gzip_types. Every stylesheet, script, JSON response, RSS feed and SVG goes out raw.

If your CSS is inlined in the HTML you will never notice, because the HTML is the thing being compressed. The day you move that CSS into a file — a good change, for caching — your compression silently stops applying to it. The regression is invisible and its cause is four hundred lines away in a file you did not touch.

Uncomment the block. Add image/svg+xml, application/rss+xml and application/manifest+json, which the stock list omits. And set gzip_vary on so intermediate caches keep compressed and uncompressed variants apart.


The pattern

None of these are obscure. All four are in the documentation, and three of the four are in a config file shipped by the distribution. What they share is that the failure mode is a success response: a 200 with the wrong body, a header that is absent rather than wrong, a stale asset served perfectly, a response that is merely larger than it needed to be.

Monitoring will not find these, because nothing is down. Tests will not find them, because the application is behaving. The only thing that finds them is going and looking — curl -I against production, reading the response rather than the config, and asking what the server is actually saying rather than what you intended it to say.

Correspondence

Write to the lab

Commissions, collaborations, or a quiet hello. The person you write to is the person who does the work — I reply within two working days.