Skip to main content
Back to Insights

When something goes wrong: how we respond to and recover from an incident

Valentin Zsigmond
Valentin ZsigmondOct 16, 20264 min read

Most of what we write about is prevention: updates, firewalls, permissions, monitoring, the quiet routines that keep a site out of trouble. This article is about the other side, the plan for the day something goes wrong anyway. Every site we look after has one, written down, and we think every business should know what theirs says. Not because incidents are likely on a well-maintained site, but because knowing exactly what would happen is what lets everyone stop worrying about it.

What counts as an incident

An incident is any event that affects the availability, integrity or confidentiality of your site or its data. That covers more than a break-in. A deployment that takes the site down, a certificate that expires, a disk that fills up, a third-party service that fails, a compromised editor account, a defaced page, a leak of form submissions: all of these are incidents, and the plan handles all of them with the same structure, because the structure is what keeps a response calm.

Detection: knowing within minutes

The plan starts before anyone is involved. Monitoring watches availability from several locations, response times, error rates, certificate expiry, failed logins, unexpected changes to files and configuration, and the health of each service the site depends on. When a signal crosses its threshold, an alert reaches a person, day or night, and the clock starts. Most incidents on our platform are detected this way, before a visitor or a client notices, and a good number of them are resolved before anyone outside the team knows they happened.

Triage: the first fifteen minutes

The person on call has three questions to answer quickly. What is affected: one page, one site, one server, one service? Is it still happening, or has it stopped? And does it involve data, which changes the legal obligations and who has to be told? The answers set the severity, and the severity decides who else is pulled in and how fast the client hears from us.

For anything that affects visitors or data, the client is notified within the hour with what we know, what we are doing, and when the next update will come. That message goes out even when the picture is incomplete, because silence is the thing that damages trust, not incomplete information.

Containment: stop it getting worse

Before fixing anything, we stop the incident from spreading. The exact step depends on the type. A compromised account is disabled and its sessions ended. A suspicious IP range is blocked at the edge. A misbehaving integration is switched off. A site that is being actively attacked can be put behind a maintenance page or a stricter firewall rule set in a minute. Because every site runs in its own isolated container with read-only code, an incident in one place stays in that place; the architecture does most of the containment for us.

Recovery: back to a known good state

Recovery is where preparation pays off most visibly. Every release we deploy is kept, so if a deployment caused the problem, rolling back is a single step and takes seconds. If content or data was damaged, we restore from a backup that we have already proven restores, in a time we have already measured. If code was tampered with, the read-only runtime means the tampering cannot have reached the deployed release; we redeploy from the repository and the site is clean.

The goal in recovery is not to understand everything yet. It is to have visitors on a working, trustworthy site as quickly as possible. Understanding comes next.

Investigation: what actually happened

With the site stable, we reconstruct the sequence. Logs are kept for exactly this purpose, stored where the site itself cannot alter them, and they let us trace an incident from first signal to root cause. We identify what allowed it to happen, whether the same condition exists elsewhere, and what change closes it permanently. If personal data was involved, this is also where we establish what data, whose, and whether the thresholds for notifying authorities and affected people were reached, so the client can meet their obligations within the legal windows.

Communication throughout

Clients hear from us at each stage: when we detect, when we contain, when we recover, and when we understand. Updates go through the client portal as well as by direct message, so there is a record. If visitors or customers need to be told, we help write that message, plainly, with what happened, what it means for them and what they should do. Written well, an honest notice strengthens trust rather than spending it.

Afterward: the post-incident review

Within a few days, we write a short review: timeline, cause, impact, what worked, what did not, and the changes we are making. It is shared with the client and, where the lesson is general, applied to every site we run. The review is written without blame. Its purpose is that the same thing does not happen twice, anywhere, and that the next incident, whatever it is, is handled a little better than this one.

Why it matters that this is written down

Every step above is ordinary once it is planned. What makes incidents stressful is improvisation: deciding who to call, where the backups are, how to roll back, what to tell people, all under pressure. A written plan, rehearsed through restore drills and rollback tests, turns a potential crisis into a procedure. It is one of the quieter parts of managed hosting, and one of the most valuable.

If you would like to know what would happen to your site on a bad day, ask us. We will walk you through the plan, and if you do not yet have one, help you write it.

Share this article
Valentin Zsigmond
Written by
Valentin Zsigmond
Founder & Lead Engineer

Full-stack Drupal architect with almost two decades of experience leading large teams in enterprise environments. Founded Tilizy Digital to bring senior-level expertise directly to the organizations that need it. Mentor to dozens of Drupal developers and creator of SDX — a Drupal extension for building modern frontends with React and Vue.

Enjoyed this article?

If this resonated, imagine what we could do working together on your Drupal site.