I was not looking for an interesting systems failure. I was expecting a boring one.
On August 16, 2026, I received an UptimeRobot notification saying that one of my WordPress sites had gone down. This was running on a very small Scaleway VPS, and my first thought was almost automatic: the disk was probably full again. I had seen that kind of problem before. UpdraftPlus could leave backup archives on a small root filesystem, free space would disappear, WordPress would stop behaving properly, and the repair was usually uncomplicated: SSH into the server, identify old backups, delete what was no longer needed, and move on.
That assumption was reasonable because it came from previous operational experience. It was also wrong this time.
The first sign that this incident was different was not actually the website. It was SSH. I could not log into the VPS at all. I tried again from another SSH client. I tried a hard reboot from the Scaleway control panel. The machine appeared to boot, but SSH still would not let me in. The familiar five-minute cleanup had suddenly become a situation in which both the application plane and my normal management plane were unavailable. A small VPS is quite capable of reminding you that 10 GB is a perfectly respectable amount of storage right up until it decides that it is zero.
The UptimeRobot incident gave me a useful external timestamp. The monitor recorded the outage beginning at:
2026-08-16 16:43:31 UTC HTTP 522 - CloudFlare Timeout
The site sat behind Cloudflare’s reverse proxy, so a 522 was already telling me something more specific than “WordPress returned an error page.” Cloudflare was no longer getting a timely response from the origin. At this point that still left many possibilities: Nginx, PHP, MariaDB, resource exhaustion, networking, firewall state, or the entire host. The monitor could tell me where the failure became externally visible; it could not yet tell me where the failure began.
The VPS was small, IPv6-only, and slightly unusual
The architecture matters because this was not a conventional dual-stack VPS. The original system was Ubuntu 22.04.4 LTS running Linux 5.15.0-187-generic on KVM, with one virtual CPU, roughly 1 GB of RAM, a 1 GB swap file, and a 10 GB virtual disk. The usable root filesystem was about 8.9 GB. The web stack included Nginx, PHP 8.1 FPM, MariaDB 10.6.23 and WordPress.
The machine had native public IPv6 but no normal native IPv4 route. Incoming web traffic was proxied through Cloudflare, so the public-facing site did not require the origin itself to expose a public IPv4 address. That is an important distinction that I had not been thinking about when I first built the server. Cloudflare acting as a reverse proxy in front of a website and Cloudflare WARP running as a client inside that website’s server are two different pieces of architecture.
Server-side WARP had been installed months earlier. I no longer remembered every decision that led to the exact configuration, which became another reason not to reconstruct the incident from memory alone. The surviving shell history was more trustworthy. Among the commands were:
sudo apt-get install cloudflare-warp warp-cli settings sudo warp-cli tunnel ip add-range ::/0 warp-cli settings warp-cli registration new warp-cli connect
The installed Linux client was Cloudflare WARP 2026.3.846.0. The particularly interesting command was:
sudo warp-cli tunnel ip add-range ::/0
The surviving WARP configuration later showed this as an exclusion of ::/0. In practical terms, the intended architecture was approximately:
Inbound HTTP/HTTPS
|
v
Cloudflare reverse proxy
|
v
native IPv6 origin
Outbound IPv6
|
+---- native VPS IPv6
Outbound IPv4
|
+---- Cloudflare WARP
That design makes sense for an IPv6-only machine that occasionally needs to initiate a connection to an IPv4-only service. The goal was not to replace the server’s working IPv6 network with a VPN for its own sake. The surviving configuration strongly suggests that WARP was being used as an IPv4-egress workaround while native IPv6 remained outside the tunnel.
I am emphasizing “surviving configuration” here because it is easy in a postmortem to rewrite one’s intentions to match the evidence discovered later. I did not have a perfect memory of why every WARP command had been issued. The shell history and runtime configuration established what the system had actually been told to do; my recollection merely supplied the broader context.
The machine booted, but SSH did not become SSH
After the outage I hard-rebooted the instance from Scaleway and watched the serial console. If the kernel had panicked, the root filesystem had been corrupted, or the VM simply could not boot, the problem would at least have been conceptually straightforward. Instead, the boot looked surprisingly normal.
The filesystem check reported the root filesystem as clean. Swap activated. IPv6 networking appeared. Nginx started. PHP-FPM started. MariaDB started. The Cloudflare client started. Cloud-init completed. The machine reached its login prompt. There was no visible kernel panic, no obvious out-of-memory crash and no catastrophic ext4 failure.
There was, however, one line in the filesystem check that was much less reassuring:
cloudimg-rootfs: clean, 148660/1193472 files, 2397738/2412986 blocks
Almost all filesystem blocks were occupied. On a subsequent boot the count became:
2399256/2412986 blocks
That put the root filesystem at roughly 99.4% block usage. The situation was already severe enough that disk exhaustion immediately became the leading operational problem.
Still, it did not completely explain what I was seeing. A later boot explicitly showed that OpenSSH itself had started:
[ OK ] Finished SCW fetch ssh keys from metadata.
Starting OpenBSD Secure Shell server...
[ OK ] Started OpenBSD Secure Shell server.
So the earlier absence of an SSH startup line in one console excerpt had been misleading. sshd had not simply failed to start. The authentication logs subsequently showed it listening on port 22 over IPv6 as well.
The client-side transcript narrowed the problem further. I normally reached this IPv6-only server through a local SOCKS path, so I ran OpenSSH with verbose diagnostics:
ssh -vvv \ -o 'ProxyCommand=nc -x 127.0.0.1:7890 -X 5 %h %p' \ root@2001:db8::10
The important part was not the hundreds of ordinary diagnostic lines. It was where the exchange stopped:
debug1: Executing proxy command: exec nc -x 127.0.0.1:7890 -X 5 2001:db8::10 22 ... debug1: Local version string SSH-2.0-OpenSSH_8.6 kex_exchange_identification: Connection closed by remote host Connection closed by UNKNOWN port 65535
This established several useful facts. My client could reach the path to TCP port 22. OpenSSH started the protocol exchange and sent its own identification string. What never arrived was the server’s corresponding SSH-2.0-... banner. Public-key authentication had not failed, because authentication had not even begun.
The UNKNOWN port 65535 line looked dramatic but was not the server mysteriously deciding to move SSH to port 65535 overnight. It was an artifact of the ProxyCommand connection path. The diagnostically important line was kex_exchange_identification: Connection closed by remote host.
I also tried another SSH application and obtained the same practical result. This made an ordinary client configuration problem increasingly implausible. More importantly, the server’s own SSH logs did not contain a corresponding authentication failure from these attempts. The service was starting and listening, while connections were disappearing before the normal SSH exchange became visible in the logs.
That still did not prove what was killing them. Extreme disk pressure could produce strange secondary failures. WARP was also present and capable of manipulating networking and firewall state. There were other possibilities. At this stage I could describe the boundary of the failure much more accurately than its cause:
VM boot works root filesystem mounts works IPv6 interface comes up works major application services start sshd service starts TCP path to port 22 reachable SSH client sends its banner yes server SSH banner never arrives SSH key authentication never reached
Repeating hard reboots was therefore becoming less useful. In fact, the filesystem block count had moved in the wrong direction between boots. A reboot was capable of restarting the same persistent state; it was not removing that state. Eventually I stopped treating “reboot it again” as a diagnostic method.
Rescue mode changed the kind of problem I was solving
At this point Scaleway’s rescue mode became essential. Instead of booting the installed Ubuntu environment, rescue mode starts an independent temporary operating system while leaving the original disk attached. That distinction is enormously valuable when the installed system’s own networking, DNS, firewall or storage state may be part of the failure.
It also created an out-of-band administrative path. The production OS could be completely confused while the block device containing it remained perfectly readable from another system. Without something equivalent to rescue mode, I would have been very close to an administrative deadlock: the server needed repair, but the server itself was the only normal route through which I could perform the repair.
The rescue environment happened to be Ubuntu 24.04.2 LTS with a 6.8-series kernel. That was not an upgrade of the VPS. It was a separate temporary system, and distinguishing the rescue filesystem from the original one was the first task.
I started with block-device identification rather than mounting whatever looked plausible:
lsblk -f findmnt /
The important part of the output was:
NAME FSTYPE LABEL sda ├─sda1 ext4 RESCUE-ROOTFS ... vda ├─vda1 ext4 cloudimg-rootfs ├─vda14 └─vda15 vfat UEFI / overlayroot
sda belonged to the rescue system. vda1 was the original VPS root filesystem. I mounted that original partition read-only first:
mkdir -p /mnt/original mount -o ro /dev/vda1 /mnt/original df -h /mnt/original df -i /mnt/original
The result immediately confirmed that the full-disk suspicion was real:
Filesystem Size Used Avail Use% Mounted on /dev/vda1 8.9G 8.8G 46M 100% /mnt/original Filesystem Inodes IUsed IFree IUse% /dev/vda1 1193472 148662 1044810 13%
This was block exhaustion, not inode exhaustion. Only 13% of the inodes were occupied. The filesystem was simply out of useful storage space.
That result might seem to have vindicated my original diagnosis. The site was down, I had suspected a full disk, and the disk was indeed at 100%. Case closed?
Not quite. “The disk is full” describes a state. It does not identify the process that created that state, and it certainly does not explain every symptom that occurs after the state has been reached.
The disk was full for a very different reason
Because the filesystem was still mounted read-only, I could investigate without allowing the installed system to generate more logs or update more state. I worked down the directory tree:
du -xhd1 /mnt/original 2>/dev/null | sort -h du -xhd1 /mnt/original/var 2>/dev/null | sort -h du -xhd1 /mnt/original/var/log 2>/dev/null | sort -h find /mnt/original -xdev -type f -size +50M \ -printf '%s %p\n' 2>/dev/null | sort -n | tail -40
The top-level numbers immediately changed the story:
/usr 2.2G /var 5.5G total 8.8G /var/www 578M /var/lib 1.3G /var/log 3.5G
My entire web tree was only about 578 MB. System logs occupied 3.5 GB.
Looking further into /var/log gave:
/var/log/cloudflare-warp 62M /var/log/journal 801M /var/log/nginx 556K /var/log 3.5G
Those directory totals still left a couple of gigabytes unexplained, which meant the next thing to inspect was the ordinary files directly inside /var/log. There they were:
/var/log/syslog approximately 2.38 GB /var/log/syslog.1 approximately 390 MB older compressed syslogs tens of MB persistent journal approximately 801 MB
This was no longer a story about a WordPress backup directory quietly occupying most of the disk.
I checked the actual UpdraftPlus storage as well. At forensic inspection time, the wp-content/updraft directory contained only about 104 KB, while the UpdraftPlus plugin itself occupied roughly 31 MB. The entire /var/www tree remained around 578 MB. Against 3.5 GB of system logs, the imbalance was unmistakable.
That does not prove that no temporary backup archive had existed at some earlier moment and later disappeared. I cannot reconstruct a deleted temporary file merely because it is absent at inspection time. So I would not write that UpdraftPlus played absolutely no role. What the rescue evidence did establish was narrower and stronger: persistent WordPress backup storage was not the dominant consumer of the filesystem when the machine failed; logging was.
The next observation made that much more interesting. The oversized syslog was packed with Cloudflare WARP activity. I was seeing DNS-related failures, connectivity checks, network state changes, firewall-related operations and repeated client messages at extremely high density. Even before reconstructing their complete chronology, the log volume made WARP impossible to treat as a minor background service.
This was the first major revision of my mental model:
Initial expectation:
WordPress backup
|
v
disk fills
|
v
website fails
What rescue mode showed:
system logging
|
v
multi-gigabyte /var/log
|
v
disk fills
|
v
multiple services become suspect
The distinction mattered. If a backup simply consumed the remaining disk, deleting the backup would be sufficient treatment. If some service was continuously generating extraordinary amounts of logging, freeing space without stopping the generator would merely reset the countdown.
Freeing space without immediately destroying the evidence
There is a tension in incident response between restoring service quickly and preserving a perfect forensic record. I did not have the luxury of making a complete bit-for-bit forensic image before touching the machine. This was a personal production VPS that needed to come back online. Still, rescue mode allowed me to do considerably better than blindly deleting files from a live system.
Before changing anything, I had identified the original disk, mounted it read-only, recorded filesystem and inode usage, measured the important directory sizes, inspected the authentication logs, identified the huge syslog files, checked the persistent journal and inspected the surviving WARP logs. Those observations were already enough to rule out several simplistic explanations.
The persistent systemd journal—about 801 MB—and the separate /var/log/cloudflare-warp tree were especially valuable because they survived the cleanup and later became the basis for reconstructing the incident in much finer detail.
Only after that inspection did I remount the original filesystem read-write:
mount -o remount,rw /mnt/original mount | grep /mnt/original
Then I removed the immediate storage pressure by truncating the enormous active and rotated syslogs and deleting older compressed rotations:
truncate -s 0 /mnt/original/var/log/syslog
truncate -s 0 /mnt/original/var/log/syslog.1
rm -f /mnt/original/var/log/syslog.2.gz \
/mnt/original/var/log/syslog.3.gz \
/mnt/original/var/log/syslog.4.gz
df -h /mnt/original
The change was immediate:
Filesystem Size Used Avail Use% Mounted on /dev/vda1 8.9G 6.2G 2.7G 70% /mnt/original
Roughly 2.7 GB had been recovered. That was operationally important, but I did not want to boot the normal system yet. The logs had already made WARP suspicious enough that allowing it to start automatically would have changed network, DNS and possibly firewall state before I had even regained control.
Instead, I disabled and masked the WARP service in the offline installation:
systemctl --root=/mnt/original disable warp-svc.service systemctl --root=/mnt/original mask warp-svc.service systemctl --root=/mnt/original is-enabled warp-svc.service systemctl --root=/mnt/original is-enabled ssh.service
The result was:
warp-svc.service masked ssh.service enabled
Masking rather than uninstalling WARP was deliberate. I wanted the next boot to occur without the daemon modifying anything, but I did not want to destroy its package state or configuration before I understood what had happened. Disabling the suspect while preserving it for examination is a much better debugging move than enthusiastically deleting the suspect and then wondering where the evidence went.
Validating SSH exposed a harmless trap and then another real fault
Before leaving rescue mode, I also wanted to establish whether the installed OpenSSH configuration itself was syntactically valid. Running sshd -t inside the mounted system initially produced:
Missing privilege separation directory: /run/sshd 255
For a moment this looked like another SSH failure. It was actually a property of the testing environment. /run is runtime state created during a normal boot, and a simple chroot of an offline filesystem does not reproduce that runtime environment.
I created the expected directory inside the mounted system and repeated the configuration test:
install -d -m 0755 /mnt/original/run/sshd chroot /mnt/original /usr/sbin/sshd -t echo $?
This time the result was:
0
That was useful negative evidence. The installed sshd configuration parsed successfully, ssh.service was enabled, and the root account’s authorized-key file existed. It made a broken SSH configuration an increasingly poor explanation for the original pre-banner failure.
Then another command revealed a much more concrete problem:
ls -l /mnt/original/etc/resolv.conf cat /mnt/original/etc/resolv.conf
The output was:
-rw-r--r-- 1 root root 0 Aug 16 16:37 /mnt/original/etc/resolv.conf
/etc/resolv.conf was a zero-byte regular file.
That was not the normal resolver arrangement I expected on this Ubuntu installation. It also matched messages I had already seen from WARP complaining that it could not find usable nameservers and could not read the system DNS configuration. At that moment it was extremely tempting to construct a neat explanation immediately: perhaps the empty resolver file had broken WARP, WARP had entered a retry loop, and the retry loop had filled the disk.
I resisted making that a conclusion. I knew the state of the file and its timestamp. I knew the disk was full. I knew WARP had produced huge quantities of logs. I did not yet know the temporal ordering of those events.
That distinction became crucial later.
I preserved the empty file before replacing it:
cp -a /mnt/original/etc/resolv.conf \
/mnt/original/etc/resolv.conf.rescue-backup
Then I confirmed that systemd-resolved was enabled and restored the usual stub-resolver symlink:
systemctl --root=/mnt/original \ is-enabled systemd-resolved.service rm /mnt/original/etc/resolv.conf ln -s ../run/systemd/resolve/stub-resolv.conf \ /mnt/original/etc/resolv.conf ls -l /mnt/original/etc/resolv.conf
The resulting state was:
/etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
At this point the offline installation had four important properties:
root filesystem 70% used, about 2.7 GB free ssh.service enabled sshd configuration valid warp-svc.service masked systemd-resolved enabled /etc/resolv.conf restored to stub-resolver symlink
I synchronized the filesystem and cleanly unmounted it:
sync umount /mnt/original
Then I shut down the rescue environment normally, changed the Scaleway boot mode from Rescue back to Local, and started the VPS from its original disk.
The same VPS came back
The next normal Ubuntu 22.04 boot looked different in exactly the ways I wanted. The ordinary network stack came up, the resolver service started, Nginx, PHP-FPM and MariaDB started, and OpenSSH started normally. Server-side WARP did not start because it remained masked.
Most importantly, I could finally SSH into the machine again.
Post-recovery checks showed the root filesystem still around 70% usage with roughly 2.7 GB available. Inode usage remained only about 13%. ssh.service was enabled and active. warp-svc.service was masked and inactive. systemd-resolved was enabled and active. The VPS had its native IPv6 address and route, with no ordinary IPv4 route.
The resolver had returned to its normal stub mode:
nameserver 127.0.0.53 options edns0 trust-ad search . resolv.conf mode: stub
sshd was listening normally on port 22 and another sshd -t returned zero. The newly created /var/log/syslog was only about 124 KB shortly after recovery instead of gigabytes. The persistent journal remained at roughly 800 MB, which was fortunate because it contained much of the history I still needed.
The website recovered as well.
Operationally, this was the point at which the emergency ended. Forensically, it was the point at which the interesting part began.
Recovery did not prove the cause
It would have been easy to stop here and write a simple postmortem: “the disk filled with WARP logs, I disabled WARP, fixed DNS, and everything worked.” That would contain several true observations and still overstate what had actually been demonstrated.
The recovery changed three important variables at approximately the same time:
- I freed about 2.7 GB of filesystem space.
- I repaired the zero-byte
/etc/resolv.conf. - I prevented WARP from starting.
When the machine then recovered, that proved the combined intervention was sufficient to restore it. It did not isolate which intervention restored SSH, which one restored the web path, or which condition had originally begun the failure.
The SSH evidence was particularly awkward. We had directly observed sshd start and listen on port 22, while remote clients reached the port and died before receiving the server banner. Yet there was no clean log line saying, for example, “WARP firewall dropped this SSH connection,” and there was no controlled test in which I changed only the disk state while leaving every network component untouched. Claiming the exact low-level mechanism of the SSH failure would therefore have gone beyond the evidence.
The zero-byte resolver file created the same epistemic problem. It was unquestionably broken. Its recorded modification time was around 16:37 UTC, only a few minutes before the externally observed 16:43:31 outage. WARP also complained about missing nameservers. That made it an extremely interesting clue. But a close timestamp is evidence of sequence, not automatic proof of origin. I did not yet know which process had truncated the file, nor whether that event was the first fault or a late consequence of something that had already been wrong for days.
The disk, meanwhile, was certainly full. But the discovery that /var/log occupied 3.5 GB, with approximately 2.38 GB in syslog, meant that “disk full” had itself become a question requiring an explanation. Why had logging exploded? When had it started? Was WARP merely reporting another network problem, or was it part of that problem? Had the server gradually accumulated logs for months, or had something changed abruptly? Did the resolver failure create the logging storm, or did storage exhaustion damage the resolver later?
I used AI assistance throughout this debugging process as an interactive second pair of eyes: to suggest narrow read-only commands, interpret unfamiliar log patterns, challenge premature explanations and keep competing hypotheses visible. But the distinction between assistance and evidence mattered. The claims I trusted came from the VPS itself—filesystem measurements, service state, timestamps, package history, journals, WARP logs and network tests. Deciding how strongly those observations justified a causal statement remained my responsibility.
By the time the server was stable again, the original UpdraftPlus hypothesis had already been substantially displaced. The filesystem really had reached 100%, but persistent WordPress backups were not what occupied most of it. WARP was present in an unusual IPv6-only architecture and was producing extraordinary amounts of logging. /etc/resolv.conf had somehow become an empty file. SSH had failed before banner exchange even while its daemon appeared to be running normally.
The machine was alive again. The explanation was not.
Fortunately, rescue mode had left me with the surviving systemd journal, the WARP-specific logs, several distinct boot records, package history and the preserved empty resolver file. Those traces meant I no longer had to reason forward from symptoms. I could begin reasoning backward through time.
And that changed the question completely. I was no longer asking, “What do I delete to get my VPS back?” I was asking, “What actually happened first?”
Reconstructing the timeline from what survived
The first question was deceptively simple: what actually happened first?
Once the VPS was stable again, I stopped treating the recovered machine as something to “fix” and started treating it as evidence. I did not update Ubuntu, did not unmask WARP, and did not immediately clean up every old log. Any of those operations would have made the system tidier while simultaneously making the history harder to reconstruct.
The surviving evidence was fragmented. The giant syslog files had already been truncated as part of emergency recovery. The persistent systemd journal still contained several boots, but parts of the critical interval were missing. WARP’s own rotating files retained different pieces of history from the journal. Some data was highly detailed but recent; other data consisted only of periodic counters stretching further backward in time. In a way, the failure had started eating its own black box recorder.
I began by listing the surviving journal boots:
journalctl --list-boots
The relevant structure looked approximately like this after anonymizing the boot identifiers:
-4 boot-A Thu 2026-08-13 05:22:00 UTC — Sun 2026-08-16 17:56:59 UTC -3 boot-B Sun 2026-08-16 17:58:19 UTC — Sun 2026-08-16 18:10:38 UTC -2 boot-C Sun 2026-08-16 18:13:02 UTC — Sun 2026-08-16 18:25:41 UTC -1 boot-D Sun 2026-08-16 18:28:52 UTC — Sun 2026-08-16 18:32:51 UTC 0 boot-E Sun 2026-08-16 18:52:28 UTC — Sun 2026-08-16 19:01:29 UTC
At first glance, it was tempting to interpret 2026-08-13 05:22:00 as the literal moment that long-running boot began. I later became more cautious. What journalctl --list-boots gives me is the first and last journal entry still available for that boot. In a system that had suffered extreme logging volume, rotation failures and storage exhaustion, “first surviving journal message” and “machine powered on at exactly this second” are not necessarily equivalent.
This detail mattered because one of my early reconstructions placed the beginning of the WARP failure at the apparent August 13 boot boundary. That hypothesis was reasonable from the evidence I had then. It was also one of several diagnoses that would later move backward in time.
The visible outage was already very late in the incident
The UptimeRobot alert had anchored the public outage at 2026-08-16 16:43:31 UTC. The preserved empty /etc/resolv.conf had a modification timestamp of 16:37:08.928 UTC, only about six minutes earlier. Initially that looked enormously significant, and it was significant—but not in the way I first thought.
The deeper journal search produced an older and much more damaging timestamp:
2026-08-14 02:27:24 UTC rsyslog: ... No space left on device
The exact surrounding records varied because of lost journal data, but the important fact was unambiguous: ENOSPC already existed more than two days before the website became externally unavailable.
That immediately invalidated a neat but wrong chronology in which the resolver file became empty on August 16, WARP began malfunctioning, logging exploded, and the disk filled shortly afterward. The disk had already been unable to accept writes long before that resolver timestamp.
By August 15, the damage had reached one of the mechanisms whose job was specifically to stop logs from consuming the filesystem. The evidence showed normal log rotation could no longer operate, and WARP’s own log rotation was also failing. The sequence had become self-reinforcing:
large logging volume
|
v
free disk approaches zero
|
v
ENOSPC
|
+--> logrotate cannot run normally
|
+--> WARP cannot rotate its own logs
|
+--> services begin failing writes
|
v
more warnings and errors
|
v
still more logging
This was the point where “the disk is full” stopped being a terminal diagnosis and became an active participant in the failure. Storage exhaustion was no longer simply the result of what had gone wrong earlier. It was creating new failures of its own.
At one point the logging pressure was high enough that rsyslog reported approximately 9,092 messages lost inside a five-second rate-limiting interval. That single number was useful because it demonstrated that this was not a leisurely accumulation of ordinary service messages over several months. Something was producing error traffic on a completely different scale.
Then came secondary application failures. By August 16 around 05:07 UTC, MariaDB was also encountering storage-related write problems while trying to create temporary state. It was no longer meaningful to ask whether “WordPress” was healthy in isolation. The database, logging infrastructure, resolver state and network-control software were all sharing the same exhausted root filesystem.
The chronology now looked approximately like this:
| UTC time | Observed state | What it established |
|---|---|---|
| Before Aug 14 | WARP already producing extreme DNS-related logging | The abnormal condition predated the visible outage |
| Aug 14 02:27 | rsyslog reports No space left on device |
Root filesystem already exhausted |
| Aug 15 around 00:00 | System log rotation and WARP rotation fail | Storage containment mechanisms themselves are failing |
| Aug 16 05:07 | MariaDB encounters disk-full write failure | ENOSPC is affecting unrelated services |
| Aug 16 16:37:08 | /etc/resolv.conf becomes a zero-byte regular file |
Late resolver-state failure |
| Aug 16 16:43:31 | Cloudflare origin check reaches HTTP 522 | Application plane becomes externally unavailable |
| shortly afterward | SSH reaches TCP/22 but dies before server banner | Management plane also becomes unusable |
This made one thing very clear: the UptimeRobot alert marked the point at which I discovered the incident, not the point at which the incident began.
Measuring the WARP log storm
Knowing that logs filled the disk was still not enough. I wanted to know whether WARP merely happened to appear frequently in them or whether its failure rate could quantitatively explain the growth.
A compact hourly aggregation of the surviving warp-svc journal was extremely revealing. The figures for the morning of August 13 were approximately:
2026-08-13T05
WARP messages: 96,206
WARN: 95,105
DNS-related: 95,181
2026-08-13T06
WARP messages: 153,162
WARN: 151,403
DNS-related: 151,523
2026-08-13T07
WARP messages: 152,730
WARN: 151,022
DNS-related: 151,142
2026-08-13T08
WARP messages: 125,755
WARN: 124,326
DNS-related: 124,422
More than half a million WARP messages appeared in only those four hourly buckets, and almost all of them were warnings related to DNS.
The representative error was almost boring in its consistency:
WARN dns_proxy::errors:
DnsProxy Io
ResolveError {
kind: Proto(
ProtoError {
kind: Io(
Os {
code: 101,
kind: NetworkUnreachable,
message: "Network is unreachable"
}
)
}
)
}
Linux error 101 is ENETUNREACH: Network is unreachable.
More striking than the wording was the timing. A short excerpt showed the same error occurring over and over inside fractions of a second:
07:56:07.346 WARN ... NetworkUnreachable 07:56:07.347 WARN ... NetworkUnreachable 07:56:07.347 WARN ... NetworkUnreachable 07:56:07.348 WARN ... NetworkUnreachable 07:56:07.348 WARN ... NetworkUnreachable 07:56:07.349 WARN ... NetworkUnreachable 07:56:07.349 WARN ... NetworkUnreachable 07:56:07.350 WARN ... NetworkUnreachable ... 07:56:07.365 WARN ... NetworkUnreachable
Dozens of essentially identical failures could occur within a few tens of milliseconds. The machine was not waiting patiently for a remote resolver and occasionally recording a timeout. It was failing as quickly as the local networking stack could reject the operation.
This distinction became much clearer when I found WARP’s own DNS statistics. One interval reported:
Queries: 4978 Success: 0.0% TimedOut: 0.0% NoRecordsFound: 0.0% Other Error: 100.0% Avg Duration: 0.02ms
The 0.02ms average duration is diagnostically important. A DNS request going to an upstream server over a real network does not meaningfully time out in twenty microseconds. There is not enough time for the request to traverse the network, wait for a remote service, and fail normally.
The statistics themselves classify the failures consistently:
TimedOut: 0.0% Other Error: 100.0%
Together with errno 101, the best interpretation is that WARP attempted to send its DNS-over-WARP traffic and immediately encountered a local routing/socket failure:
DNS request
|
v
WARP DNS proxy
|
v
DNS-over-WARP path
|
v
local route lookup / socket operation
|
X
ENETUNREACH immediately
This is much stronger evidence than a generic message saying “DNS failed.” It tells me the class of failure. The resolver was not merely slow; its transport path was unusable.
WARP was connected and DNS was still broken
Another clue initially looked contradictory. While DNS was failing almost completely, WARP’s tunnel health remained excellent.
Repeated network-health telemetry reported approximately:
MonitorTunnelStats {
rtt_ms: 1,
estimated_loss: 0.0
}
This continued through periods in which the DNS proxy was producing thousands of ENETUNREACH failures.
So saying “WARP went down” was too imprecise. The outer WARP transport was plainly capable of communicating. The problem was deeper inside the dependency chain.
The configuration evidence helped explain how those two observations could coexist. The recovered client state showed:
operation_mode = Warp dns_mode = DNS Proxy tunnel_mode = Exclude-only
And the earlier shell history showed the deliberate exclusion:
warp-cli tunnel ip add-range ::/0
My original intention had been conceptually simple:
IPv6 traffic
|
+--> native VPS IPv6
IPv4 traffic
|
+--> WARP
There was nothing obviously irrational about that. The VPS had good native IPv6 and lacked normal IPv4 egress. WARP could fill that gap.
But the actual architecture contained a dependency that was easy to overlook: split-tunnel routing and DNS handling were not the same thing. Excluding ::/0 meant that ordinary IPv6 IP traffic could remain native. It did not mean that DNS resolution was simply handed back to the ordinary native resolver path.
In the recovered configuration, WARP was operating with a local DNS proxy. Conceptually the machine looked more like this:
native IPv6 network
|
v
outer MASQUE tunnel
|
WARP
|
+------------+------------+
| |
v v
IPv4 traffic WARP DNS proxy
|
v
DNS-over-WARP
|
v
DoH connection
The distinction between the outer MASQUE tunnel and the inner DNS-over-WARP path ended up being central to the whole incident.
The outer tunnel could remain healthy over IPv6. WARP could therefore report one-millisecond RTT and effectively zero loss. At the same time, an inner route required for DNS-over-WARP could become unusable and immediately return ENETUNREACH.
The failed-boot logs contained another useful implementation clue. WARP attempted to create a DNS-over-HTTPS resolver using an IPv4 endpoint, conceptually like:
resolver:
address: 192.0.2.53:443
protocol: HTTPS
hostname: cloudflare-dns.example
ip_strategy: IPv4ThenIPv6
The exact public resolver address is not important here. The important property is that it was IPv4. The host itself had no native IPv4 route, so that logical path depended on WARP’s tunnel and internal routing being correct.
This produced a rather interesting dependency:
IPv6-only host
|
+--> native IPv6 works
|
v
WARP tunnel over IPv6
|
healthy
|
+--------+---------+
| |
v v
IPv4 egress DNS-over-WARP path
|
X
route unusable
|
v
ENETUNREACH
In other words, “the WARP tunnel is healthy” and “WARP DNS is completely unusable” are not contradictory statements once they are describing different layers.
That realization also corrected another possible diagnosis. There was no strong evidence that Scaleway’s native IPv6 network had simply disappeared. If that had happened, maintaining an outer MASQUE connection with approximately 1 ms RTT and 0% measured loss would have been difficult to explain.
The health check stayed green while DNS success was effectively zero
Then I found one of the stranger pieces of evidence in the entire investigation.
At 05:23:00, WARP reported:
Queries: 4978 Success: 0.0% Other Error: 100.0%
Only about twenty seconds later:
05:23:20 DNS proxy health status: Healthy
Two minutes later:
05:25:00 Queries: 5155 Success: 0.0% Other Error: 100.0% 05:25:20 DNS proxy health status: Healthy
And the same pattern continued.
By around 05:57:
Queries: 5252 Success: 0.0% Other Error: 100.0% 05:57:20 DNS proxy health status: Healthy
It would be difficult to invent a better example of the difference between component health and functional service health. The proxy process itself may well have been alive. Its socket may have been listening. Its worker thread may have passed whatever internal liveness test the message represented. In that narrow sense, perhaps “Healthy” was technically doing exactly what its implementer intended.
But another part of the same client knew that essentially every real DNS request was failing.
The operational state was closer to:
local DNS proxy process healthy outer MASQUE transport healthy DNS-over-WARP route failed DNS functional success approximately 0% actual resolver service unusable
That is what I mean here by a false-green health state. I do not need to claim that the internal health-check code was “wrong” according to its specification. The problem is that the label did not represent what an operator reasonably needs to know: can this DNS service actually resolve names?
There was independent evidence from applications as well. DNS-dependent operations produced errors such as:
Temporary failure in name resolution
So the contradiction was not confined to two internal counters disagreeing with one another. Real callers could not resolve names, while the subsystem continued publishing a green health message.
The internal error counter made the scale almost absurd. At one point the WARP statistics showed approximately:
dns_proxy.doh_err_other mode="dns-over-warp" count=7,441,921
Sixteen minutes later:
count=7,482,811
That is an increase of 40,890 errors in sixteen minutes:
~42.6 errors / second ~2,556 errors / minute ~153,338 errors / hour
Those figures independently matched the roughly 150,000 DNS warning messages per hour visible in the journal. This was an important moment in the reconstruction because it connected two different evidence sources quantitatively.
The logs were not merely “full of WARP messages.” WARP’s own internal error counter was increasing at almost exactly the same rate as the external warning stream.
The mechanism was therefore very strongly demonstrated:
DNS-over-WARP request
|
v
ENETUNREACH
|
v
WARP warning
|
v
next request
|
v
ENETUNREACH
|
v
another warning
|
...
At roughly 42 failures each second, ordinary error logging becomes a storage workload.
The server was, in a very literal sense, logging itself to death. (It was at least admirably thorough about announcing the process.)
The retry rate was itself part of the failure
Another subtle detail emerged when I compared DNS traffic before and after the transition.
Under normal conditions, the WARP statistics typically showed only a handful of queries per two-minute reporting interval:
Queries: 4 Queries: 6 Queries: 4 Queries: 4
After the failure, the numbers became:
Queries: 3832 Queries: 3999 Queries: ~5000 Queries: ~5200
The demand increased by roughly three orders of magnitude.
That meant the server was not simply performing its normal DNS workload and having every request fail. The failure itself appears to have induced retries. A caller asks for DNS, gets a near-instant failure, retries quickly, fails again, and some combination of applications, system services and possibly WARP’s own machinery repeats the process.
I cannot attribute every retry to one specific process from the surviving evidence. WordPress/PHP, package-related services, NTP, system components, WARP’s connectivity machinery and other daemons may all have contributed. The logs do not justify assigning the entire retry storm to one caller.
What they do show is a feedback structure:
normal DNS request
|
v
DNS-over-WARP has no usable route
|
v
immediate ENETUNREACH
|
v
caller does not wait for a long timeout
|
v
retry happens quickly
|
v
another immediate failure
|
v
query volume rises dramatically
Then WARP attached another feedback path to that one:
each failed query
|
v
WARN record
|
v
syslog + journal growth
|
v
free disk decreases
|
v
ENOSPC
|
+--> logrotate fails
|
+--> services produce new failures
|
+--> more retries
|
v
more logging
This is the point where the incident stopped looking like a simple sequence of unrelated faults. It was becoming a dynamical system with positive feedback.
A route failure did not merely remain a route failure. It changed the behaviour of callers. Their retries changed the behaviour of the logger. The logger changed the amount of available storage. Storage exhaustion changed the behaviour of unrelated services and of the observability mechanisms themselves. Those new failures then generated still more messages.
Locally, each component’s action was understandable. Retry after failure. Log an error. Rotate logs. Write a database temporary file. Reconfigure networking when conditions change. Globally, the combination was pathological.
The empty resolver file moved from “cause” to “late-stage symptom”
The zero-byte /etc/resolv.conf had initially seemed like the obvious starting point because it matched WARP’s terminal DNS messages so neatly.
During one of the later failed states WARP reported messages along the lines of:
Could not determine resolv.conf file owner: File was empty systemd-resolved is operating in a non-standard mode, continuing with overwrite mode=Foreign
Then:
Starting Warp Connection dns_mode=DNS Proxy
followed by:
FailedToParseDnsConfig no nameservers found in config
That correspondence was real. An empty resolver file was unquestionably a problem for WARP’s DNS initialization. It simply was not the original problem.
The timestamp disproved that chronology.
The preserved file showed:
Size: 0 Modify: 2026-08-16 16:37:08.928 UTC
Yet the DNS-over-WARP failure was already fully established in surviving WARP records from August 13, and later evidence pushed its actual beginning back even further.
So this sequence:
resolv.conf becomes empty
|
v
WARP DNS breaks
|
v
warning storm
|
v
disk fills
could no longer be correct.
A chronology that fit the evidence much better was:
WARP DNS path fails
|
v
massive retry/warning activity
|
v
disk fills
|
v
system spends days under ENOSPC
|
v
later resolver/network transition
|
v
resolv.conf becomes empty
|
v
terminal network state becomes worse
This was a useful reminder of why timestamps matter so much in debugging. Two states can have an obvious functional relationship and still occur in the opposite causal order from what intuition suggests.
The empty resolver file was still important. In fact, it may have helped turn an already unhealthy server into the final externally visible outage. But by this stage I considered it a secondary failure, not the initiating fault.
What happened around 16:37 on August 16
The WARP-specific files happened to preserve unusually precise timestamps around the final transition, even though the ordinary journal had a large blind spot there.
At approximately:
16:37:02.696
the WARP state directory under /var/lib/cloudflare-warp changed.
About 25 milliseconds later:
16:37:02.721 DEBUG route-change: Routes changed
Then at:
16:37:08.615
WARP entered a connectivity/captive-network detection sequence.
The detailed sequence was particularly interesting because different tests did not all fail together. At approximately 16:37:08.648, a DNS check reported success. Shortly afterward an HTTPS retrieval also succeeded. Native IPv6 connectivity was evidently still functional enough for those checks.
But a connectivity attempt to an IPv4 destination failed immediately:
connect to 192.0.2.80:80 Network is unreachable
That mattered because providing IPv4 egress was the whole reason WARP was installed on this IPv6-only machine in the first place. At that moment, its narrow practical purpose was already unavailable.
Then, at:
16:37:08.928
the preserved /etc/resolv.conf acquired its zero-byte modification timestamp—only around 220 milliseconds after that connectivity sequence.
Six minutes and roughly twenty-two seconds later:
16:43:31 Cloudflare origin request -> HTTP 522
The late-stage timeline was therefore very tight:
16:37:02.696
WARP state changes
|
| ~25 ms
v
16:37:02.721
route-change event
|
| ~6 sec
v
16:37:08.615
WARP connectivity detection
|
+--> DNS test succeeds
|
+--> IPv6 path works
|
+--> IPv4 path -> ENETUNREACH
|
v
16:37:08.928
/etc/resolv.conf becomes 0 bytes
|
| ~6m22s
v
16:43:31
Cloudflare -> HTTP 522
|
v
SSH management also becomes unusable
That is strong temporal evidence that WARP was actively involved in the networking state surrounding the resolver transition. It is not, however, process-level proof that a particular WARP write system call truncated the file. I never captured that write operation directly.
Several alternative low-level mechanisms remain possible: WARP itself may have rewritten the file and failed partway through; another resolver/network component may have interacted with WARP; a write under ENOSPC may have produced an incomplete state; some restoration operation may have removed content before replacement failed; or another process may have touched the file during the same transition.
The timestamps justify saying that the empty file appeared during active WARP route/connectivity activity on a system already under severe storage pressure. They do not justify pretending I watched WARP execute truncate("/etc/resolv.conf", 0).
There was another complication: the ordinary journal contained essentially no usable records from the exact critical window. A query around 16:37 returned:
-- No entries --
The next surviving ordinary logging evidence appeared much later, with rsyslog still reporting No space left on device. In other words, the failure had degraded the system responsible for documenting the failure.
Even some WARP-specific files showed signs of damaged or concatenated lines, with timestamps from different moments appearing joined together. I treated those carefully. When a record looked malformed, I did not use it as the sole basis for a causal claim.
This created an interesting forensic asymmetry: the most severe part of the incident was also the least reliably recorded part of it.
Then the beginning moved backward again
For a while I thought the DNS failure had probably begun around the apparent August 13 journal boundary, because the first partial hour already contained around 95,000 WARP warnings. That was a reasonable inference from the journal.
Then I searched WARP’s own periodic DNS-statistics files rather than only the detailed per-query errors.
The earliest retained statistics showed completely normal operation on August 10:
2026-08-10T00:01:00Z Queries: 4 Success: 100.0% Other Error: 0.0% 2026-08-10T00:03:00Z Queries: 4 Success: 100.0% Other Error: 0.0% 2026-08-10T00:05:00Z Queries: 4 Success: 100.0% Other Error: 0.0%
And that pattern continued for hour after hour: normally four to six queries per interval, almost always 100% success, with average durations around zero to a few milliseconds.
More importantly, the same healthy state continued into August 11:
06:45:00 Queries: 8 Success: 100.0%
06:47:00 Queries: 4 Success: 100.0%
06:49:00 Queries: 4 Success: 100.0%
06:51:00 Queries: 4 Success: 100.0%
06:53:00 Queries: 6 Success: 100.0%
Other Error: 0.0%
Avg Duration: 1.83ms
Then, two minutes later:
06:55:00 Queries: 722 Success: 0.6% TimedOut: 0.0% Other Error: 99.4% Avg Duration: 0.16ms
Two minutes after that:
06:57:00 Queries: 3832 Success: 0.1% TimedOut: 0.0% Other Error: 99.9% Avg Duration: 0.01ms
And at 06:59:
Queries: 3999 Success: 0.2% TimedOut: 0.0% Other Error: 99.8% Avg Duration: 0.01ms
This was one of the most decisive discoveries in the entire investigation.
The configuration had not been continuously malfunctioning from the day I installed it. It had not slowly drifted from 100% success to 90%, then 50%, then zero over months. The historical telemetry showed a step transition.
At 06:53 UTC on August 11, DNS was healthy.
By 06:55 UTC, it was essentially dead.
And by 06:57, the query rate had already exploded from a handful every two minutes to thousands.
The beginning of the incident had moved backward by almost two days from where I had initially placed it.
That also answered an important practical question: how could this VPS have worked for months if the architecture was fundamentally flawed?
Because the architecture did work.
Whatever its latent risks, the surviving telemetry demonstrated normal DNS operation under this exact general configuration until August 11. The failure required a state transition.
The route-change cluster around 06:54
Once the DNS statistics narrowed the transition to a roughly two-minute window, I extracted only the WARP network events surrounding it instead of dumping yet more millions of repetitive errors.
The tunnel-health records remained almost boringly stable throughout:
06:52:04 RTT 1 ms estimated_loss 0.0 06:52:19 RTT 1 ms estimated_loss 0.0 06:52:34 RTT 1 ms estimated_loss 0.0 06:52:49 RTT 1 ms estimated_loss 0.0 06:53:04 RTT 1 ms estimated_loss 0.0 06:53:19 RTT 1 ms estimated_loss 0.0 06:53:34 RTT 1 ms estimated_loss 0.0 06:53:49 RTT 1 ms estimated_loss 0.0 06:54:04 RTT 1 ms estimated_loss 0.0 06:54:19 RTT 1 ms estimated_loss 0.0 06:54:34 RTT 1 ms estimated_loss 0.0 06:54:49 RTT 1 ms estimated_loss 0.0 06:55:04 RTT 1 ms estimated_loss 0.0 ... 06:57:49 RTT 1 ms estimated_loss 0.0
The outer transport did not show a corresponding collapse.
At the same time, WARP was observing repeated route changes:
06:52:38.821 Routes changed 06:52:47.781 Routes changed 06:53:23.877 Routes changed 06:53:31.814 Routes changed 06:54:06.885 Routes changed 06:54:15.845 Routes changed 06:54:15.847 Routes changed 06:54:51.942 Routes changed 06:54:59.877 Routes changed 06:54:59.879 Routes changed 06:56:28.197 Routes changed 06:56:35.877 Routes changed 06:57:13.253 Routes changed 06:57:19.910 Routes changed 06:57:58.309 Routes changed
And against those events, the DNS statistics were:
06:53:00 Success: 100.0% 06:55:00 Success: 0.6% Other Error: 99.4% 06:57:00 Success: 0.1% Other Error: 99.9%
The temporal relationship is extremely suggestive. Several route-change notifications cluster directly across the moment when DNS moves from perfect success to almost complete failure.
But this is exactly where I have to stop one step short of certainty.
Route-change notifications also occurred before the failure and continued afterward. The logs say “Routes changed”; they do not give me a complete semantically decoded diff of WARP’s internal routing state at every one of those milliseconds. I cannot point to, for example, the 06:54:15.845 event and claim, “This exact event removed the DNS route.”
What I can say with much greater confidence is:
06:53
DNS-over-WARP functional
|
| repeated route/state events
v
06:55
DNS-over-WARP almost completely non-functional
outer MASQUE tunnel remains healthy
That is evidence of a sudden state transition, but the implementation-level transition responsible for it remains hidden behind WARP’s internal routing logic.
The possible classes of explanation had narrowed considerably:
- an internal WARP route/state transition;
- an interaction between the WARP client and this IPv6-only topology;
- a split-tunnel or virtual-interface state problem;
- a firewall/routing interaction inside the client;
- an operating-system/WARP DNS-routing interaction;
- or a transient external network event that WARP handled badly and failed to recover from.
A simple “Internet outage” no longer fit the evidence. Neither did “the WARP package was updated and immediately broke,” because package history showed that the installed WARP version was still 2026.3.846.0, installed months earlier, with no corresponding package upgrade at the August 11 transition.
The specific later DNS regressions documented for other WARP release lines were therefore interesting background, but they were not evidence that this VPS had hit that exact published bug. I deliberately kept that out of the proven causal chain.
What the causal structure looked like by this point
By now I could separate several layers that had originally been mixed together under the phrase “the VPS failed.”
The beginning was no longer the website outage. It was no longer the full filesystem. It was no longer the empty resolver file. It was no longer even the apparent August 13 boot.
The earliest strongly demonstrated transition I could locate was this:
Before 2026-08-11 06:54 UTC
WARP DNS statistics:
success ~100%
normal query volume
outer MASQUE:
healthy
|
| abrupt state transition
v
Around 2026-08-11 06:54 UTC
DNS-over-WARP:
route becomes unusable
|
v
Linux:
errno 101
ENETUNREACH
|
v
DNS success:
~0%
query volume:
rises from ~4-6 / 2 min
to thousands / 2 min
|
v
WARP:
emits repeated WARN per failure
|
v
syslog + journal:
grow at extreme rate
|
v
2026-08-14
root filesystem:
ENOSPC
|
+--> logrotate fails
+--> WARP rotation fails
+--> service writes fail
+--> observability degrades
|
v
2026-08-16 16:37
resolver/network state changes
/etc/resolv.conf becomes empty
|
v
2026-08-16 16:43
Cloudflare origin:
HTTP 522
|
v
SSH:
TCP reachable
protocol dies before server banner
One more fact made the upper part of that chain particularly strong: WARP simultaneously reported its DNS proxy as Healthy while its own functional statistics showed essentially zero successful DNS resolution. The mechanism that might normally have recognized a degraded state therefore did not appear to trigger any effective circuit breaking, recovery mode or suppression of the millions of repeated errors.
By this stage, I was comfortable saying that a WARP DNS-routing failure initiated the observable cascade and that WARP’s repeated warning behaviour amplified it into filesystem exhaustion. I was also comfortable saying that ENOSPC later damaged the system broadly enough to create secondary DNS, logging, database and network-management failures.
What I still could not say was what code path, route calculation, virtual-interface transition or firewall state inside WARP produced that first ENETUNREACH state around 06:54 on August 11.
The forensic reconstruction had reached the edge of what the host itself could tell me.
I could see the system immediately before the transition. I could see it immediately after. I could see route-change events happening across the boundary. I could see that the MASQUE transport stayed healthy. I could see DNS collapse from 100% success to effectively zero within two minutes. What I could not see was the one internal implementation decision that connected those states.
That distinction would become central to how I ultimately described the root cause.
Drawing the root-cause boundary without inventing the missing piece
At that point I had enough evidence to describe the incident rigorously, but only if I resisted the temptation to force the final unknown into a neat answer.
The simplest question was: was this just “poor WARP design”?
I do not think that is a sufficiently precise conclusion. The configuration had worked normally for months, and the historical telemetry demonstrated 100% DNS success immediately before the August 11 transition. I therefore cannot argue that the basic topology was inherently non-functional from the beginning. Nor do I have enough evidence to identify a particular Cloudflare source-code defect as the event that suddenly made the DNS-over-WARP path return ENETUNREACH.
What the evidence does support is a layered root-cause model.
The initiating fault
Between approximately 06:53 and 06:55 UTC on August 11, WARP’s DNS-over-WARP path abruptly moved from normal operation into a state in which essentially every DNS request encountered a local routing failure.
The transition is directly visible:
06:53:00 Queries: 6 Success: 100.0% Other Error: 0.0% Avg Duration: 1.83ms 06:55:00 Queries: 722 Success: 0.6% Other Error: 99.4% Avg Duration: 0.16ms 06:57:00 Queries: 3832 Success: 0.1% Other Error: 99.9% Avg Duration: 0.01ms 06:59:00 Queries: 3999 Success: 0.2% Other Error: 99.8% Avg Duration: 0.01ms
Later detailed records identify the error class:
code: 101 kind: NetworkUnreachable message: "Network is unreachable"
Meanwhile the WARP network-health monitor continued reporting roughly:
RTT: 1 ms estimated_loss: 0.0
So the initiating failure was not a complete disappearance of WARP’s outer transport. It was much narrower and more interesting: an inner DNS-over-WARP routing path became unusable while the outer MASQUE tunnel remained healthy.
That is the lowest causal layer I can establish from the host evidence.
The next question—why did that inner path become unusable?—remains unanswered. The route-change cluster around 06:54 is suspicious, and the IPv6-only topology, split-tunnel state, WARP’s virtual networking, firewall manipulation and Linux resolver integration are all relevant possibilities. But none of the surviving records exposes the exact internal route calculation or state-machine transition responsible for the first failure.
So my final wording would be:
Initiating fault:
WARP DNS-over-WARP routing abruptly became unusable.
Exact implementation trigger:
unknown from retained host evidence.
That distinction is important. “Unknown” here does not mean the entire root cause is unknown. It means that the causal reconstruction has reached an implementation boundary for which the necessary internal instrumentation is not available to me.
The amplification mechanism
The initiating networking fault alone should not have destroyed the VPS.
A DNS route can fail. A tunnel can reconnect. A service can return an error. Those are ordinary operational events. What made this incident catastrophic was what happened after that fault persisted.
Query volume rose from roughly four to six requests every two minutes into several thousand. Nearly every failed request generated a WARP warning. The internal DNS error counter and the externally visible logging rate matched remarkably closely:
40,890 additional DoH errors / 16 minutes ≈ 42.6 errors / second ≈ 2,556 errors / minute ≈ 153,338 errors / hour
At the same time, WARP continued saying:
DNS proxy health status: Healthy
while its own statistics were effectively saying:
DNS success: ~0% DNS errors: ~100%
This is where I think criticism of the failure-handling design becomes justified even though the initiating implementation defect remains unknown.
First, the health representation was false-green at the service level. Perhaps the specific health check only meant that a local proxy task or socket remained alive. If so, the message may have been internally consistent with its narrow definition. Operationally, however, another part of the same software knew that virtually no DNS request was succeeding.
A more useful system-level state would have distinguished those layers:
Local DNS proxy process: HEALTHY Outer MASQUE transport: HEALTHY DNS-over-WARP transport: FAILED DNS success rate: ~0% Overall DNS service: UNHEALTHY WARP state: DEGRADED
Instead, Healthy was doing an impressive amount of semantic work.
Second, the repeated error handling had effectively unbounded amplification. A persistent identical networking error became a high-rate logging workload capable of consuming gigabytes of storage. Rate limiting, duplicate suppression, exponential backoff, aggregation or a circuit breaker could all have changed the trajectory.
For example, a daemon could conceivably transform:
WARN DNS path unreachable WARN DNS path unreachable WARN DNS path unreachable WARN DNS path unreachable WARN DNS path unreachable ... hundreds of thousands more
into something like:
WARN DNS-over-WARP path unreachable WARN suppressed 82,416 equivalent failures during the last 60 seconds
That would not have fixed the route. It might, however, have prevented a network fault from becoming a storage catastrophe.
Third, the WARP client controlled more of the machine than my actual requirement demanded. I wanted a narrow capability:
native IPv6 already available
need:
occasional IPv4 egress
The installed solution effectively introduced dependencies on:
IPv4 egress + local DNS proxy + DNS configuration + routing-table manipulation + virtual networking + firewall state + daemon logging
None of those features is inherently unreasonable for a general-purpose secure network client. The architectural lesson is about scope: a mechanism introduced to solve one narrow reachability problem had acquired authority over several critical host subsystems.
The resource-exhaustion cause
The next stage is considerably stronger evidentially.
WARP’s DNS-warning storm drove system logging into a completely abnormal regime. By recovery time:
/var/log/syslog ≈ 2.38 GB systemd journal ≈ 0.80 GB /var/log total ≈ 3.5 GB root filesystem ≈ 8.9 GB usable
On a root filesystem of that size, this was fatal.
The first surviving ENOSPC evidence appeared on August 14. Then log rotation itself failed. WARP’s own rotation failed. MariaDB later failed writes. Observability deteriorated. The system remained in that condition for days.
So the cause of the resource catastrophe is much less ambiguous:
persistent DNS routing failure
|
v
retry/query explosion
|
v
per-query WARN amplification
|
v
multi-gigabyte logging
|
v
root filesystem exhausted
|
v
ENOSPC
At that point, ENOSPC became a new causal force rather than merely an outcome.
The terminal collapse
The final stage remains partly inferential.
The system spent more than two days operating at or near absolute storage exhaustion. On August 16 at 16:37, WARP state and route activity occurred; IPv4 connectivity was observed failing; then /etc/resolv.conf became a zero-byte regular file. Approximately six minutes later Cloudflare could no longer reach the origin and returned HTTP 522. SSH subsequently became unusable before authentication even though sshd itself started and listened normally.
This strongly supports a terminal chain roughly like:
ENOSPC
+
already-broken WARP DNS/routing state
|
v
late network/resolver transition
|
v
/etc/resolv.conf becomes empty
|
v
WARP cannot reconstruct normal DNS state
|
v
routing / firewall / resolver instability
|
+----------------------+
| |
v v
HTTP origin failure SSH management loss
Cloudflare 522 pre-banner close
But two exact arrows in that lower section remain unobserved.
I cannot prove which process performed the write that left /etc/resolv.conf empty. The timestamps strongly associate the event with active WARP/network state changes, but association at millisecond resolution is still not a captured system call.
Likewise, I cannot prove the precise layer that killed inbound SSH. The evidence established a TCP connection to port 22 and demonstrated that sshd was listening, yet the remote SSH banner never arrived and normal authentication logging did not appear. WARP’s routing/firewall activity makes it a plausible participant, especially in the terminal degraded state, but I did not isolate a specific firewall rule, packet drop or userspace failure responsible for the pre-banner close.
Those uncertainties belong in the postmortem rather than being edited away.
Why rebooting could not rescue it
One thing that had initially seemed strange was the persistence of the failure through reboots. Usually, rebooting a malfunctioning network daemon is at least worth trying. Here it was largely ineffective because the important state was not transient RAM state.
A reboot did not remove:
the full root filesystem the multi-gigabyte logs WARP's installed configuration WARP's autostart the damaged resolver state persistent networking configuration
So rebooting effectively performed:
load same filesystem
|
v
start same services
|
v
start WARP again
|
v
restore same persistent configuration
|
v
encounter same broken conditions
A reboot is a restart of execution. It is not necessarily a rollback of state.
That distinction explains why the server could boot cleanly enough to show normal filesystem checks, start Nginx, PHP, MariaDB, SSH and WARP, and still be operationally unreachable. Each service starting successfully was only a statement about that service at one layer of the stack.
The rescue environment succeeded because it changed the control plane entirely. Instead of asking the failed operating system to repair itself while its own network client, resolver and logging system were active, I booted a separate operating system and mounted the original root filesystem from outside.
That was not merely convenient. It broke the dependency cycle.
FAILED NORMAL ENVIRONMENT
root filesystem full
DNS damaged
WARP active
network state unstable
SSH inaccessible
X
|
| cannot reliably repair itself
|
v
INDEPENDENT RESCUE ENVIRONMENT
separate OS
separate network stack
WARP not running
original filesystem mounted externally
|
v
persistent state can be repaired offline
This is one of the strongest design lessons of the incident: a recovery mechanism is most valuable when it does not depend on the subsystem that has failed.
Recovery was successful, but it was not a controlled experiment
The machine recovered after three substantial changes had been made in rescue mode:
1. approximately 2.7 GB of disk space freed 2. /etc/resolv.conf restored to normal systemd-resolved configuration 3. warp-svc disabled and masked
That combined intervention demonstrated operational sufficiency: after those changes, normal boot, SSH and the website returned.
It did not tell me which single change, independently, was necessary and sufficient for restoring SSH or HTTP.
A perfectly controlled causal experiment would have changed one variable at a time:
free disk only
|
v
boot and test
then perhaps restore DNS only
|
v
boot and test
then disable WARP
|
v
boot and test
But this was a production VPS whose management path was already lost. The correct objective was recovery, not experimental purity. Once I had mounted the system externally and found both a full filesystem and a broken resolver configuration while WARP was deeply implicated in the network failure, deliberately leaving one known fault in place merely to obtain cleaner causal isolation would have been an unnecessary operational risk.
This is another useful distinction between incident response and laboratory debugging. Sometimes the safest repair necessarily collapses several experimental variables at once.
Validating the recovered machine
Recovery was not complete merely because I could finally SSH into the server again. I wanted a structured post-recovery snapshot showing that the relevant layers were actually sane.
I collected time, storage, memory, service, networking, DNS and SSH state together:
{
echo '=== TIME ==='
date -u
timedatectl
uptime
echo
echo '=== STORAGE ==='
df -h /
df -i /
du -sh /var/log
ls -lh /var/log/syslog*
journalctl --disk-usage
echo
echo '=== MEMORY ==='
free -h
swapon --show
echo
echo '=== SERVICES ==='
systemctl is-enabled ssh.service
systemctl is-active ssh.service
systemctl is-enabled warp-svc.service
systemctl is-active warp-svc.service
systemctl is-enabled systemd-resolved.service
systemctl is-active systemd-resolved.service
echo
echo '=== NETWORK ==='
ip -br address
ip -4 route
ip -6 route
echo
echo '=== DNS ==='
ls -l /etc/resolv.conf
cat /etc/resolv.conf
resolvectl status
echo
echo '=== SSH ==='
ss -lntp | grep ':22' || true
sshd -t
echo "sshd-test=$?"
} | tee /root/incident-YYYY-MM-DD/current-state.txt
The recovered storage state was:
Filesystem Size Used Avail Use% /dev/vda1 8.9G 6.2G 2.7G 70% Inodes: IUse% 13% /var/log: 865M /var/log/syslog: 124K persistent + active journal: 800.4M
The inode count was still normal, confirming again that the incident had been block-storage exhaustion rather than inode exhaustion.
Memory also looked ordinary for this small machine:
Mem total: 951 MiB used: ~327 MiB available: ~312 MiB Swap: 1.0 GiB configured
There was no indication that an unrecovered memory crisis was lurking underneath the storage incident.
The service state was exactly what I wanted:
ssh.service:
enabled
active
warp-svc.service:
masked
inactive
systemd-resolved.service:
enabled
active
The resolver had returned to the standard stub arrangement:
/etc/resolv.conf
-> ../run/systemd/resolve/stub-resolv.conf
nameserver 127.0.0.53
options edns0 trust-ad
And resolvectl showed native IPv6 DNS servers on the real network interface, represented here with documentation addresses:
Link 2
Current Scopes: DNS
DefaultRoute: yes
Current DNS Server:
2001:db8::53
DNS Servers:
2001:db8::53
2001:db8::54
SSH validation was also explicit:
LISTEN 0 128 0.0.0.0:22 0.0.0.0:* LISTEN 0 128 [::]:22 [::]:* sshd-test=0
The IPv4 wildcard listener does not imply that the VPS suddenly gained native IPv4 routing; it simply shows how sshd bound its sockets. The machine’s externally useful network remained its native IPv6 path.
Most importantly, actual remote SSH now worked and the WordPress site was reachable through Cloudflare again. Syntax validation, local socket state, resolver state and external functional behaviour all agreed.
That combination is much stronger than any single “service active” result.
I left WARP masked
Once the machine was working, there was an obvious temptation to re-enable WARP “just to see whether the problem comes back.” I deliberately did not do that.
At that point, reproducing a production outage would have produced little new evidence relative to the risk.
More importantly, I reconsidered why WARP was on this server at all.
The public WordPress site sits behind Cloudflare’s reverse proxy. That inbound path is conceptually separate from server-side WARP:
WEB VISITOR
|
v
Cloudflare edge
|
v
IPv6 origin VPS
An IPv4 visitor does not require the origin VPS itself to establish an outbound WARP tunnel merely to reach a Cloudflare-proxied website. Cloudflare is already the public-facing intermediary.
WARP had been useful for a different problem: outbound IPv4 connectivity from an otherwise IPv6-only host.
Those two functions should not be confused:
Cloudflare reverse proxy:
inbound website reachability
Cloudflare WARP client:
host-side outbound networking
+ DNS/routing/firewall integration
If I later discover a genuine server workload that requires IPv4 egress, I would prefer to solve that requirement as narrowly as possible rather than automatically restoring the same broad dependency graph. Depending on provider capabilities, that could mean a dedicated egress mechanism, a proxy, NAT64 or another constrained solution. The exact replacement is a separate engineering decision.
For the recovered production system, “WARP remains masked” was not unfinished repair. It was a deliberate reduction of failure surface.
Preserving the evidence before improving the server
There were hundreds of pending package updates on the VPS after recovery. Ordinarily that would immediately invite maintenance. During a forensic investigation, however, an upgrade is also evidence destruction.
A package upgrade could replace binaries, alter service units, rotate logs, modify configuration, restart networking and change precisely the software version involved in the incident. So before doing routine maintenance, I preserved the relevant artifacts.
I created an incident directory and copied the surviving evidence:
INC=/root/incident-YYYY-MM-DD
mkdir -p "$INC/preserved"
cp -a /var/log/cloudflare-warp \
"$INC/preserved/"
cp -a /var/log/apt/history.log* \
"$INC/preserved/" 2>/dev/null || true
cp -a /var/log/dpkg.log* \
"$INC/preserved/" 2>/dev/null || true
cp -a /etc/resolv.conf.rescue-backup \
"$INC/preserved/" 2>/dev/null || true
journalctl --list-boots \
> "$INC/preserved/journal-boots.txt"
systemctl cat warp-svc.service \
> "$INC/preserved/warp-svc.service.txt"
dpkg-query -W cloudflare-warp \
> "$INC/preserved/warp-version.txt"
Then I created a compressed archive while retaining filesystem metadata:
tar --xattrs --acls \ -C /root \ -czf /root/incident-evidence.tar.gz \ incident-YYYY-MM-DD sha256sum /root/incident-evidence.tar.gz \ | tee /root/incident-evidence.sha256
I will not publish the real checksum or raw diagnostic bundle here. The archive contains system-specific evidence that is useful for investigation but unnecessary for a public article.
The SHA-256 has a simple purpose: if I later hand the archive to a vendor or inspect it after other maintenance has occurred, I can verify that the preserved evidence bundle is still exactly the one created at the end of the investigation.
That is a small but useful step from ordinary debugging toward proper incident forensics.
Collecting WARP diagnostics without reintroducing the failure
I also ran Cloudflare’s WARP diagnostic collector while leaving the service masked:
cd /root/incident-YYYY-MM-DD warp-diag
As expected, the collector complained:
warp_diag: Gathering data from WARP service... warp_diag: Gathering system info and log files... warp_diag: Failed to communicate with WARP service over IPC: No such file or directory (os error 2) warp_diag: Some information will be missing in final output... warp_diag: Debugging information stored in: ./warp-debugging-info-YYYYMMDD-HHMMSS.zip
That was normal under the circumstances. warp-svc was intentionally masked and inactive, so the diagnostic tool could not query the live daemon over IPC. The archive still captured useful static and historical information.
I specifically chose not to restart WARP merely so the diagnostic collector could obtain a prettier report. The evidence already showed that WARP had participated in a severe production incident. Re-enabling the suspected subsystem in order to improve the quality of its own bug report would have been an oddly faithful reenactment of the problem.
What I would report to Cloudflare
By the end of the investigation, the vendor report could be much more precise than “WARP broke my VPS.”
A suitable subject was:
Linux WARP DNS reports Healthy while DNS-over-WARP queries fail with ENETUNREACH, causing log storm and disk exhaustion
The important facts for such a report are:
Environment:
Ubuntu 22.04
IPv6-only VPS
cloudflare-warp 2026.3.846.0
native IPv6
::/0 excluded from WARP
WARP used primarily for IPv4 egress
Historical behaviour:
configuration worked normally for months
Transition:
2026-08-11 06:53 UTC
DNS success 100%
2026-08-11 06:55 UTC
DNS success 0.6%
Other Error 99.4%
2026-08-11 06:57 UTC
DNS success 0.1%
Other Error 99.9%
At the same time:
MASQUE RTT ≈ 1 ms
estimated loss ≈ 0
Failure class:
errno 101
ENETUNREACH
Network is unreachable
failures effectively instantaneous
Observability:
"DNS proxy health status: Healthy"
while functional DNS success ≈ 0%
Amplification:
~100,000-150,000 DNS warnings/hour
millions of dns-over-warp errors
Consequence:
multi-gigabyte system logs
root filesystem reaches 100%
rsyslog ENOSPC
logrotate failure
WARP rotation failure
MariaDB write failures
later resolver/network collapse
Cloudflare HTTP 522
SSH management unavailable
The questions I would want Cloudflare engineering to answer are similarly narrow:
- What internal condition can make DNS-over-WARP return local
ENETUNREACHwhile the outer MASQUE tunnel remains healthy? - What exactly does
DNS proxy health status: Healthymeasure, and should end-to-end functional DNS failure affect that health state? - Should repeated identical DNS errors be aggregated or rate-limited so that a persistent network failure cannot exhaust the host filesystem through logging?
- Does an IPv6-only host using an exclude-only configuration with
::/0expose any known route, resolver or DNS-over-WARP edge case?
Those questions do not presume that Cloudflare is definitely responsible for the original state transition. They identify the points where vendor implementation knowledge is required.
That is also where I think my own root-cause investigation should stop. Reproducing the state on the production VPS with deeper tracing might answer more, but it would create a poor risk-to-information trade. Cloudflare can inspect implementation details I cannot see; I should not manufacture certainty by experimenting destructively with a recovered server.
What the incident changed in how I think about monitoring
The original monitoring setup told me when the website finally became unreachable.
That was useful, but dramatically late.
The pathological transition occurred around August 11. The public outage did not happen until August 16. For several days the machine was in a state that was seriously abnormal yet externally functional enough to escape notice.
A better monitoring model would therefore look for deterioration rather than only death.
At minimum, on a small VPS I now want alerts for:
- root filesystem usage crossing warning and critical thresholds;
- unexpected growth rate in
/var/log, not merely its absolute size; - inode exhaustion separately from block exhaustion;
- persistent journal size;
- repeated
ENOSPCevents; - failed
logrotateruns; - DNS functional success rather than only resolver-process liveness;
- loss of an administrative/control-plane check independently of HTTP availability.
The rate of change matters especially here.
A filesystem at 70% usage is not necessarily alarming. A filesystem moving from 70% to 80% to 90% at several hundred megabytes per hour is a different phenomenon entirely.
Similarly:
DNS success = 99.9%
is a state measurement, while:
100% -> 0.6% -> 0.1% within four minutes
is a trajectory.
The latter tells me that the system has moved into another operating regime.
I would also treat the management plane as something worth monitoring independently. The most unpleasant part of this incident was not that WordPress stopped serving pages; it was that the ordinary administrative path disappeared at almost the same time.
A public service and its repair mechanism should ideally not fail together.
Logging needs a resource budget
The incident also changed how I think about logs on very small machines.
Logging is usually discussed as an observability problem: how much information do I need to understand failures?
But logging is also resource consumption.
event | v format message | v write message | v filesystem blocks | v I/O | v rotation / compression / retention
None of those resources is infinite.
On a server with roughly 9 GB of root storage, allowing one daemon’s repeated warnings to accumulate into multiple gigabytes means the logging policy has become part of the system’s availability architecture.
I would now put explicit limits around both systemd-journald and traditional syslog retention. The exact numbers should depend on the machine and workload, but the principle is simple: observability must not be allowed to consume the resources required for basic operation.
More importantly, log rotation needs headroom. A configuration saying “logs rotate daily” is not sufficient protection if the disk can reach absolute ENOSPC before the next successful rotation.
The incident demonstrated this brutally:
logs fill disk
|
v
logrotate needs system resources to run
|
X
filesystem already at ENOSPC
|
v
rotation mechanism unavailable
The safety mechanism failed because the condition it was meant to control had already removed its ability to act.
So capacity planning should leave enough reserve not only for normal workload but for the operation of recovery mechanisms themselves.
Graceful degradation would have changed everything
There are several places where the system could have degraded instead of cascading.
If functional DNS success had remained near zero for a sustained interval, WARP could have surfaced an overall degraded state.
If identical failures had been rate-limited, log growth could have remained bounded.
If callers had backed off more aggressively after immediate ENETUNREACH, the query storm could have been smaller.
If the filesystem had retained a protected reserve, logging and resolver state updates might have remained possible.
If WARP’s failure had automatically fallen back to native DNS/native IPv6 rather than maintaining a broken DNS dependency, the machine might have lost only outbound IPv4.
If SSH management had been insulated from optional network manipulation, the incident would have remained much easier to repair.
None of these requires a magical perfectly reliable component. They require the system to fail in narrower compartments.
The desired architecture is closer to:
optional IPv4 mechanism fails
|
v
IPv4 egress unavailable
BUT
native IPv6 remains
DNS remains
SSH remains
logging bounded
disk reserve protected
operator alerted
What actually happened was:
optional IPv4/DNS mechanism fails
|
v
DNS error storm
|
v
storage exhaustion
|
v
logging failure
database impairment
resolver degradation
network degradation
HTTP outage
SSH loss
The blast radius expanded far beyond the original capability that failed.
That is the engineering smell I would pay most attention to in future designs.
Health checks must measure the service people actually depend on
The phrase DNS proxy health status: Healthy may be the line from this incident that stays with me longest.
It illustrates a common systems mistake: measuring whether a component is alive rather than whether its purpose is being fulfilled.
These are different questions:
Is the DNS proxy process running? Is its socket open? Is the WARP tunnel established? Can a DNS query actually be resolved? Can an application use the result? Is the system as a whole still safe?
A lower-layer green result cannot automatically answer the questions above it.
The same principle applies far beyond WARP:
process running
!= service working
port listening
!= protocol usable
HTTP 200
!= application correct
database accepting TCP
!= queries succeeding
disk mounted
!= sufficient writable capacity
tunnel connected
!= all traffic paths functional
In this incident, I had several components that were individually “up” while the machine was progressively becoming unusable.
That is why system health needs cross-layer evidence.
Locally reasonable automation can create globally unreasonable behaviour
Perhaps the broadest engineering lesson is that no individual mechanism in the cascade needed to be absurd.
DNS failed, so callers retried.
A daemon encountered errors, so it logged them.
Linux tried to preserve those logs.
Logrotate attempted to manage them.
WARP reacted to route changes.
Services kept trying to perform their normal work.
Each mechanism can be defended locally.
Yet together:
reasonable local rule
+
reasonable local rule
+
reasonable local rule
+
finite resources
+
one persistent abnormal state
=
globally pathological behaviour
This is why I would not describe the incident simply as “WARP crashed.” WARP did not even crash in the ordinary sense. Parts of it remained alive, reported health, reacted to network state and kept doing work.
The more interesting failure was that the automated system entered a state in which normal responses to failure amplified the failure.
That is a much more general problem than VPN software.
Why the independent rescue plane mattered so much
If I had to choose one architectural feature that prevented this incident from becoming genuinely disastrous, it would not be Nginx, WordPress, systemd or WARP.
It would be the provider’s independent rescue environment.
The normal control path had disappeared:
Internet | v normal VPS network stack | X SSH unavailable
The rescue mechanism did not travel through that path:
provider control plane
|
v
rescue operating system
|
v
mount original virtual disk
|
v
repair persistent state
That is the kind of redundancy I trust most: not a second copy of the same mechanism, but a control path with different dependencies.
If two recovery mechanisms depend on the same DNS, firewall, filesystem and daemon stack, they may only provide the appearance of redundancy.
Out-of-band access deserves to be considered part of production architecture, especially on machines where a network-control service can manipulate routing and firewall state.
The epistemic discipline mattered as much as the commands
Looking back, the most satisfying part of the investigation was not any single shell command. It was watching several plausible explanations fail.
The first hypothesis was basically:
website down
|
v
small VPS
|
v
probably WordPress backups filled disk
Rescue mode disproved the storage source.
Then:
empty resolv.conf
|
v
WARP DNS failure
|
v
logs fill disk
was disproved by older ENOSPC and WARP evidence.
Then I placed the beginning around the surviving August 13 journal boundary.
Historical WARP statistics moved it to August 11.
Then “WARP tunnel failure” seemed plausible.
MASQUE health disproved that simplification.
The model kept becoming narrower because the evidence kept removing possibilities.
That process can be summarized as:
observation
|
v
hypothesis
|
v
search for discriminating evidence
|
v
hypothesis survives?
|
/ \
no yes
| |
v v
revise increase confidence
\ /
v v
search again
That is a very different mentality from finding the first plausible explanation and decorating it afterward with supporting logs.
It also forced me to keep several categories separate:
OBSERVED
TCP/22 reachable
sshd listening
no SSH banner
DNS success collapses
ENETUNREACH
disk full
resolv.conf zero bytes
DEMONSTRATED CAUSAL RELATION
WARP DNS failures generate warning storm
warning storm consumes substantial disk
filesystem reaches ENOSPC
ENOSPC disrupts log rotation and services
STRONGLY SUPPORTED INFERENCE
late WARP/DNS/storage degradation contributes
to terminal HTTP + SSH failure
UNRESOLVED
exact WARP internal transition at ~06:54
exact writer that left resolv.conf empty
exact packet/process layer that ended SSH
I think a technical postmortem becomes more credible, not less, when the last category is allowed to exist.
A note on the larger ideas this incident suggests
There are several directions I want to explore separately because this failure turned out to be more conceptually interesting than I expected.
One is the resemblance between reliable computing and biological homeostasis. A system can remain outwardly functional while its reserve is being consumed, then cross into decompensation when compensatory mechanisms can no longer maintain stability—or when the compensatory response itself begins causing damage. The progression from a local DNS fault to retry amplification, resource exhaustion and multi-subsystem failure makes that analogy unusually tempting.
Another is mathematical and cybernetic. The VPS can be viewed as moving through a state space, with feedback loops, finite resource constraints, stable regions, unstable regions and thresholds beyond which the dynamics change. The step from six healthy queries to hundreds and then thousands of failures is almost begging to be analysed that way.
A third is methodological: forensic debugging as causal reconstruction. The process was surprisingly close to a digital autopsy—preserve evidence, distinguish observations from interpretations, establish chronology, reject hypotheses that violate timestamps, quantify mechanisms, and stop exactly where the evidence stops.
And behind all of those sits a more philosophical question: at the physical level, the processor, memory cells and network hardware may all continue obeying their rules perfectly while the higher-level system becomes catastrophically “wrong.” Where, exactly, does failure exist in such a hierarchy of abstractions? That deserves its own essay rather than being squeezed into a VPS postmortem.
For this article, it is enough to notice that one ordinary infrastructure incident opened all four questions.
What I would change before trusting a similar VPS again
Concretely, I would make several changes before considering a server of this size well protected.
I would cap persistent journal consumption and ensure traditional syslog retention cannot occupy an unbounded fraction of root storage. I would alert on both absolute disk usage and its rate of growth. I would keep enough free-space reserve that rotation, package management, databases and emergency administrative operations can still write when something begins going wrong.
I would monitor real DNS resolution periodically rather than assuming that systemd-resolved or a DNS proxy being active means resolution works.
I would monitor a management-plane signal independently of the website—at minimum whether the host remains reachable in a way that predicts whether I can still repair it.
I would be cautious about allowing an optional connectivity client to control DNS, firewall and routing when I only need a narrow egress capability.
I would preserve access to an out-of-band rescue environment and make sure I know how to use it before the next incident, not during it.
I would also treat unusual repeated logs as an availability signal. A daemon generating 150,000 near-identical warnings per hour is not merely being verbose; on a constrained machine, it is consuming a finite safety reserve.
And after any severe incident, I would preserve the evidence before upgrading, reinstalling or “cleaning everything up.” A tidy system can be much harder to understand than a messy but intact crime scene.
The final causal model
After all the recovery work and all the revisions, this is the model I am willing to stand behind:
NORMAL OPERATION
WARP configuration works for months
DNS success ≈ 100%
|
v
INITIATING TRANSITION
2026-08-11 ~06:54 UTC
DNS-over-WARP loses usable route
|
v
Linux ENETUNREACH
|
| outer MASQUE remains healthy
v
FUNCTIONAL DNS FAILURE
success falls to ≈0%
|
v
RETRY / QUERY AMPLIFICATION
few queries -> thousands per 2 minutes
|
v
LOG AMPLIFICATION
near-per-query WARP WARN messages
~100k-150k warnings/hour
millions of errors
|
v
RESOURCE EXHAUSTION
multi-GB syslog/journal
small ~9 GB root filesystem fills
|
v
ENOSPC
|
+------------------+------------------+
| | |
v v v
logrotate fails WARP rotation fails service writes fail
including MariaDB
\ | /
\ | /
+----------------+----------------+
|
v
PROGRESSIVE SYSTEM DEGRADATION
logging impaired
observability impaired
resolver/network state increasingly fragile
|
v
LATE TRANSITION
2026-08-16 16:37 UTC
WARP state/route activity
IPv4 path unusable
/etc/resolv.conf becomes 0 bytes
|
v
TERMINAL COLLAPSE
2026-08-16 16:43 UTC
Cloudflare HTTP 522
+
SSH management unavailable
|
v
OUT-OF-BAND RECOVERY
provider rescue environment
|
+--> mount original root filesystem
+--> free ~2.7 GB
+--> restore resolv.conf
+--> mask WARP
|
v
NORMAL BOOT RECOVERED
SSH active
DNS native and functional
WARP inactive
website reachable
root filesystem ~70% used
Within that model, my confidence is deliberately uneven.
The August 11 DNS transition is demonstrated. The ENETUNREACH failure mode is demonstrated. The massive DNS-warning amplification is demonstrated. The connection between that warning volume and multi-gigabyte system logging is extremely strong. ENOSPC, failed rotation and secondary service failures are directly observed.
The later resolver/network collapse is strongly supported by timing and state evidence, but the exact operation that produced the zero-byte resolver file is not captured. The exact mechanism that prevented SSH from completing its protocol exchange is also not isolated.
And the microscopic trigger inside WARP around 06:54 on August 11 remains the one important unanswered implementation question.
That is as far as I can responsibly go.
What began as a full disk was really a systems failure
I started this incident expecting one of the least interesting jobs in server administration: SSH into a small VPS, delete some oversized WordPress backups, and move on.
Instead, SSH itself disappeared.
Rescue mode showed a full disk, but WordPress was not the main consumer. Logs were. The logs led to WARP. WARP led to millions of DNS failures. The DNS failures led backward through timestamps until the apparent August 16 outage became an August 11 event. Then the tunnel that seemed “up” turned out to contain a DNS path that was effectively dead. A health indicator remained green while functional success approached zero. The warning mechanism consumed the filesystem. The filesystem’s exhaustion disabled the mechanism intended to contain the logs. Later, resolver and networking state deteriorated far enough that both the public application path and the administrative path disappeared.
What looked at first like:
disk full
was really:
a local routing failure
|
v
a retry problem
|
v
an observability problem
|
v
a resource-exhaustion problem
|
v
a recovery-mechanism failure
|
v
a resolver/network problem
|
v
an application outage
+
a management-plane outage
That is why I found this incident so much more interesting than the ordinary failure I had expected.
No single component needed to “decide” to destroy the VPS. There was no intelligent agent, no malicious process and no dramatic kernel crash. Every layer continued following ordinary rules. The catastrophe emerged from their interaction.
And perhaps the most useful lesson is precisely that: a system does not need to stop executing correctly at the lowest level in order to behave catastrophically at the highest level.
The CPU can keep executing instructions. The kernel can keep scheduling processes. The tunnel can keep reporting one-millisecond latency. The DNS proxy can remain alive. sshd can be listening. Nginx can have started successfully. Every one of those facts can be true while the system as a whole is already moving toward failure.
Reliability therefore cannot be reduced to asking whether the individual pieces are still running. It depends on whether the relationships between those pieces remain inside a region where the whole machine can recover from disturbance without consuming its own capacity to recover.
That is the part I did not expect to learn from a 9 GB WordPress VPS.
I went in looking for something to delete.
I came out thinking about feedback, state, evidence, failure containment, control planes, homeostasis, and the strange boundary between a machine whose components are still working and a system that is no longer healthy.