Drive Failure 2: Electric Boogaloo
Whelp! It happened again…
Just as I was getting ready for bed, I start getting blasted with notifications that my plex server is going haywire. All services on the machine were reporting as down.
Uptime Kuma, Healthchecks, ntfy… every alarm I set up to make me feel like a responsible sysadmin went off at once. Right around midnight the whole box just dropped off the network. No SSH, no Plex, no nothing.
Suspect #1: the 10 gig card
Once I got a screen on it, the logs told the story. For days the Aquantia 10GbE card had been quietly spamming PCIe errors (AER, if you want to sound smart at parties). I never noticed because everything kept working… until it didn’t. The card finally locked up hard, file copies to the NAS hung, and the server froze with them.
So the fancy 10 gig card got yanked. The server is back on the boring built-in 1 gig port. To keep the router from handing it a new IP (and breaking every single thing that points at it), I gave the onboard port the old card’s MAC address. Identity theft, but for network cards.
Suspect #2: the SSD. Again.
Here’s where the title comes in. After the reboot, my 4 TB WD Blue SA510, the SSD holding all of Plex’s data, was just… gone. The system could see that something was plugged in, but the drive never answered when asked who it was.
I tried the “power it up and leave it alone” trick that sometimes revives a confused SSD. Nothing. This drive has now died twice, and I’m not giving it a third chance.
The best part: with the drive missing, Plex booted up as a brand new, completely empty server, like nothing ever happened. Very cool. Very calming at midnight.
Backups to the rescue
This is the part where past me finally did something right. Every night the server backs itself up to the NAS, and the NAS mirrors that offsite to Backblaze. So instead of rebuilding years of watch history and settings by hand, the recovery happened in two rounds.
Round 1, before bed: get the server booting safely again without the dead drive, move Plex over to the server’s main NVMe drive, and restore the Plex database and settings from the Nov 3 and Nov 4 backups. Watch history re-synced, everything looked good, lights out.
Round 2, the morning after: turns out “looked good” was doing a lot of heavy lifting. The metadata wasn’t up to date, and Anime was completely broken. The culprit? My last full metadata backup was from Nov 1, before I switched Anime over to the HAMA agent, and I never took a fresh one after the switch. So the restore put back metadata for a setup that didn’t exist anymore. The fix was making Plex throw it out and refetch the metadata for everything from scratch. A long refresh later, all good.
Lesson learned the hard way: as soon as the refresh finished, I kicked off a fresh full backup. The next drive that dies gets a restore with the right metadata.
The only thing I left out on purpose is the video preview thumbnails (the little pictures when you scrub through a video). They take about 280 GB and the main drive only has about 150 GB free, so they’re turned off for now.
And the users? Barely any trouble. A few hiccups while Anime was broken, and after the restore, nobody was any the wiser.
What’s next
A Samsung 870 EVO is on the way, plus a fresh SATA cable, because I’m no longer ruling anything out. When it arrives it gets a full write-and-verify test before any data goes near it, then Plex moves over and the thumbnails come back.
Lessons learned:
- Monitoring is only useful if you actually look at it. Those PCIe errors were sitting in the logs for days.
- Backups you’ve never restored are just hopes. These ones actually worked, and I’ve never been happier to see a progress bar.
- Never ever buy a WD Blue SA150 drive if you value your data!
Momma still ain’t raise no bitch. But she did raise someone who now checks his logs.