Keep Your Head on a Swivel: How Serbian Hackers Hijacked my Site

Being a one-man administrative team for my digital presence has its perks. There is no bureaucratic logjam (aside from the Wife Acceptance Factor) when new stuff needs to be delivered. Nobody argues with me about what Linux flavor is the best, or whether I should (god forbid) switch over to IIS. But this comes with downsides: I'm one man, with one pair of eyes and hands, and when stuff breaks down, it breaks invisibly and can stay broken for a long time.

On April 21 around 4:25 UTC, I discovered that a Serbian hacker (or group thereof) had defaced the front of this site. Anyone with an inkling of web knowledge can tell that it runs Ghost, and the npm packages, 18 months out of date, were the assumed attack vector. The attackers erased all my posts and pictures, but were blessedly contained to the website as it was hosted: my DigitalOcean login, and the box itself appeared to have been untouched. What appeared on the front of the site (to the best of my memory) was:

Check out our forum at xyz.rs!

I was a bit panicked at first. Dot R S! Russian Hacker! The Kremlin is onto me! (Not realizing that the Russian extension is .ru). But then it occurred to me: I'm a halfway decent devOps engineer. I've factored in microservices. This blog doesn't run on the only computer I own; it's cattle, not a pet.

So without further trouble, I shot the cow.

DigitalOcean might not be the premier business cloud solution suite, but they're fantastic for the solo, at-home or small business user. I pulled a backup of the droplet this site runs on from several months ago (shortly after my most recent post) and spun it back up. Since SSH revealed that my last login had occurred sometime around then (and the server itself wasn't compromised, just the blog software), I was able to start to remediate the issue. There was simply no need: a clean solution was going to be as greenfield as it got.

First order of business was getting everything up-to-date: NPM packages, Linux upgrades, the kitchen sink. We also made sure to check the Ghost user table and API keys. Only me. Great. Ghost was upgraded from 5.96.1 to 5.130.6. I rotated my credentials to a long, hard-to-guess key. 2FA login was added with a bit of help from another cloud service, and enforced against every user (again, only me).

So what went well here? Restoring from the backup was fast and easy. Kudos to DigitalOCean. The poorer end was that I realized that I have no monitoring, alerting, or auto-upgrading in place. These enhancements still have yet to be set up, and that's going to be a job for future Jack to take care of, hopefully before the defacers come back around to poke me in the eye again.

And as for the future? I might migrate this blog to an on-prem solution in my at-home cluster. That will have its own issues: no lovely DO GUI to easily pick everything up. I'll go from paying my cloud provider to deal with that nonsense to really being a one-man army. So that's something to consider.

At the end of the day, the blog works. Which is great. It's part of my online portfolio, and I'd like potential employers, peers, colleagues, and friends to know that I am A Serious Tech Guy ™ who likes his stuff to work properly and with ease. The downside (and this was always true, and cannot be controlled) is that the internet is a place where nobody knows if you're a dog. My advice is simple and curt: keep doing what you're doing. You will come back with skinned knees and lessons learnt.