Hi, everyone. I'm going to cut right to the chase.
blargbot was killed.
As of April 3rd 2021, 23:57 UTC, a malicious actor gained access to my RethinkDB database, and deleted it. Every single table is gone. I tried doing a disk recovery, but it failed. I tried looking for backups, but couldn't find any. There were no snapshots or anything else that could be recovered.
What was lost?
Instead of breaking down what got lost, it would be quicker to state what didn't.
- Chat logs, as they're stored in a separate Scylla database
- Variables, as they're stored in a separate PostgreSQL database
Additionally, I have been able to restore public tags from state logging, however no metadata (usages, favourites, etc.) has been retained.
Unless backed up to variables or some external storage, all custom commands have been lost. All server configuration has been lost. All user information has been lost.
How did this happen?
The actor gained access to the database using an exposed cluster port in RethinkDB, which apparently gives full unauthenticated access to the entire database. The wipe was clean; RethinkDB had logs of each individual table being deleted. I am confident than the actor did not gain access to the server itself.
There are several things that I am directly responsible for:
- Ports were exposed. This meant that anyone could hit any port, given they could access my server's IP.
- RethinkDB's IP bindings were set to 'all'. This meant that anyone from anywhere was able to connect to my RethinkDB instance. The reason this was set was because I was under the impression that it would only bind the driver port, which does have authentication, but instead it also exposed the unauthenticated cluster port. Note that the behaviour of this configuration made no mention of it impacting the cluster port.
- Backups weren't being made. There's really no excuse for this, I should have been making backups. But unfortunately, I didn't. The only backup I could find was from 2017, which likely wouldn't even load due to how many changes blargbot has gone through since.
What's going to happen?
This is the tough one. I don't think it's any secret that I've been incredibly burnt out from this project for a long time now. I simply don't have the time or the motivation. And something like this happening only twists the knife further.
But at the same time, I know how much this bot means to all of you, and I don't want to just give up and throw it away. So it's back up in this reset state. Most data might be lost, but we still have variables. And for those of you who back up their commands, that might be enough to get a rocky start back up.
I have already increased the security of my server to prevent anything else from happening. All ports have been restricted, all SSH access keys replaced. I'll be making sure that an incident like this should never happen again.
Going forward, I have plans to
- Make regular backups, for worst case scenarios
- Move away from RethinkDB, as it's caused me nothing but grief since I started using it
- Add extra utilities to help users back up their servers, including retrieving active variables, shrinkwrapping an entire server, etc.
There's really nothing else I can do right now but apologize. I'm sorry. Due to my negligence, your hard work and time spent using and setting up my bot has been lost. Nothing I can say or do will ever make up for that.
Going forward, I'm going to be taking extra caution. I'm going to be doing regular, tangible backups. I'm going to be tightening security.
But there's no way I can ask for your forgiveness, or ask for your trust again.
I can't blame you if you walk away. I can't blame you if you remove blargbot and never think about it again.
The past 5 years have been a wild ride, full of joy, pain, triumphs, mistakes, and regrets. And I can't thank any of you enough for the amount of support I've been given.
So, thank you. And goodbye for now.