Last week on Thursday (4th of August) there was an incident on our backend side that led to configuration values being reverted and some events being processed in a different way for some of our AbuseHQ customers. This postmortem will give insights into what happened and what we did to fix it as well as what we are doing so it doesn't happen again.
On Thursday we added another database node to our database cluster to prepare for more load. During the sync between the existing servers and the new one, there was a failure of one of the existing servers that led to connection problems between our backend services and the database. This happened at ~12:25 (pm) UTC. These connection problems were only happening for a few minutes and, by themselves, didn't break anything. On the backend side, there was a bit of code, that is usually rarely called, that makes sure that new abusehq instances are properly initialized. Due to faulty error handling, this code overwrote the settings of some existing customers with the default settings. These settings were the inbound processing configuration, whether repshare is turned on and the default transitions of the default playbook. Nothing else was changed. The events that were processed after the incident until we fixed it, didn't go through the same sort of filtering, resolving, etc. that they usually go through.
For affected customers, new cases might've been created based on subscribers with the event IP as the subscriber ID as this is the default setting for a new AbuseHQ instance. Also, events that were usually dropped, might not have been dropped, because the filters weren't active.
It took us a bit to find out about the issue because it wasn't an apparent system error. Events were processed, so the monitoring didn't complain. After seeing anomalies in our processing statistics we started looking into it, though.
We apologize for any errors the incident has caused on your side. If you notice anything weird on your instance, we will gladly find a solution with you to clean that up.