Over night the system caught up with all queues. Everything is back to normal.
Posted Jun 02, 2021 - 06:30 UTC
Monitoring
The fix was successfully tested and rolled out to our production workers. The data pipeline will now catch up with the events that have queued up. That will take some time. We will monitor this over the next hours and give another update tomorrow.
Posted Jun 01, 2021 - 18:11 UTC
Update
A hotfix has been created. It is currently running through the usual automated testing pipelines and will then be tested on our dev environment. If successful, the fix will be rolled out to production.
Posted Jun 01, 2021 - 17:23 UTC
Identified
We have identified the issue. There is information parsed from specific shadowserver events that clashes with some internal keys we have. When we adapt between our new, enhanced internal event schema to the current incident schema, that piece of information messes up the resulting document and then leads to issues when sending them to the case assignment API.
We will now work on a hotfix and improve test coverage for situations like these as well.
Posted Jun 01, 2021 - 16:43 UTC
Investigating
Today we started seeing queues that built up in front of the case assignment (adding events to an existing case or creating a new one). We are guessing this is related to changes to the shadowserver parser that were rolled out. We are investigating this issue. Thanks for your patience.
Posted Jun 01, 2021 - 15:49 UTC
This incident affected: AbuseHQ (Mail & Event Processing).