All queues are down to 0. The inbound components have caught up with the remaining queues.
Posted Apr 24, 2021 - 15:19 UTC
Identified
We have scaled up our backend infrastructure as we identified a lack of resources for specific parts of the inbound system. Currently we are monitoring the effects of the change. So far it's looking like processing is pretty much back to normal, except that some customers have big queues in front of their API resolver and in front of the case creation/assignment component. We will continue monitoring these queues (currently the size is decreasing slowly but surely).
The other reason, apart from the shortage of processing power, is also an big increase in incoming reports + an increase in internal reports coming in for a subset of customers.
Posted Apr 23, 2021 - 11:58 UTC
Update
We are continuing to investigate this issue.
Posted Apr 23, 2021 - 06:24 UTC
Investigating
Currently events are queuing for some customers. We have seen a bigger amount of reports/events come in overall for the last few days and need to investigate where the main bottlenecks are that lead to these unusual queues.
Posted Apr 23, 2021 - 05:58 UTC
This incident affected: AbuseHQ (Mail & Event Processing).