tag:status.abusix.com,2005:/historyAbusix Status - Incident History2024-03-28T12:51:42ZAbusixtag:status.abusix.com,2005:Incident/197969892024-01-23T09:45:14Z2024-01-23T09:45:14ZAWS Route53 Query Issues (SERVFAIL Errors)<p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>09:45</var> UTC</small><br><strong>Resolved</strong> - No further issues reported, so switching this to Resolved/Operational.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>19:51</var> UTC</small><br><strong>Monitoring</strong> - This appears to have been caused by Amazon Route53 sending some unexpected responses during a migration. We're not seeing any issues with any public DNS servers now, but if you're still having issues, please flush the DNS cache of your nameserver and that should correct it. Please contact support@abusix.com if you are experiencing issues after that.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>18:17</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing issues with AWS Route53, resulting in SERVFAIL errors for certain customers' queries.</p>tag:status.abusix.com,2005:Incident/195599222023-12-30T06:53:16Z2023-12-30T06:53:16ZAbuseHQ Inbound Outage<p><small>Dec <var data-var='date'>30</var>, <var data-var='time'>06:53</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>29</var>, <var data-var='time'>19:56</var> UTC</small><br><strong>Monitoring</strong> - All systems are back up, and we are still monitoring the situation. <br />Some of the core systems for the data processing chain had connectivity issues within the AWS data center in the EU. That way, the processing chain only worked sporadically, leading to a queue size build-up. The issues seem to be resolved right now, and we are in contact with AWS to see if we can get further details on the exact problem. <br />We apologize for the inconvenience and keep you posted in case we get further information.</p><p><small>Dec <var data-var='date'>29</var>, <var data-var='time'>17:49</var> UTC</small><br><strong>Investigating</strong> - Earlier today the Mail & Event Processing of some AbuseHQ customers slowed down and hours later came to a halt. We are investigating the issue right now.</p>tag:status.abusix.com,2005:Incident/191957332023-11-22T17:44:43Z2023-11-22T17:44:43ZSome AHQ services are currently experiencing a decline in performance.<p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>17:44</var> UTC</small><br><strong>Resolved</strong> - We experienced temporary performance issues, but have resolved the issue and all services are back to normal. Our team is actively monitoring and implementing measures to prevent future incidents. We appreciate your understanding and remain committed to delivering a seamless experience.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>15:36</var> UTC</small><br><strong>Investigating</strong> - Experiencing performance issues in certain AHQ services. Our team is actively addressing the issue to restore optimal functionality.</p>tag:status.abusix.com,2005:Incident/191871762023-11-21T14:05:51Z2023-11-21T14:05:51ZSome AHQ services are currently experiencing a decline in performance.<p><small>Nov <var data-var='date'>21</var>, <var data-var='time'>14:05</var> UTC</small><br><strong>Resolved</strong> - The performance issue in AHQ services has been resolved, and all systems are back to normal. Thank you for your understanding and patience. If you encounter any further issues, please let us know. We appreciate your cooperation!</p><p><small>Nov <var data-var='date'>21</var>, <var data-var='time'>13:05</var> UTC</small><br><strong>Investigating</strong> - Experiencing performance issues in certain AHQ services. Our team is actively addressing the issue to restore optimal functionality.</p>tag:status.abusix.com,2005:Incident/185100352023-09-15T12:34:36Z2023-09-15T12:34:36ZFailing logins on app.abusix.com<p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>12:34</var> UTC</small><br><strong>Resolved</strong> - One of our services showed unexpected behaviour and was exceeding memory. We applied measures to fix the issue, and everyone should be able to log in again. We will investigate the root cause of this as a next step. Sorry for any inconvenience this caused.</p><p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>11:19</var> UTC</small><br><strong>Investigating</strong> - Our alerting indicates issues with our authentication system. This affects app.abusix.com as well as AbuseHQ customers. We will provide further updates with the ongoing investigation.</p>tag:status.abusix.com,2005:Incident/181039082023-08-10T16:49:55Z2023-08-10T16:49:55ZInfrastructure partly down<p><small>Aug <var data-var='date'>10</var>, <var data-var='time'>16:49</var> UTC</small><br><strong>Resolved</strong> - Our systems are now running however due to problems with one of our third party vendors we are in degraded performance. We are currently working through a high backlog of data and processing times will be longer than normal. We are investigating the causes behind the infrastructure breakdown. And we will review our partnership with the vendor in question in the coming weeks.</p><p><small>Aug <var data-var='date'>10</var>, <var data-var='time'>15:26</var> UTC</small><br><strong>Identified</strong> - AMI and ATI up and running again. AHQ is back soon.</p><p><small>Aug <var data-var='date'>10</var>, <var data-var='time'>14:30</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Aug <var data-var='date'>10</var>, <var data-var='time'>09:46</var> UTC</small><br><strong>Investigating</strong> - Parts of our infrastructure are down at the moment. Unfortunately, this affects all of our systems. As soon as we know more we will let you know.</p>tag:status.abusix.com,2005:Incident/174319142023-06-01T11:00:12Z2023-06-01T11:00:12ZMaintenance on Listing Reports<p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>11:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>10:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>31</var>, <var data-var='time'>13:21</var> UTC</small><br><strong>Scheduled</strong> - Due to maintenance of a backend service, listing information in the `Network Management`section on app.abusix.com will be out of date or unavailable. Regular Lookup and Delist functionality is not affected.</p>tag:status.abusix.com,2005:Incident/167464562023-04-04T11:00:25Z2023-04-04T11:00:25ZInternal Maintenance<p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>11:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>09:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>08:01</var> UTC</small><br><strong>Scheduled</strong> - We are upgrading some internal services. Please stand by it won't take long.</p>tag:status.abusix.com,2005:Incident/166592472023-03-27T10:59:43Z2023-03-27T10:59:43ZDelisting<p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>10:59</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>07:06</var> UTC</small><br><strong>Investigating</strong> - Delists are not currently working, we are investigating the cause of this</p>tag:status.abusix.com,2005:Incident/164781232023-03-15T07:59:08Z2023-03-15T07:59:08ZData Inbound Processing - Partial Outage<p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>07:59</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>18:55</var> UTC</small><br><strong>Update</strong> - The issue has been resolved. All queues are back to normal since this morning.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>18:19</var> UTC</small><br><strong>Monitoring</strong> - We found a workaround and will implement a fix tomorrow. We'll keep monitoring the data queues but for now all data that was queued as well as new data is being processed again. We had to temporarily disable the 'report unparsed' feature, sorry if this causes any inconvenience.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>17:24</var> UTC</small><br><strong>Identified</strong> - We have identified a possible issue with a secondary service and are working on a fix.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>16:34</var> UTC</small><br><strong>Investigating</strong> - We're investigating an issue with our data inbound routing affecting data inbound for certain AbuseHQ customers. We'll provide updates as soon as we have more information. Sorry for any inconvenience this causes.</p>tag:status.abusix.com,2005:Incident/164915172023-03-14T11:10:20Z2023-03-14T11:10:20ZScheduled Maintenance<p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>11:10</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>10:10</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>10:09</var> UTC</small><br><strong>Scheduled</strong> - We will be undertaking some routine maintenance on the Portal API. We're not expecting any customer impact but you might experience login problems.</p>tag:status.abusix.com,2005:Incident/159757662023-02-02T11:01:58Z2023-02-02T11:01:58ZScheduled maintenance - database upgrade (North America)<p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>11:01</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>09:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>16:17</var> UTC</small><br><strong>Scheduled</strong> - We will be upgrading a central database of AbuseHQ in our North America region. In this avalibility region, AbuseHQ Events UI will not be available for customers as well as AbuseHQ inbound data processing will experience delays for this group of customers.</p>tag:status.abusix.com,2005:Incident/158743392023-01-23T14:00:40Z2023-01-23T14:00:40ZScheduled maintenance - database upgrade<p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>14:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>10:56</var> UTC</small><br><strong>Update</strong> - The database upgrade takes longer than expected due to storage issues with our cloud provider. We're extending the maintenance window by 3 hours until 2pm UTC (3pm CET).</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>08:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>16:50</var> UTC</small><br><strong>Scheduled</strong> - We will be upgrading a central database of AbuseHQ. AbuseHQ Events UI will not be available during that time as well as AbuseHQ inbound data processing will experience noticeable delays.</p>tag:status.abusix.com,2005:Incident/158900512023-01-20T13:54:20Z2023-01-20T13:54:20ZAbuseHQ: Inbound data processing queues<p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>13:54</var> UTC</small><br><strong>Resolved</strong> - The queues have been processed, and inbound processing is not experiencing delays anymore.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>11:06</var> UTC</small><br><strong>Update</strong> - We need to extend the time window for possible delays in inbound processing by 2 hours until 2 pm CET.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>07:59</var> UTC</small><br><strong>Monitoring</strong> - Due to a high amount of events in the past 24 hours, inbound processing might experience delays for the upcoming 4 hours. We will monitor the system and provide an update as soon as the situation normalizes. The scheduled maintenance on Monday will tackle this amongst other things. Sorry for any inconvenience this is causing.</p>tag:status.abusix.com,2005:Incident/141194842022-12-02T10:02:22Z2022-12-02T10:02:22ZAbusix Mail Intelligence<p><small>Dec <var data-var='date'> 2</var>, <var data-var='time'>10:02</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>23:20</var> UTC</small><br><strong>Investigating</strong> - We have a suspected hardware failure, listing and delisting are unavailable until this can be fixed.</p>tag:status.abusix.com,2005:Incident/136352112022-11-22T15:32:33Z2022-11-22T15:32:33ZShort inbound processing maintenance<p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>15:32</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>15:15</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>15:03</var> UTC</small><br><strong>Scheduled</strong> - We need to replace a cache server. We will scale down inbound processing which will delay processing of reports. They will be queued, no data will be lost. We hope that it will be a quick replacement of less than 30 minutes.</p>tag:status.abusix.com,2005:Incident/127755702022-11-10T15:14:45Z2022-11-10T15:14:45ZAbuseHQ Inbound Processing Delays<p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>15:14</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>11:53</var> UTC</small><br><strong>Identified</strong> - Inbound Processing in AbuseHQ is being delayed by database performance issues. We have identified the culprit and processing should now start to catch up.</p>tag:status.abusix.com,2005:Incident/124247672022-10-21T10:33:50Z2022-10-21T10:33:50ZAbusix Mail Intelligence<p><small>Oct <var data-var='date'>21</var>, <var data-var='time'>10:33</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>21</var>, <var data-var='time'>09:49</var> UTC</small><br><strong>Identified</strong> - Due to a Kubernetes issue, the database backend is currently unavailable which means that listing and delisting is presently unavailable. This does not affect RSYNC or DNS lookups, however the data currently being served via both will be stale.</p>tag:status.abusix.com,2005:Incident/113419962022-10-04T11:34:15Z2022-10-04T11:34:15ZPartial Portal Outage<p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>11:34</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>04:23</var> UTC</small><br><strong>Monitoring</strong> - Our monitoring shows that the networking issue was fixed by one of the cloud providers at around 10pm UTC yesterday.<br />The problems should be gone, we'll continue monitoring it and decide further steps.</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>17:29</var> UTC</small><br><strong>Identified</strong> - After in-depth analysis we have identified a networking issue between two cloud providers that are in use. We are in contact with both of them to help them identify the exact location of the problem and to bring our services back up to full stability as quickly as possible.<br />We also tried to find workarounds for the problem, but didn't succeed so far. <br />The symptoms are: A low percentage of calls to app.abusix.com are failing. Until it's fixed please try to reload pages/resend requests when you experience issues.<br />Background processing for AbuseHQ and AMI is not impacted by this outage.<br />Please let us know if you have further questions.</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>18:19</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.abusix.com,2005:Incident/107610802022-08-04T18:50:02Z2022-08-08T16:40:02ZIssue with AbuseHQ instance settings<p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>18:50</var> UTC</small><br><strong>Resolved</strong> - We restored the settings of affected AbuseHQ customers and started inbound data processing again. We will follow up with a post-mortem for this incident soon. If you notice any unexpected behavior within your AbuseHQ instance, please reach out via intercom!</p><p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>16:31</var> UTC</small><br><strong>Identified</strong> - We noticed an issue with AbuseHQ instance settings after adding a node to one of our database system. We are working on a fix and have stopped data processing until the settings are restored.</p>tag:status.abusix.com,2005:Incident/106997092022-07-27T12:40:24Z2022-07-27T12:40:24ZHigh load on our main AbuseHQ databases<p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>12:40</var> UTC</small><br><strong>Resolved</strong> - The issue has been resolved. There are some remaining queues for customers that receive huge batch reports over night, but that's nothing unusual and will be worked through in the next hours. <br />Sorry for the inconveniences this performance degradation might have caused.</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>11:02</var> UTC</small><br><strong>Monitoring</strong> - The database node has recovered and we are ramping up backend services to full speed again, while monitoring the performance. Some queues have built up over the last hours so we need to play catch up a bit. Frontend and API performance should be back to normal.</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>10:11</var> UTC</small><br><strong>Update</strong> - We are still investigating the issue. One of our database nodes is recovering and it takes some time to sync all data.</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>09:12</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.abusix.com,2005:Incident/106992732022-07-27T10:42:06Z2022-07-27T10:42:06ZIncorrect listings on Exploit and AuthBL lists<p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>10:42</var> UTC</small><br><strong>Resolved</strong> - All incorrect records have been purged now.</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>07:48</var> UTC</small><br><strong>Investigating</strong> - Overnight we received some bad data from a 3rd party supplier which has caused valid hosts to be incorrectly listed in the "exploit" and "authbl" lists. We're working on a fix and should have this deployed soon.</p>tag:status.abusix.com,2005:Incident/103936302022-06-24T09:58:28Z2022-06-24T09:58:28ZAbusix Mail Intelligence<p><small>Jun <var data-var='date'>24</var>, <var data-var='time'>09:58</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>24</var>, <var data-var='time'>09:57</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Jun <var data-var='date'>24</var>, <var data-var='time'>08:31</var> UTC</small><br><strong>Identified</strong> - We're currently experiencing a high error rate on one of our backend clusters, this is causing some delist requests to fail along with some listings to fail and some slowness in generating zone files.</p>tag:status.abusix.com,2005:Incident/103664842022-06-21T08:29:03Z2022-06-21T08:29:03ZCustomer portal outage<p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>08:29</var> UTC</small><br><strong>Resolved</strong> - The two providers report that they resolved the issue. We apologize for any inconvenience this has caused you.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>07:35</var> UTC</small><br><strong>Update</strong> - We still see errors from our authentication provider Auth0. This causes errors for users logging in to AbuseHQ and app.abusix.com.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>07:22</var> UTC</small><br><strong>Monitoring</strong> - Our services have recovered. Two of our external providers Auth0 https://status.auth0.com/ and Cloudflare https://www.cloudflarestatus.com/ report incidents with their systems. This seems to be the origin of this incident. We will investigate how we can further improve the mitigation of such external outages.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>07:04</var> UTC</small><br><strong>Investigating</strong> - We're investigating an outage with our customer portal app.abusix.com. This also affects access to the AbuseHQ UI unfortunately. It seems like an external provider is experiencing an outage. We will update this incident as soon as we have more information. Sorry for any inconvenience this is causing.</p>tag:status.abusix.com,2005:Incident/99586832022-05-10T15:57:19Z2022-05-10T15:57:19ZAbusix Mail Intelligence Backend<p><small>May <var data-var='date'>10</var>, <var data-var='time'>15:57</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>15:56</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>14:24</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing an issue on the backend of Abusix Mail Intelligence, this is causing some lookup slowness, delists are unavailable at this time and no zone builds are currently being run.</p>