Incident Report: Anomalous Sign-Ups on July 25th and 27th
Two major waves of false membership registration requests were made on July 25th and 27th. These requests were made directly to the API1. The requests originated from American, Russian, and Ukrainian IP addresses2.
The impact of the incident was low despite approximately 100,000 “members” being added to our database. Our response was swift enough to prevent the attack doing significant damage, although we did not notice until the morning of the 27th. No important data was lost, and we were able to quickly roll back the database to a backup taken on the 25th of July prior to the incident. We do not believe any significant data was lost via this rollback, however, backups of the database taken on the 26th and 27th of July have been saved.
Member data (including PII, etc) is safe. There is no indication that any breach occurred or that sensitive information was unintentionally released.
|Date - AEST (UTC+1000)||Event|
|2020-07-25 19:53:23||Spammer does a first test registration. They initially attempted to put HTML in the payload (tags:
|2020-07-25 20:00:06 to 21:01:02||Spammer begins sending 500-1500 requests per minute for approx 1hr|
|2020-07-25 20:00:23||First log entry indicating per-second sending rate exceeded (approx 6k emails failed to send for this reason, an estimated 2/3 are notification emails)|
|2020-07-25 22:02:49||First mention of exceeding daily email-sending quota (approx 44k emails failed to send for this reason, an estimated 2/3 are notification emails)|
|2020-07-26 01:25:51||Spammer resumes, having changed the encoding of email addresses which prevents emails being sent to the designated addresses, though notifications are still sent (approx 22k emails failed to send for this reason, none are notification emails)|
|2020-07-26 01:58:58||Spammer halts|
|2020-07-27 03:49:06||Spammer tests a registration (presumably manually)|
|2020-07-27 03:57:43||Spammer resumes using badly encoded email addresses. The notification emails do not trigger Gmail’s spam filter and notifications are delivered unlike the previous waves.|
|2020-07-27 04:02:37||Last log message; MK terminates servers|
|2020-07-27 08:56:07||MK patches API to disable registration and brings API back online|
|2020-08-02 19:11:00||MK finishes integrating ReCAPTCHAv3 to the sign-up form and API. Registration live once again.|
We have received no complaints with regards to emails sent during this period, likely due to the vast majority of emails bound for spam recipients failing to send. This is good as it means we are unlikely to risk a disruption to emails from the API.
We had approximately half-a-dozen people attempt to sign up as members to the party between July 27th and August 2nd. Due to the method of disabling registration these people were not notified before they attempted to sign up (no indication was given on the sign up form prior to submitting it) and the details of their submission were not saved so we have no way to contact them. This is unfortunate, but not a big deal in the scheme of things.
# Follow up actions
The primary follow up action is to document things we’ve learnt, actions and improvements to make following the incident, and the completion of this report. For the sake of time management, this report is being published as it is.
There are a number of lessons to take away from this incident.
For example: we have 2 mechanisms in place to notify us of issues with the API and both failed.
The first is performed by our logging system based on detection of the words “panic”, “exception”, and “error”. This mechanism failed due to syntax errors in the search query (which have now been fixed).
The second mechanism is performed by the API itself when certain methods throw exceptions. When the API detects such mechanisms it will send an email with the details of the exception. As we had exceeded our email quota these emails were not sent. This can be solved by using a method of sending emails which will not be affected if this happens again. The fix is not currently implemented, but should be quick to do.
- That the requests were made directly to the API was determined via inspection of payloads.
- It’s not uncommon for spam to originate from Russian or Ukrainian IP addresses. It’s unlikely this had any political motive; the payloads appear to be attempting to advertise online casinos and the like.
Comments powered by Talkyard.