Incident Report: Anomalous Sign-Ups on July 25th and 27th

Author: Max Kaye

# Events

Two major waves of false membership registration requests were made on July 25th and 27th. These requests were made directly to the API1. The requests originated from American, Russian, and Ukrainian IP addresses2.

The impact of the incident was low despite approximately 100,000 “members” being added to our database. Our response was swift enough to prevent the attack doing significant damage, although we did not notice until the morning of the 27th. No important data was lost, and we were able to quickly roll back the database to a backup taken on the 25th of July prior to the incident. We do not believe any significant data was lost via this rollback, however, backups of the database taken on the 26th and 27th of July have been saved.

Member data (including PII, etc) is safe. There is no indication that any breach occurred or that sensitive information was unintentionally released.

# Timeline

Date - AEST (UTC+1000) Event
2020-07-25 19:53:23 Spammer does a first test registration. They initially attempted to put HTML in the payload (tags: a, img, center, and br), however, we sent plaintext emails so this could never have worked. The spammer does two more test registrations to confirm they are working.
2020-07-25 20:00:06 to 21:01:02 Spammer begins sending 500-1500 requests per minute for approx 1hr
2020-07-25 20:00:23 First log entry indicating per-second sending rate exceeded (approx 6k emails failed to send for this reason, an estimated 2/3 are notification emails)
2020-07-25 22:02:49 First mention of exceeding daily email-sending quota (approx 44k emails failed to send for this reason, an estimated 2/3 are notification emails)
2020-07-26 01:25:51 Spammer resumes, having changed the encoding of email addresses which prevents emails being sent to the designated addresses, though notifications are still sent (approx 22k emails failed to send for this reason, none are notification emails)
2020-07-26 01:58:58 Spammer halts
2020-07-27 03:49:06 Spammer tests a registration (presumably manually)
2020-07-27 03:57:43 Spammer resumes using badly encoded email addresses. The notification emails do not trigger Gmail’s spam filter and notifications are delivered unlike the previous waves.
2020-07-27 04:02:37 Last log message; MK terminates servers
2020-07-27 08:56:07 MK patches API to disable registration and brings API back online
2020-08-02 19:11:00 MK finishes integrating ReCAPTCHAv3 to the sign-up form and API. Registration live once again.

# Impacts

We have received no complaints with regards to emails sent during this period, likely due to the vast majority of emails bound for spam recipients failing to send. This is good as it means we are unlikely to risk a disruption to emails from the API.

We had approximately half-a-dozen people attempt to sign up as members to the party between July 27th and August 2nd. Due to the method of disabling registration these people were not notified before they attempted to sign up (no indication was given on the sign up form prior to submitting it) and the details of their submission were not saved so we have no way to contact them. This is unfortunate, but not a big deal in the scheme of things.

# Follow up actions

The primary follow up action is to document things we’ve learnt, actions and improvements to make following the incident, and the completion of this report. For the sake of time management, this report is being published as it is.

There are a number of lessons to take away from this incident.

For example: we have 2 mechanisms in place to notify us of issues with the API and both failed.

The first is performed by our logging system based on detection of the words “panic”, “exception”, and “error”. This mechanism failed due to syntax errors in the search query (which have now been fixed).

The second mechanism is performed by the API itself when certain methods throw exceptions. When the API detects such mechanisms it will send an email with the details of the exception. As we had exceeded our email quota these emails were not sent. This can be solved by using a method of sending emails which will not be affected if this happens again. The fix is not currently implemented, but should be quick to do.

# Graphs

requests per minute

log entries per minute

email stats - deliveries

email stats - all

# Footnotes

  1. That the requests were made directly to the API was determined via inspection of payloads.
  2. It’s not uncommon for spam to originate from Russian or Ukrainian IP addresses. It’s unlikely this had any political motive; the payloads appear to be attempting to advertise online casinos and the like.

Comments powered by Talkyard.

Join the movement

Help us Upgrade Democracy!


Loading... members registered

Flux is Australia's most transparent political party!
Read our post on why this matters.


share


ABOUT FLUX / WHY

Why Australia needs Flux

Learn more

ABOUT FLUX / VISION AND PURPOSE

Issue Based Direct Democracy (IBDD) - The Flux Vision For Australian Politics

Learn more

ABOUT FLUX / WHAT

Use Your Own Vote Compass With Flux Voting App

Learn more

ABOUT FLUX / HOW

How Does The Flux Voting App Work? Digital Direct Democracy Explained

Learn more

ABOUT FLUX / WHO

Who is behind Flux?

Learn more

ABOUT FLUX / ANSWERS

Frequently asked questions

Learn more

ABOUT FLUX / WAYS TO HELP

Volunteer With A Political Party — Join Flux & Vote Democratically In Australia

Learn more