On 2025-08-04, we enabled a “trial expiration escalation” job, which runs every day to automatically mark accounts who have completed their free trial without upgrading as “pending deletion”, 90 days after trial expiration.
When accounts are marked as “pending deletion”, they can no longer be accessed, but the data still needs to be deleted. On 2025-08-26, we executed a batch of these deletions, which fully deactivated 50k+ users, deleting their data, and moving them from a “pending deletion” to “deactivated” state.
Due to a bug in the “trial expiration escalation” job, a large portion of these deleted users were moved backwards from a “deactivated” status to a “pending deletion” status, resulting in a large batch of associated notification emails being sent to the already-deactivated account emails.
A fully deactivated account has an email which looks like deactivated-<userid>@storj.io, and prior to this incident, deactivated@storj.io was not a registered email address. In other words, on 2025-08-25, we attempted to send 50k+ emails that bounced, to deactivated@storj.io. Our email service provider temporarily disabled our ap1.storj.io sending domain as a result, due to “poor mailing list quality”.
Once we became aware of the issue, we quickly registered deactivated@storj.io so that future emails sent to that address would not bounce. Additionally, we disabled our “trial expiration escalation” job to give us time to investigate and identify the issue described above. Finally, we corresponded with the support team for our email service provider to get email sending for AP1 reinstated as soon as possible.
The full range of time that email service for AP1 was down spanned from roughly 2025-08-27 at 18:46 UTC, to 2025-08-28 at 16:35 UTC.