Issue with edge services' outage affecting customers in Europe and North America

Incident Report for Storj DCS

Resolved

After extended monitoring, we are marking this incident as resolved.

We again apologize to all who are affected by the disruption and appreciate your patience and support. The team will conduct a post mortem and we share more details as we investigate fully.
Posted Nov 08, 2024 - 23:34 UTC

Update

All services have been restored for nearly all customers. We are currently addressing isolated cases with individual customers to ensure full resolution. We continue to monitor.
Posted Nov 08, 2024 - 20:03 UTC

Update

We are deploying an additional fix. All services are stabilizing, but the situation is ongoing so we continue to monitor and fix as we see issues.
Posted Nov 08, 2024 - 19:10 UTC

Update

US1 API services are operational. We are still monitoring US1 linksharing and gateway elevated error rates. EU1 Linksharing and EU1 gateway are also now seeing elevated error rates.
Posted Nov 08, 2024 - 18:44 UTC

Update

US1 API has been restored, we are monitoring linksharing and gateway but error rates are decreasing. EU1 and AP1 continue to function normally.
Posted Nov 08, 2024 - 18:18 UTC

Update

US1 API is now experiencing errors affecting all US1 services (gateway, linksharing, uplink). EU1 and AP1 are unaffected.
Posted Nov 08, 2024 - 17:44 UTC

Monitoring

All services are operational. We are continuing to monitor.
Posted Nov 08, 2024 - 17:29 UTC

Update

Linksharing is being brought online in the U.S. and error rates are dropping. E.U. will be restored next.
Posted Nov 08, 2024 - 17:15 UTC

Update

Gateway services (S3 API) are still online and functioning normally. Linksharing is being brought back online.
Posted Nov 08, 2024 - 17:08 UTC

Update

E.U. Gateway services (S3 API) are restored. The team is focusing on bringing back online linksharing next.
Posted Nov 08, 2024 - 16:35 UTC

Update

U.S. Gateway (S3 API) services are restored. E.U. Gateway services are seeing decreased error rates and will be restored soon.
Posted Nov 08, 2024 - 16:30 UTC

Update

We are continuing to bring gateway services (S3 API) back online, requests in U.S. are seeing error rates decrease. E.U. gateway services are coming back online.
Posted Nov 08, 2024 - 16:20 UTC

Update

We've implemented a fix and gateway services (S3 API) is gradually coming back online. Linksharing is still experiencing an issue.
Posted Nov 08, 2024 - 15:58 UTC

Identified

The issue has been identified and a fix is being implemented. The team will send another update in 15 minutes.
Posted Nov 08, 2024 - 15:38 UTC

Investigating

We are experiencing an issue with edge services' outage affecting customers in the Europe and North America regions.

Our engineering team is actively investigating.

We apologize to all who are affected by the disruption.
Posted Nov 08, 2024 - 14:43 UTC
This incident affected: US1 (US1 - API, US1 - Linksharing, US1 - Gateway) and EU1 (EU1 - Linksharing, EU1 - Gateway).