Datacake - Delayed email notifications – Incident details

Delayed email notifications

ResolvedPartial outage13 hours 46 minutes

Affected

Outbound Email Sending

Partial outage from 5:05 PM to 9:06 PM, Operational from 9:06 PM to 6:51 AM

  • ResolvedUTC

    We need to correct our previous update. Not all emails during this incident were delivered.

    Cause: at about 17:05 UTC on 14 Nov, our external email provider put a hold on all outgoing email from our account without notifying us. During the hold, their system kept accepting our emails as if they were being sent normally, but held them without delivering. We noticed the problem through our own monitoring and contacted the provider. They released the hold the same evening.

    What happened:

    - 14 Nov, 17:05 – 21:10 UTC: all platform emails were held by the provider. They were all delivered late, at about 23:30 UTC.

    - 14 Nov, 21:10 – 15 Nov, 06:39 UTC: while we worked around the hold, email notifications from rules were not sent. Device offline notifications, scheduled reports and other emails were delivered normally in this window. Rule email notifications from this window were not delivered and can't be resent.

    Since 06:39 UTC on 15 Nov, all emails, including rule notifications, are being delivered normally again.

    What wasn't affected: device monitoring, data collection and rule evaluation kept working the whole time. Rules still ran, and their other actions (webhooks, SMS, downlinks and so on) worked as usual. Only the email part of rule notifications was missing.

    What you can do: if you rely on rule email notifications, please check your devices' data and your rule history in the dashboard for the window from 21:10 UTC on 14 Nov to 06:39 UTC on 15 Nov.

    Next steps: a provider holding our email without notice isn't acceptable for notifications our customers rely on, so we're evaluating alternative email providers. We're also changing how rule notification emails are sent, so that switching providers can't affect them again.

    We're sorry, both for the outage and for the incorrect information in our previous update.

  • MonitoringUTC

    Email delivery is working again. Since 21:00 UTC, all platform emails are being sent normally again, including device offline notifications, scheduled report emails and rule-based email alerts.

    Between about 17:05 and 21:00 UTC, emails were not delivered. We're working with our email provider to find out whether emails from that window can still be delivered. If they can, you may receive some notifications late. If they can't, we'll send scheduled reports from that window again. We'll post an update once we know.

    Device monitoring, data collection, rules and all other notification channels worked normally the whole time.

    We're sorry for the trouble and will keep monitoring.

  • InvestigatingUTC

    Since about 17:05 UTC today, emails from the platform are not being delivered. This includes:

    - Device offline notifications

    - Scheduled report emails

    - Rule-based email alerts

    The problem is with our email delivery provider, and we're working with them to fix it. Everything else works normally: devices are monitored, data is collected and stored, rules run, and reports are generated. Webhooks, SMS and other non-email actions are not affected.

    Until this is fixed, please check device status and reports directly in the dashboard. Generated reports can be downloaded from the report page. We'll post an update as soon as email delivery is working again.