Smaily - Notice history

All systems operational

Operational

API - Operational

100.0% uptime
Mar 2025
Apr 2025
May 2025

Automation workflows - Operational

100.0% uptime
Mar 2025
Apr 2025
May 2025

Campaigns - Operational

100.0% uptime
Mar 2025
Apr 2025
May 2025

Statistics - Operational

100.0% uptime
Mar 2025
Apr 2025
May 2025

Templates - Operational

100.0% uptime
Mar 2025
Apr 2025
May 2025

UI - Operational

100.0% uptime
Mar 2025
Apr 2025
May 2025

Webcopies - Operational

100.0% uptime
Mar 2025
Apr 2025
May 2025
Operational

Marketing - Operational

99.87% uptime
Mar 2025
Apr 2025
May 2025

Transactional - Operational

99.73% uptime
Mar 2025
Apr 2025
May 2025

Website - Operational

100.0% uptime
Mar 2025
Apr 2025
May 2025

Notice history

View current status

May 2025

No notices reported this month

Apr 2025

Outage in message delivery
ResolvedMajor outage49 minutes
  • UpdateUTC

    Timeline of the incident on Saturday, 19th April 2025

    • at 10:16 number of connections between internal relay services dropped to zero (0), causing the internal relay egress rate also drop to zero (0);

    • at 10:18 memory of all internal relay servers was at maximum capacity, causing swapping and server lock-up;

    • at 10:22 message dispatching halted;

    • at 10:27 internal relay idling alert was triggered;

    • at 10:34 investigation into alert was started and incident created;

    • at 10:53 first internal relay server was restarted, resuming delayed message delivery;

    • at 11:17 last of deferred bulk messages were delivered;

    • at 11:23 the issue was considered resolved.

    What caused the incident?

    The incident was caused by our internal relay, which is configured to consume up to 75% of server's physical memory. Under normal conditions this wouldn't be an issue, but as the number of connections between relays dropped to zero (0) the relay service started to consume multiple times more memory before it could start shedding the load and shrinking the queue. This caused the server to start swapping, increasing CPU load and in turn locked up the server.

    How we address the issue?

    We configured relay services to start load shedding when 60% of server's physical memory is consumed. We believe it gives enough headroom to services to start shrinking the queue before the server's memory is fully consumed without sacrificing performance.

  • ResolvedUTC

    This incident has been resolved.

    We will compile a post mortem after we have fully analyzed the cause of the outage.

  • IdentifiedUTC

    We have identified the cause of the outage have mitigated the issue.

    Messages are begin sent out. We are still working on a permanent fix, delays are to be expected.

  • InvestigatingUTC

    We are currently investigating outage in message delivery affecting all channels (bulk, transactional, etc).

    We apologize for any inconvenience caused.

Mar 2025

Message delivery to KeMIT mail servers blocked
ResolvedPartial outage5 hours 58 minutes
  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC

    The block has been removed. Messages are being delivered.

    We will continue to monitor message delivery.

    Note! Any bounced messages of domains listed in the initial incident report should be retried manually.

  • IdentifiedUTC

    We have managed to remove the block partially and are seeing messages go through.

    We are continuing to work on getting the block removed entirely.

    Thank you for your patience!

  • InvestigatingUTC

    We are currently investigating an incident where message delivery to The Information Technology Centre of the Ministry of the Environment (KeMIT) mail servers is blocked.

    The affected recipient domains include:

    consumer.ee
    ecaa.ee
    egt.ee
    envir.ee
    kemit.ee
    keskkonnaamet.ee
    kki.ee
    kliimaministeerium.ee
    loodusmuuseum.ee
    maaamet.ee
    maaruum.ee
    transpordiamet.ee
    ttja.ee

    We apologize for inconveniences caused!

Outage in transactional message delivery
ResolvedMajor outage4 hours 27 minutes
  • UpdateUTC

    Incident post-mortem

    What went wrong?

    Outage was caused by multiple coinciding issues - inline images breaking MIME message plain-text part, and the logic of delivery cycle and message retrying.

    SMTP protocol (RFC 2821) doesn't allow a text line to be longer than 1000 characters. The message that caused the outage had more than 14 000 characters on a single text line. In general this wouldn't have caused an outage, however the MIME message builder failed to encode the message properly as it attempts to find the most optimal encoding for each message. As the content of the message was only an inline image, the most optimal way to compile the messages was to use no encoding at all. This resulted in MIME message being rejected by SMTP server with an error 550 5.2.3 line too long.

    Rejected messages are delayed to attempt delivery of other queued messages and throttle load on other system services. In addition number of messages fetched from the queue is limited to minimize data loss on catastrophic failure. As the limit to number of messages fetched from the queue was reached, the same invalid MIME messages were retried over and over.

    Failure to detect an issue in expected time was caused by misconfiguration of delivery channel alerting rules.

    What is our plan moving forward?

    As one temporary measure we are applying plain-text content clean up before MIME message is built. This should avoid templating caused outages.

    As a more permanent solution we are reworking on changing the delivery cycle by placing invalid messages in a backoff-queue or reject the message entirely. With this we hope to resolve the issue of retrying the same invalid messages over and over again.

    Align our monitoring rules to updated delivery channel logic.

    We apologize for inconvenience caused.

  • ResolvedUTC

    The incident has been resolved.

  • MonitoringUTC

    Queued transactional messages have been delivered.

  • IdentifiedUTC

    We have identified the cause of the outage and have applied a fix.

    Transactional messages are being delivered.

  • InvestigatingUTC

    We are investigating an outage in transactional message delivery.

    No transactional messages are being delivered.

    We apologize for any inconvenience caused.

Previous

Mar 2025 to May 2025

Next