Smaily - Notice history

This incident has been resolved.

All queued events have been projected (incl. Message and Subscriber Action Log), no data loss detected.

We have implemented the fix and are monitoring the result.

We have identified the issue and are working on the fix.

Timeline of the incident on Saturday, 19th April 2025

• at 10:16 number of connections between internal relay services dropped to zero (0), causing the internal relay egress rate also drop to zero (0);

• at 10:18 memory of all internal relay servers was at maximum capacity, causing swapping and server lock-up;

• at 10:22 message dispatching halted;

• at 10:27 internal relay idling alert was triggered;

• at 10:34 investigation into alert was started and incident created;

• at 10:53 first internal relay server was restarted, resuming delayed message delivery;

• at 11:17 last of deferred bulk messages were delivered;

• at 11:23 the issue was considered resolved.

What caused the incident?

The incident was caused by our internal relay, which is configured to consume up to 75% of server's physical memory. Under normal conditions this wouldn't be an issue, but as the number of connections between relays dropped to zero (0) the relay service started to consume multiple times more memory before it could start shedding the load and shrinking the queue. This caused the server to start swapping, increasing CPU load and in turn locked up the server.

How we address the issue?

We configured relay services to start load shedding when 60% of server's physical memory is consumed. We believe it gives enough headroom to services to start shrinking the queue before the server's memory is fully consumed without sacrificing performance.

This incident has been resolved.

We will compile a post mortem after we have fully analyzed the cause of the outage.

We have identified the cause of the outage have mitigated the issue.

Messages are begin sent out. We are still working on a permanent fix, delays are to be expected.