Incident post-mortem
What went wrong?
Outage was caused by multiple coinciding issues - inline images breaking MIME message plain-text part, and the logic of delivery cycle and message retrying.
SMTP protocol (RFC 2821) doesn't allow a text line to be longer than 1000 characters. The message that caused the outage had more than 14 000 characters on a single text line. In general this wouldn't have caused an outage, however the MIME message builder failed to encode the message properly as it attempts to find the most optimal encoding for each message. As the content of the message was only an inline image, the most optimal way to compile the messages was to use no encoding at all. This resulted in MIME message being rejected by SMTP server with an error 550 5.2.3 line too long.
Rejected messages are delayed to attempt delivery of other queued messages and throttle load on other system services. In addition number of messages fetched from the queue is limited to minimize data loss on catastrophic failure. As the limit to number of messages fetched from the queue was reached, the same invalid MIME messages were retried over and over.
Failure to detect an issue in expected time was caused by misconfiguration of delivery channel alerting rules.
What is our plan moving forward?
As one temporary measure we are applying plain-text content clean up before MIME message is built. This should avoid templating caused outages.
As a more permanent solution we are reworking on changing the delivery cycle by placing invalid messages in a backoff-queue or reject the message entirely. With this we hope to resolve the issue of retrying the same invalid messages over and over again.
Align our monitoring rules to updated delivery channel logic.
We apologize for inconvenience caused.
