Email Spam Coordination Across Different Services?

Surprise! It’s been 1,743 days since my last post. This isn’t a music post, so even if you are momentarily pleased to see a new one pop up in your feed, you’ll likely be disappointed by this “techie” one…

Disclaimer: this is 100% speculation on my part, I have done zero research to see if my theory is correct. I’m sharing it mostly to remember these thoughts as they occur, and perhaps in the distant hope that someone will either confirm or disprove the hypothesis with real proof.

Hypothesis

There is high level coordination among disparate email providers to determine bulk senders, in a completely unintuitive but ostensibly extremely clever manner.

I put the above hypothesis first, so that you can stop reading right now if you have no interest in this topic, because as is my usual style, this is likely to get long, quickly…

Background

I have operated my own email server for over 20 years. My CTO friends think I’m insane (perhaps for other reasons as well, but definitely for running my own email server). For clarity, it’s Postfix, not an email server that I wrote, just one that I operate on a dedicated server (where this WordPress blog is hosted as well).

I’ve been through the wars with email, sometimes caused by my misconfiguration or misunderstanding, but sometimes entirely out of my control (e.g., when my dedicated server was transitioned to a new data center and the static IP changed, and it had previously been on many RBL blacklists!).

Over time, I’ve tamed the configuration into a very stable setup. That has included complying with SPF, DKIM, DOMAIN-KEYS, DMARC, etc. Basically, anything that Google or Microsoft claim will help them validate email from me as a sender, and not mark it as spam or worse, just bounce it back to me.

The Problem

While my setup works flawlessly most of the time, on occasion, Lois or I will get a bounce back from someone (typically Google/Gmail, but sometimes Microsoft/Outlook/Hotmail). Once that bounce occurs, we’re often shut out from sending email to that service for a full day (rarely, longer!).

As you can imagine, it’s wildly frustrating to not be able to send an individual mail (this is not spam or bulk mail, but rather one to one emails to friends).

This is the error message we get in the bounce:

               The mail system

XXX@gmail.com: host gmail-smtp-in.l.google.com[74.125.20.26] said:
550 Action not taken (in reply to end of DATA command)

Wow, very helpful, “Action not taken”. Nothing indicating what we did wrong and why Google rejected the email. It feels like it should be a transient error, but it not only persists, it typically stops us from sending any further emails to anyone on that service for the rest of the day.

This has been going on for at least a couple of years. Just not often enough for me to pull out my (one remaining) hair trying to track it down.

What is going on?

Until this week, I literally had no idea (perhaps I should have diagnosed it earlier…). While I can’t say with certainty that my new understanding covers all aspects of this error, I can say with certainty one use case that definitely causes the error, and it might explain every single occurrence that we’ve had in this regard.

We (Lois more than I, but I do it too) share identical emails individually with a variety of friends. Specifically, if a group that we love puts out a new music video, Lois will send a link to that video to a group of people, but each will get their own separate email with the same email body and subject, so that they can reply just to us and not be BCC’ed in a large group.

I had never made the connection before that this somehow triggered the bounces, even though they were short emails, sent to people we’ve sent emails to 100’s of times, that almost always respond to those emails, and that we’re likely in the contacts list of the receivers. I couldn’t imagine that we were tripping any spam filter.

What do I think is going on?

I now believe (very firmly, with zero proof) that the body of the email (not including the subject, or the receivers) is being hashed. When another email with the identical hash (of the body) comes through (not sure if there is a time-limit or not), the service bounces it immediately with the above error message of “Action not taken”.

How did I come to this conclusion?

A week ago, I sent out invitations to a number of people to a house concert in March. Most of them went out fine, a very few bounced, so I didn’t have any suspicions over those bounces just yet.

This week, we discovered that one of the band members couldn’t make it, and we agreed with the head of the band that we would simply cancel the show. So, I sent an email to everyone that I had previously written to (except those that already said they couldn’t make it) telling them that the show was cancelled.

The cancellation emails were all bouncing, except for the first one!

I complained to Lois that the cancellations were bouncing, and that we would likely have to wait at least a day for the bounces to clear before I could send them again (still having no idea why they were bouncing).

Lois asked me why I thought the vast majority of the invitations didn’t bounce (to the same people, and those had links to the band in them, so if anything, they would have appeared to be more spammy).

It was a good question, to which I had no answer.

But, my brain often needs to sleep on problems before enlightening me, and indeed, the next morning I woke up with a theory to test.

While the bodies of the invitations were nearly identical, they each started with “Dear XXX,”. So, they couldn’t have hashed into the same exact body. On the other hand, each of the cancellations were identical in every way (copy/paste) without the lead Dear XXX. So, they indeed would hash into the same body.

To test the theory, I added back the “Dear XXX” to the cancellations, and sure enough, every single one went out without any bounces!

How is that Cross Provider Coordination?

Aha! It turns out that once Gmail (for example) bounced an email, so did Microsoft, Verizon/AOL/Yahoo, Apple (via @mac.com) and likely others (like Comcast).

To be clear, once Gmail bounced an email, if the next email went to hotmail.com, but was the very first such email to any Microsoft address, it was bounced immediately.

That implies (to me, proves) that it’s not just that they too hash incoming emails and bounce duplicates, but that they share the hash (somehow) among the competitive service providers, in order to more efficiently identify bulk senders quickly.

Please don’t ask me to conjecture on how they do that, I don’t have a clue.

Summary

We are both unbelievably relieved to understand what has been going on with these bounces (or at least to delude ourselves into thinking we understand it now).

Postscript

I doubt anyone who normally reads my blog will have an interest in this, but I needed to get it off my chest anyway.

I can only hope that an expert will weigh in and either confirm or provably deny my hypothesis.

Also, I have at least one more (completely picayune) email issue (rant?) that I will share if I get any feedback that this kind of stuff is interesting to anyone…