Modwest Blog

Email Delays Resolved

Mail on the Modwest shared hosting system is flowing again now. Nothing was lost, but throughout the day yesterday (Friday), incoming messages could have been delayed several hours, webmail didn't work for most customers, and about half the mailboxes we host were totally unavailable for several hours last night.  A blow-by-blow is available on our offsite status page.

At the high-water mark for yesterday's problems, close to 200,000 messages were hung up in the queue, awaiting delivery to customer mailboxes. That included messages coming in to our support team, so it wasn't until late last night that a bazillion support request emails came in. Needless to say, we're a little behind.

The problem, as best as we could tell, was some sort of deadlock issue with our Cyrus IMAP software. The resulting behavior was all the 'stuff the message in a mailbox' processes believed they needed to wait for access to do so. All of them. I can sort of imagine them all politely saying "no please, I insist, you first" to each other, with no deliveries happening at all.

Anyway, the situation is resolved now; not necessarily in a permanent way, but everything is working at the moment and we're monitoring servers closely. Sorry about the trouble.

-JM

End of Forwarding to Comcast

Spam, spam, spam. The never-ending battle continues.

Some may remember about how we stopped allowing email forwarding to AOL last year. Spam that originates elsewhere, passes through Modwest, arrives at AOL, and is then reported to AOL by the recipient as being spam generates a complaint against Modwest as the source of the spam -- even though we were merely a conduit, forwarding mail as per our customer's preferences.

We now know that the same situation affects email auto-forwarded to Comcast, which has a similar policy. In the past month, we've asked Comcast to please resume accepting mail from Modwest customers at least a dozen times. They're usually responsive, but it's becoming an almost daily occurrence. At the moment I write this, nearly 1,000 messages are stuck here at Modwest because Comcast refuses to accept them. We hope to push them through to their final destinations at Comcast by the end of the day, but we need Comcast's cooperation to make that happen.

Therefore, because Comcast's anti-spam policies are regularly causing significant email delays (hours or days) for any Modwest customer who attempts to communicate with any Comcast customer, we cannot allow automatic forwarding to Comcast any longer. We'll soon be getting in touch with customers who rely on Comcast forwarding about their options.

(Incidentally, Yahoo is also blocking most Modwest customer mail at the moment, and we're working on clearing things up with them. UPDATE Feb 24: Yahoo is still blocking us, and not responding to requests for information and resolution.

Update Feb 27: Yahoo has not been terribly communicative but has let the vast majority of deferred messages through, finally.)

We always try to avoid removing features that customers have been using, and we may rescind this policy if other options become available, but we'll be preventing auto-forwarding to Comcast addresses starting next week.

-JM

Email changes: Attachment Size, Spam Filtering

This past week we made a couple improvements on our email systems that we think will be be better for everyone.

First, we changed the maximum email attachment size to twenty megabytes. This is more in line with other large email providers and will help some of our graphic designer and photographer customers send and receive the large files associated with their trade. Remote recipients might not be able to receive 20MB attachments sent through our system (since their email provider might have a different policy), but incoming attachments should come through just fine as long as they can make it through our dangerous content filters.

Second, all newly created mailboxes will have their spam filter turned on, and anything detected as spam will be placed in the Spam/ subfolder of the mailbox. Note that messages in the Spam/ and Trash/ folders are auto-deleted after a period of time.

We're committed to offering maximum control over your incoming messages and had to think hard about enabling the filter by default. But two things gave us confidence: a relatively commonly expressed concern from new customers that since transferring to Modwest, they seem to receive more spam, and the (directly related) discovery that the majority of our customers had not chosen to enable spam filtering at all. So this change should help with that phenomenon. Existing mailboxes are not affected by this change.

Questions? Comment below, or contact our support team.

-JM

Maximizing Spam Filter Effectiveness

As we discussed earlier this year, spam filtering is a topic we think a lot about and a service we are always trying to improve. Meanwhile, the spammers are trying to figure out how to circumvent anti-spam measures, and so the spam 'arms race' is continuously evolving.

We recently ran some numbers to evaluate how we're doing with regards to keeping spam out of customer mailboxes. Using a dataset of around 650,000 recent email deliveries on our shared system, we compiled statistics on the effectiveness of our filter.  Our conclusion in a nutshell: the majority of our customers could easily reduce the amount of spam they receive by 50% or more with just a few clicks.

I'll explain. First though, a bit about how spam filtering works at Modwest.

How Spam Filtering Works

In addition to a number of 'front line' defenses, we use SpamAssassin,the world's #1 open source spam filtering system. The way it works is to compare each incoming message to hundreds of rules, assigning "points" for any matched rules to compute a total score. You can think about that score as a confidence level that a message is unwanted spam.

Some (but not all) the rules concern the actual text of the message. For example, a message that includes the all-caps phrase "FREE SHIPPING" might receive one point, but if it also contains "HERBAL PHARMACEUTICALS" and "NO PRESCRIPTION REQUIRED!!!", it could accumulate several more points for a higher total spam confidence score.  In OnSite, the "Normal" level of filtering corresponds with a score of 8 or more, "High" with 5 or more, and "Very High" with 2 or more. Really egregious and awful spam can score 15+ points.

You can see the spam-confidence score each message receives by checking the "X-Spam-Status" line in the full headers of any message in your Modwest-hosted mailbox. (The Mnenhy add-on to Thunderbird makes this even easier.)

Our philosophy is that you should have as much control over your mailbox as possible, and so each incoming message's spam score is compared to the mailbox owner's preferences and handled accordingly.  If you haven't set any preferences, then the server will tag messages based on the "Normal" level of filtering but deliver everything to your Inbox even if it was detected as spam.

Adoption Rate

Of course the spam filter is really only effective if mailbox owners choose to set it up according to their preferences. And here's where we see an opportunity (hence this blog post).

Of the thousands of mailboxes hosted by Modwest, only about 25% of them have spam-handling preferences set up. That means for the other 75%, every piece of incoming junk mail is delivered directly to the Inbox.

Effectiveness

The good news is that we're detecting a tremendous quantity of spam.  In our 650,000 message dataset, about half were detected as spam, but remember that at least three-quarters of our customer mailboxes only have the default filtering level of 8, which is relatively tolerant. The following chart (click to enlarge) illustrates spam scores in the data we analyzed:

With around half of all messages scoring 8 points or higher, and only a quarter of our customers actively filtering spam on the server, that represents a huge opportunity to cut down on spam deliveries.

Another interesting fact: 40% of messages earned a whopping 12 or more points. At that level of confidence, there's very little chance a message is legit (unless you really are in the "herbal viagra" business I suppose). With this in mind, a few of us in the office have been experimenting with multiple levels of spam preferences.

For example, if a message gets 12+ points, it's surely safe to delete it automatically, sight unseen. But something in the 2-5 range warrants a second look, since I sure don't want to miss any important business email. I'm using just such a system and it's working great for me. If it continues to work well, we'll consider building this functionality into OnSite. 

How you can reduce the amount of spam you receive:

If you're hosting a mailbox on the Modwest shared system, you need to set up the spam filter according to your preferences. Using the spam protection tool in OnSite, make sure you've turned the filter on, as well as chosen a Spam Handling option (subject-tag, filter into a folder, or delete). Monitor the results carefully, and lower the threshold as low as you're comfortable. You can add important senders to your 'Always Allow' list, one per line, to ensure that your business contacts and family members are exempt from the spam filter, too. This and other tips are available in the FAQ.

We'll continue to monitor the effectiveness of our spam filtering services in order to provide the best possible experience to our customers. We'd love to hear your feedback on the topic.

-JM

End of auto-forwarding to AOL

We monitor just about everything at Modwest, and one of the types of alerts we occasionally receive is that one or more of the servers in our mail cluster has a backlog of messages awaiting delivery to remote destinations. That happened this week, and upon investigation we discovered just the latest occurrence of a phenomenon with which we're well acquainted by now: due to spam complaints, AOL was blocking all email from all Modwest customers hosted on our shared system.

How could this happen? Surely Modwest has strong anti-spam policies! (Yes, we do.)

Well, there is a small loophole: We allow our customers to set up mail forwarding, which means you can have mail addressed to yourdomain.com (hosted by Modwest) automatically forwarded to another email address elsewhere (such as AOL).  Sometimes, messages that get forwarded to a customer's AOL account are then reported via the "This is Spam" button at AOL; this results in Modwest being identified as the source of that message. Enough complaints, and AOL blocks Modwest altogether.

Therefore, beginning July 27th, 2007, we are going to reconfigure all forwarding rules destined for AOL addresses to deposit received messages in the main account mailbox. Further, we will be disallowing all new forwarding rules with an AOL address as the destination. We'll notify Modwest customers who currently have AOL forwarding rules soon.

We hate to make any changes that restrict features, but due to AOL's policies, continuing to allow these types of forwarding rules can adversely affect every customer of ours.  If you have any questions, don't hesitate to ask.

-JM

GoDaddy? Helo? Ehlo?

Earlier this week we learned that GoDaddy was blocking legitimate mail from Modwest customers to GoDaddy customers. A quick call to their HQ revealed that they were blocking one of our mail servers because of a "bogus helo" problem, which, in their words, generally indicated the presence of a spam or virus outbreak. Once we eliminated this possibility and ensured our strict spam policies were being followed, I asked Mike, our senior sysadmin and Operations Manager, for his analysis of the GoDaddy's policy. Here's what he had to say:

Why is the policy of checking SMTP/ESMTP HELO/EHLO bad?  RFC2821 (which supersedes RFC821 as the SMTP/ESMTP standard) Section 4.1.1.1 describes the SMTP HELO and ESMTP EHLO Command, response, and requirements.  An EHLO or HELO is a MUST for SMTP clients.  RFC2821 Sec. 4.1.1.1 states exactly this "The argument field contains the fully-qualified domain name of the SMTP client if one is available."  There are many cases where one is not available.  Or where an accurate one is not available.

Now, strictly speaking, if one can't determine a valid domain you're supposed to use what RFC2821 calls an "address literal".  This amounts to your IPv4 or IPv6 address enclosed in [].  All of this is problematic.  In the case of using a domain name, what is the "correct" domain name?  Well RFC2821 makes that very clear.  Section 3.6 addresses what a domain name is in this case.  Specifically it states:

   * The domain name given in the EHLO command MUST BE either a primary host name (a domain name that resolves to an A RR) or, if the host has no name, an address literal as described in section 4.1.1.1.
   * The reserved mailbox name "postmaster" may be used in a RCPT command without domain qualification (see section 4.1.1.3) and MUST be accepted if so used.

So the second bit isn't applicable in this case.  We're talking about a domain, not an address, but the second first part is.

In our case we have a large mail cluster whose primary host name is mail.modwest.com, which has an associated A record.  The RFC does NOT make any requirement as to that A record matching the IPv4 or IPv6 address we're connecting to the remote host from.  But when presented with this as a HELO/EHLO domain, GoDaddy refuses the message with a hard failure.  So our case of using the HELO/EHLO is clearly supported by the RFC.  GoDaddy refusing to accept the mail at that point, claiming a violation of HELO/EHLO being refused is unsupported by the governing standard in the case that our server says "EHLO mail.modwest.com".

Now, it is possible that GoDaddy is rejecting mail because of suspected virus load coming from a particular machine, but again, it should not be stating the problem to be an invalid HELO/EHLO if this is the case.  If you can't accurately identify to the sender the reason why you're rejecting a message, at the time you reject it, you should not reject it.

So why might the domain name not be available for HELO/EHLO?  It may be a case of DHCP environments, where you don't know what your rDNS should be. It may be the case of a large cluster of machines that appear as the same machine.  It may be the case of a single machine with multiple interfaces. (Only the kernel initially knows the interface and therefore IP that was used for an outbound connection, unless you explicitly force them to come from a particular IP, but that can have other nasty interactions with stateful firewalls and reverse path filters. )

So there you have it. GoDaddy, we'd love to work together with you to reduce spam and malware. But can you let our customers' legitimate mail through to your customers now? We're pretty sure both sides would appreciate it.

UPDATE 4/20: GoDaddy says they've unblocked the mail server in question, and it seems to be the case.

-JM

Trials and Travails of Email Filtering

Not counting our managed servers or VPS plans, Modwest handles close to a million messages a day. And mailbox owners (including me!) reasonably want 2 things:

1. Keep junk and malware out of my inbox.
2. Give me all my legitimate mail.

Turns out it's really difficult to meet both goals 100% of the time. We could maybe guarantee #2 alone by delivering absolutely everything, but with 87% of email traffic consisting of junk mail, that solution probably wouldn't be appreciated by most!

So we do our best to filter out only the bad stuff that nobody wants. There are a lot of ways to try and do this, each layer of effort knocking out another portion of junk mail. (We describe some of these measures in our Knowledge Base here.)

We also provide a fair bit of control over how each mailbox handles spam within OnSite -- that's a goal of ours, to put maximum feasible control over spam handling to each mailbox owner, and we've got some additional control features planned. (The importance of recipient control over filtering was explored in a controversial article by the Electronic Frontier Foundation.)

Microsoft says that certain file attachment types are "high risk", and we know that certain filenames are associated with particular viruses and phishing attempts. Most would agree that we should try and prevent these potentially malicious attachments from being delivered.

We expend a fair bit of human time and CPU cycles trying to ensure both delivery goals mentioned are achieved. And yet the system's imperfect -- we regularly hear from customers who receive too much spam, or (infrequently) had a legitimate mail blocked or flagged as spam. It's a hard problem, but one we're committed to continuously improving.

All this leads one to wonder: Is email dead?

What do you think? What additional spam controls or filtering improvements would you like to see at Modwest?

-JM