I'm finding it honestly amazing how little good mailing list software information is out there.

I want two mailing lists set up for this site. You can see a "subscribe" box for one over on the left – it's the mailing list that gets notified when I make another devlog post, such as this one. The second is the mailing list on www.mandible-games.com which should get notified when I finally make a public release of D-Net (which I am, note, working on.) The mailing list management software should be smart enough to deal with multiple mailing lists properly, so that the user doesn't have to do the whole verify-that-this-is-actually-your-email dance twice. It should also have an interface that doesn't look like it came out of the 1990s. And, finally – and what's giving me the most trouble lately – it shouldn't get flagged by gmail as spam.

Seriously. About 2/3 of the people who have signed up with gmail accounts haven't verified their addresses. From my testing, the "authorization" mail goes straight into Ye Olde Spam Bucket. Why? I haven't been able to figure that one out, and trust me, I've been trying to. And before you say "screw gmail", I should point out that the gmail readers outnumber all my other readers, put together, by a factor of three. And one of the "others" is my lone Hotmail user who hasn't authorized either. I haven't tested to see if Hotmail tosses it into Ye Olde Spam Bucket also.

If 1/3 of the people who have expressed interest are actually able to receive emails, that is not a good ratio. It is a very, very bad ratio.

I feel like kind of a dick trying to evade gmail's spam filter, but the fact is that I want my emails to actually be received. I mean call me crazy here, but these people have signed up, explicitly, by typing their email into a form on my website – I think they might be interested in getting the newsletter. And yet, they are not.

Options that I've come up with:

  •  Try another software package, and see if that one bypasses the gmail spam filtering. I suspect it won't – the exact same message content sent from my personal email account goes right through to the Inbox with no issues.
  • Use a paid provider. Problem: which one? There are dozens if not hundreds and I can't find any real information on them.
  • Deal with most of my messages vanishing. I don't like this one.
  • Figure out some way to bounce messages through the same mail servers that I use for personal mail. Eww.

I don't really like any of these, for various reasons. But I'll have to choose one. Time to start mucking with Option 4.

The advantage to all this work is that I'm getting a really, really good idea of what actually goes into a functional large-scale website – I just feel like I'm neglecting D-Net itself. Back to work though – it's almost finished, I think.

I hope.

  • Robin Battey

    2007, November 1st 2:26 AM

    I'm not sure which mailing list package you're using, but it's been my experience that out of the free offerings, the two most commonly used ones (mailman and listserv) have by far the fewest spam filtering headaches. I think this is because it's a lot more trouble than it's worth for spammers to set up a listserv or mailman list, so they instead look for the (free) "easy to use" ones such as bMail or the like. Also, mailman and listserv both add lots of headers that are commonly in legitimate email and not in spam, which most packages don't.

    Ah, found it. Apparently you're using poMMo, the successor to bMail. From the poMMo frontpage: "poMMo is versatile mass mailing software". Wanna hazard a guess at how many spammers ignore the license and use it anyway? And how that affects spam filters, especially bayes scanners that learn the headers?

    Basically, if you pay for your mass mailing software, it's 99% likely not spam. If you use one that's hard to use, it's 99% likely not spam. Kinda sucks, but the spam filters use those heuristics in their metrics. This pretty much means you need to use non-free/difficult to use mailing list management software, or be filtered as spam.

    -robin

  • Robin Battey

    2007, November 1st 3:01 AM

    Just took a look at the emails getting sent (to my gmail spambox), and I have some more useful information for you. Here are the headers your list software sent with its message, after stripping out all the ones added by the intervening servers:

    From: Mandible Games
    User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.7) Gecko/20060909 Thunderbird/1.5.0.7 Mnenhy/0.7.4.0
    MIME-Version: 1.0
    To: (redacted)
    Subject: Subscription request
    Message-ID:
    Content-Type: text/plain; charset="UTF-8"Content-Transfer-Encoding: 8bit

    You have an error in the "Content-Type:" header, which is an almost certain spamsign (the "Content-Transfer-Encoding:" header should be on its own line, and lines are separated by \r\n in SMTP). That's the first thing to fix. Also, the headers contain a User-Agent: header saying it was Thunderbird that sent it … but has different headers than if Thunderbird had *actually* sent it. You probably want to have your software not lie about what it is, because that's *also* almost certain spamsign. I'd also change the charset to "us-ascii" unless you plan on actually using the extended characters of UTF-8, and set the Content-Transfer-Encoding to "quoted-printable" (sans quotes), to match the vast majority of legitimate email.

    Hope this is helpful.

    Cheers!
    -robin

  • chris

    2007, November 1st 8:09 PM

    I've used majordomo with good results in the past — I mean, it's a big pile of perl, so it's pretty easy to tweak the messages, and it integrates well with sendmail.

    *shrug* I don't know how gmail's spam filters work. I know gmail's implemented domainkeys, though, so you might win some points if you add those headers and the appropriate DNS record.

  • chris

    2007, November 1st 8:17 PM

    Um, I appear to completely fail at logging in via openID. Which is a pity, because I like the idea a lot. Let's try that again?

  • Zorba

    2007, November 2nd 1:10 AM

    To Robin: Yeah, the headers are weird. I started mucking with them to see if they were the reason it was getting rejected. I don't think they were – it did have more normal headers to begin with but it was still being spambucketed. I'm thinking the headers won't matter a whole lot at this point. The Content-Transfer-Encoding issue is one I hadn't noticed, but if you look at the message in gmail itself, gmail seems to have no issue with the \n thing (or at least, Firefox doesn't). I'll check this one but I doubt it'll help.

    To Chris: The Gmail headers that you see inside gmail include SPF results but not Domainkeys results. Domainkeys looks like kind of a bitch to set up though, and I'm lazy, and my SPF results check out.

    I suppose I'll try Mailman next, but also try bouncing stuff through my actual email provider.

  • Robin Battey

    2007, November 2nd 2:10 AM

    I'd put more weight on the headers. I've had to administer a few spam filters, and one of the first things to set up is filtering on header mistakes, because legitimate mail so seldom has them.

    I actually grabbed those headers from the message sent to my own mail server *and* after bouncing it to gmail. The Content Encoding header was borked on both. Gmail handles it just fine for display purposes, but that's because it just ignores it — the Content-Type and Content-Encoding headers are actually both optional headers, and it defaults to plain old text if they're not there.

    I forgot to mention majordomo in my list above, but that's another one that has very few problems. One of my clients (the guy who runs http://www.zoji.com/, a facebook-like site) was having spambucket issues, and claims that installing DomainKeys solved most of his problems. He also verifies that it's a PITA to set up. I've never done it, myself.

  • Giodude

    2007, November 2nd 7:42 PM

    why don't you just use yahoo mail, i've had it for 4 years already and never been disappointed with it. All the spam mail just goes to a spam mail folder and you never get those emails saying about getting a large sum of cash if you provide your bank info, cause they have a service that makes sure the email is coming from an actual address. But that's just my opinion, you do whatever you need to do.

  • Robin Battey

    2007, November 2nd 7:58 PM

    Zorba — what's your way of dealing with spam comments, like Giodude's "I'm paid by Yahoo to post pro-Yahoo tripe on blog entries about email troubles" comment above?

  • Zorba

    2007, November 2nd 8:22 PM

    Curiously, while I got a whole ton of spam before I actually set up the real devlog here, I haven't gotten any for quite a while. That said, I have akismet installed and it caught virtually everything for quite a while, before it stopped having anything it needed to catch. Looking it over, the last spam comment is from October 26, which – in retrospect – isn't actually even spam (I've gone and unmarked it as such.) Last actual spam is October 22, and there's only two spam comments in the last 15 days.

    I'm not convince Giodude's comment is spam, I think he's just confused as to what I'm asking. Giodude, I'm not concerned about my own email since I don't even use gmail. I'm concerned about other people's email.

  • Giodude

    2007, November 3rd 3:49 AM

    Oh, heh.. I guess I wasn't reading it right, my mistake :P… anyways if I were being paid by yahoo, my comment would have been more like this, " yahoo's e-mail service is #1, it's totally awesome, it's like the best in the world and doesn't disappoint one bit." and it would have followed with tons of links to yahoo services. But I don't do crap like that. But I guess from reading it, it does sound like spam.

  • Zorba

    2007, November 3rd 3:50 AM

    Fair enough. I think Robin's just a little oversensitive to spam, at least in this case :)

    It wouldn't surprise me if it got spambucketed by Yahoo also, though. I just haven't checked yet.

  • Stephen

    2007, November 27th 2:35 PM

    I tried it with my hotmail account it it gets flagged spam as well.

    I know majordomo as a mailing list engine does not get relegated to spam to often but i do not know how user freindly they are currently ( they have been a rather big pain in the past, but there are some really nice config tools out to alter that)

  • Zorba

    2007, November 27th 2:40 PM

    Since posting this I've converted to Mailman, and at this point gmail doesn't flag it as spam. I suspect I just need a bunch of people on the standard mail services to sign up and unflag-as-spam, but almost nobody signs up with hotmail, so that'll probably take a while.

    If you unflagged it as spam, it's appreciated. :)

  • Robin Battey

    2008, January 27th 4:52 PM

    Ironically, the spambots have found the email address I posted above, and I get lots of spam at that address as a result. Ah, the internets…

  • Zorba

    2008, January 27th 4:58 PM

    Naturally. Good ol' internet.

    I edited out the email address, that might at least slow down the future flood. (I could put it back if you wanted.)

  • Robin Battey

    2008, January 27th 5:00 PM

    Thanks. I'm kind of embarrassed I didn't redact it myself.

  • Akktri

    2008, May 27th 5:21 PM

    I suspect this is because there is a kabal of e-mail servers, and if you're creating new competition, they'll work extra hard to make sure your address isn't recognized by the system. They're hoping that you'll eventually give up and use one of their servers, like Yahoo, Google, etc.
    You could always e-mail hotmail and some of them and ask them for their cooperation, though.

  • Полностью согласна!

Leave a Comment

Subscribe without commenting