Truer Words - A Journal

Replying to Email

seth@macrobyte.net — Thu, 12 Jul 2007 15:41:20 GMT

Daring Fireball (John Gruber) has been talking about email reply styles, lately, and now lots of other people are, too. (Including me!)

What I haven't yet seen mentioned is that "top posters" are notorious for only replying to one part — possibly even just one line or sentence — in a longer email. Many times over the years I have written a long-ish message to a client, explaining how I would do something (and for how much), only to receive back a copy of my entire message with a question at the top like, "How would that part work, exactly?" or "Could you send me a sketch of how you think that part would look?"

See, I've sent a message with twelve paragraphs explaining the overall flow of an application, and the question I get back could refer to almost any of it.

Even better (worse) is that my original message probably included a few questions which have gone completely unanswered.

That, to me, is the biggest benefit of the inline-reply style. You have to pay attention to what you're doing! You start your reply by quoting the entire message. As you go through the original message, you delete the stuff which needs no reply and which isn't needed for context, and then insert your own comments immediately after the relevant parts that remain. Since most email programs show different levels of quoting in different colors, it's very easy to follow the conversation.

Recently someone sent me a "breath of fresh air." It was another software developer, and we've been talking about me helping him out with the next version of his (only) application. Our conversation has stretched out over three months, but we're both sticklers for the inline-reply style so reading back through these email messages is just wonderful. Trying to have a conversation like this with a "top-poster" (someone who always quotes everything that came before, and only puts replies at the top) would be awkward, if not impossible.

Unfortunately, some email clients make inline-replying a little difficult. Gmail, MobileMail (Apple's Mail on the iPhone), and Outlook/Entourage are all good (bad) examples. They can all make very "pretty" email with bolds, colors, fonts, links and pictures, but those things are secondary (or tertiary) to good communication. At the opposite end of the spectrum are apps like Mailsmith, which *can't* create fancy-schmancy bold/colored/linked/imaged messages, but which provide tools to make inline-replying even easier than it is already. (There are other apps like that, but Mailsmith is the one I use. Claris Emailer was another great example of this type of app, back in its day.)

Mailsmith 2.2 Mostly-Semi-Sorta (OK Totally) Public Beta

seth@macrobyte.net — Tue, 03 Apr 2007 20:19:20 GMT

At long last — and folks, I *do mean* LOOOONG last — Mailsmith 2.2 has been released to (semi-)public beta.

Like John Gruber and Michael Tsai, I've been using pre-release versions of Mailsmith for years (nay, centuries!), so most of these features are not new to me. But still, I'm very happy with this release. It's good, and a very welcome improvement over the previous public release (which was already my least-disliked email client).

(That may sound like a backhanded compliment. It's not intended to be. My old thoughts on email clients still ring true today.)

The change list in this 2.2 beta is commensurate with the amount of time Mailsmith's users waited for it. Some of my favorites:

Excellent Unicode support. Their new and improved support was what inspired me, last year, to treat all messages in Conversant as UTF-8.
Zip compression. StuffIt no longer required.
Inline spell checking.
The new Simple Query window is wonderful (though this is an area that's clearly still in beta)
Replies can be automatically moved to the same mailbox as the original. Love it!
"Send PDF with Mailsmith" — handy workflow feature
The new soft wrapping options and format=flowed feature are conceptually related, and are a couple of my favorite new features.
The whole Menus preferences system (mainly because I like to hide what I know I don't need)
The clippings changes, because they stem from work I helped with (a little) in BBEdit and I have an ego ;-)
Growl support
Don't Bug Me! My number one favorite new feature is the "Do Not Disturb" feature. See the "Mailsmith" menu, or the dock icon's contextual menu. (I called it the "Don't Bug Me" feature when I requested it. I saw another beta tester refer to it as, "the greatest feature he never knew he needed.")
Spotlight support.
All the new toolbar icons.
The text version of HTML messages is much nicer. It might look surprisingly similar to some people around here... ;-)

The list is so long that I'm reminded that anybody new to Mailsmith 2.2 probably won't know where to start. So I suggest that you set the following (new/changed) preferences if they're not already:

Under "Sending," set "Move it to Original's Mailbox"
Under "Editing: General," set "Soft Wrapped Line Indentation" to "First Line"
Under "Mail Lists," set "Show Buttons in Lists"
Under "Menus," turn off the "Tools" menu if it isn't already.
Make sure you at least notice the options at the bottom of the "Menus" preference pane, where you can control which items appear in the contextual menu.
The simple query window won't default to the currently selected mailbox(es) unless you run this command in Terminal:
defaults write com.barebones.mailsmith Misc:AlwaysUseSelectedMailboxesForQuery -bool YES

Update: I see Michael agrees with me.

The issue here is that users have two conflicting "needs" with the Query windows. I almost always want the Simple Query window to open with the current mailbox pre-selected. However, there are a few occasions (and it's probably more common for other types of users) where I have carefully chosen a subset of my 63 mailboxes (folders) and run a search, then realized that I left out a search term. With the above preference set the way I have it now, reopening the Simple Query window reverts the mailbox selection to the "current mailbox", so I need to reselect the subset of mailboxes to search.

I'm not the fan of "prefs for everything" that I once was, and I totally understand the need to keep the UI clean and simple.

My thought for this particular problem would be a way to move back and forth through a history of selected mailbox sets. When a query is run, another set is added to the history unless it's identical to the most recent set.

Just thinking out loud, I guess.

If you're a Mailsmith user (past or present), give this new version a spin. The link for downloading is at the bottom of the change list.

I'm Not Budging Until You Fix It!

seth@macrobyte.net — Wed, 31 Jan 2007 02:21:05 GMT

A bug in my preferred email client annoyed me enough that I camped out in the developer's own living room until he fixed it and gave me a new build. :-)

I kid you not. Mostly.

Plus, he bought me breakfast. This was really a good deal all around.

(He even took a picture of my breakfast!)

Then we reviewed the other unreproducible bugs I've reported in the last year (I'm as good at finding the bizarre bugs in other people's software as I am at creating them in my own!), and fixed all of them except one. That one was a doozie, though... I think they had bets in the BB office about my sanity. (Sorry, Steve, you lose! Rich saw the bug with his own eyes.)

So my "breakfast" with Rich turned into a very productive, all-day event. I didn't arrive home until 7:04 pm: 12 hours (almost to the second) after I left.

PMC Software: This is Ironic

seth@macrobyte.net — Fri, 07 Jul 2006 18:02:18 GMT

The donor who wrote to me a few weeks ago asked what lessons I had learnt and what I was planning to do differently this year with the auctions.

I wrote back to say that one of the hardest parts of the auctions, last year, was contacting the donors at various times in the process.

Of course, I got a bounce message just a minute after writing back to the donor. Their mailbox is full. Case in point.

Much Faster Mail Server

seth@macrobyte.net — Tue, 02 May 2006 19:49:40 GMT

For years, we've had this nagging little problem with Macrobyte's mail server: sending mail was sometimes very slow.

Yesterday I was finally annoyed enough with it to really debug it... and found that the culprit was what's known as the ident test. When an SMTP connection is opened to the server, it in turn tries to open an IDENT connection back to the client to get the true identity of whoever just opened the connection.

Unfortunately, testing indicates that it (almost) always fails, at least with messages being sent by people. (And for the alternative, which is mail being relayed from another SMTP server, the identity of the daemon sending the mail just doesn't matter.)

Shutting off that test in Exim's config file immediately solved the delay.

Lest this make anyone feel like yelling at me, I'll just point out that it was a gigantic waste of server resources. Like I said, the calls nearly always failed. I realize some systems provide useful responses to RFC 1413 requests, but most do not.

Character Sets and Conversant (and the Eudora Problem)

seth@macrobyte.net — Thu, 20 Apr 2006 16:14:42 GMT

We recently decided to 'standardize' all Conversant text (everything from templates to messages) on UTF-8. I've been quite happy with this decision, as it made it possible (well, easier... it was always possible) to host truly international and multi-lingual sites, and made it a lot easier to deal with content coming in from a variety of sources like Microsoft Word. For example, we no longer bat an eye at ‘fancy’ characters like “curly quotes” or — for another example — long dashes.

This hasn't come without some pain on our end, though. We have to figure out what character set was used for the text being sent when a new message is created. That's supposed to be pretty easy: email, for example, generally includes a special header called "Content-Type" which lists the character set.

The problem is when the software that sent the email lies to us. This is where I'm stumped at the moment.

One of my clients uses Qualcomm's Eudora for all of his mail. He sends HTML messages to his Conversant site, and the messages always contain curly quotes.

Here's the problem: Eudora claims the message's character set is us-ascii. This is the simplest character set in use today, and converting it to UTF-8 should not be a problem... but those of you who have any experience with character sets already know what I'm going to say, right?

US-ASCII (a.k.a. ASCII) doesn't have curly quotes. Eudora must be lying about the character set, right? US-ASCII is a seven-bit set, and the character codes for the curly quotes are all in the 8-bit range, so it seems that it must be lying to me.

This has plagued me for days, and the client is beginning to think I'm being lazy by blaming his Eudora. (Surely a big company would never do anything so obviously wrong with 'established' software, right?)

Anybody have any suggestions? I haven't had much luck looking for answers in Google. What character set is Eudora really using when it sends "above ascii" text but claims it's all ASCII? Is it the platform-native character set, like windows-latin-1 or iso-8859-1, or something else entirely?

Wrapping Quoted Text in UserTalk: string.wrapQuoted()

seth@macrobyte.net — Tue, 21 Mar 2006 00:48:38 GMT

Yesterday I mentioned Frontier's string.wrap, and asked for help testing the replacement I've written for it.

A related problem is wrapping email-quoted text: you know, the text that begins with ">". It's not the same as wrapping regular text because you don't want the quote delimiters to appear anywhere but at the beginning of the line.

I've created a new Frontier (UserTalk) script which currently lives at workspace.stringWrapQuoted(). I'd like some help testing it, just like the last one. Once I'm confident that it works correctly, I'll kernelize it (rewrite it in C) and release it to the open source project. It's final location will probably be string.wrapQuoted.

The full script and testing instructions are on their own page: Wrapping Quoted Text with UserTalk.

I'm looking for testers. If you have a copy of Frontier, Conversant, Manila, Radio, or the OPML Editor, please give it a whirl!

Update 3/21/2006: Fixed the link.

Update on the Email Conundrum

seth@macrobyte.net — Fri, 10 Feb 2006 19:58:52 GMT

The email conundrum gets weirderer and weirderer.

First, a little more background. I should have pointed out from the beginning that this contractor (XXX) has been working with the client for awhile. More than a year. Initially, he took over the management of their Conversant sites. We spoke a number of times on the Conversant support site and in private email. Personable guy (though he sees the world through Windows-colored glasses).

Now, here's the additional weirdness, the 'further evidence' I mentioned a little while ago.

The client had asked me for some additional services before shutting down their site. XXX was cc'ed on that request. I responded in the affirmative, and also cc'ed XXX.

On February 7th, after the work was done, I sent a message to the client to let him know the work was done and what I was charging. I have the original email in front of me right now: it was sent ONLY and SOLELY to the client. There's only one 'To:' address, and there are no CC or BCC addresses.

The client didn't respond. The charges were only a couple hundred dollars, which I knew wasn't a problem. 28 hours later, I resent the same email, again only to him.

On the 9th (yesterday), when I still hadn't received a response, I wrote to XXX. Here's what I said (note that BBBB is my replacement for the client's name, just like XXX for the contractor):

	>XXX,
	>
	>I have been completely unable to contact BBBB. All of my email to him
	>just seems to vanish into the ether. I never get any bounces, but he
	>doesn't seem to receive my messages, either. (I've actually wondered
	>if they're being intercepted.)
	>
	>Could you please have him call me tomorrow, towards the end of his
	>work day? I just want to put the charges through for [SNIPped for privacy]
	>
	>I'm in the US (guess you knew that), at 860-572-0244.

He wrote back and said that he would call the client to have them call or email me. They didn't. So, I wrote to him again to see if he knew why.

His response is where things turn weird again:

	>Sorry, Seth it's not a conspiracy - just me.  Today was hell and I
	>forgot they take Friday afternoons off (lucky folk).
	>
	>I HAVE sent them an email and am sure they will contact you Monday.
	>
	>I had already told BBBB what your charges were and he accepted them.

OK, so he didn't call them, and it was too late by the time I reminded him. What about that last sentence, though?

This is ridiculous.

I wrote to ask him how he knows what my charges are, but have not received any response.

If the contractor was a talented individual who (I believed) really wanted what's best for the client, there would be no issue here. Instead (warning: assumption alert) this looks more and more like a more-technically-knowledgeable-than-the-client HACK who has taken the client hostage and they don't even know it.

Maybe I'm wrong and nothing shady is happening. I haven't yet come up with any realistic alternatives, but I guess it's possible.

Mystified in Mystic

An Odd Situation, an Email Conundrum

seth@macrobyte.net — Fri, 10 Feb 2006 00:09:26 GMT

One of Macrobyte's long-time hosting clients wrote to me a couple of weeks ago to say that, after almost five years, they're moving to a new platform. They contracted someone about a year ago to handle their technical needs, and he has developed a completely new site for them in ASP. This contractor convinced them that their sites needed to be redone from scratch.

Eh, whatever, these things happen sometimes. They professed to be extremely happy with our service, but their contractor really felt that this other technology was more appropriate for their needs. Here's a slightly modified (to omit names) excerpt of my response:

My guess is that, in the end, the real answer will simply be that XXX was more familiar with the other system. That's often the real reason web sites are moved from one platform to another.

If that sounds harsh, it wasn't. I was politely asking if they would mind explaining the decision a little more fully. If there's something Macrobyte could have offered, but didn't, I'd like to rectify the situation for the next client. Still, as you can see, I had my suspicions.

I still have that email. It was only sent directly to the client who wrote to me. The contractor -- who had already taken over all hosting responsibilities for this client, including email and web -- was not cc'd or bcc'd.

This is where things start to go weird.

Three hours after I sent that email, I received a response from the 'new guy'. He cc'ed the client, and confirmed that yes, in fact, the move had more to do with familiarity with the other system than with technical reasons.

Another three hours later, and the client responds to the email from the 'new guy'. Here's the first paragraph of his email (anonymized, again):

I have read your email to me at the end of XXX's email.for some reason I haven't received the email direct.

So... he never received my private email, but somehow the new contrator and hosting service both received and responded to it.

If the client didn't receive it, then he couldn't have forwarded it to the new guy. Note that we're only talking about a matter of hours, here, not days. He didn't simply forget.

I believe all of these facts to be completely correct. With that, I can only come to one conclusion, but I'd like some other opinions. Maybe there's some answer to this which I haven't considered.

Let's leave it there for now. Comments greatly appreciated!

I'm an Emailer, not a Programmer

seth@macrobyte.net — Thu, 15 Dec 2005 19:33:39 GMT

I was just looking through my stored email from the last few years. If 'output volume' is any indication, then I'm apparently an email author first, software author second.

Since late in 1999, I've saved 15,311 outgoing messages! 90% of those (almost exactly 90%, in fact) are sorted between 63 mailboxes, along with an additional 33,490 incoming messages.

There's no way I've written 15,311 "units of code" in that time (functions, objects, scripts, source files, whatever). Not even close.

Five Years Later: It Still Sucks

seth@macrobyte.net — Mon, 24 Oct 2005 16:05:12 GMT

It's been five years since I mentioned that email clients suck infinitely.

Nothing has changed. It's the same on the Mac and the PC.

This is commodity software that we all use, but because it's nearly all free the innovation happens elsewhere.

I use Mailsmith, a commercial (read: not free) email client. It's the best I've ever used, but it still doesn't support IMAP. I want to switch to IMAP, but I don't want to drop the 99 other Mailsmith features I'd hate to go without, like it's awesome filtering system.

Justin Wood started this round of the conversation. I picked it up on Brent's site this morning.

Gmail Upgrade, and Loads of Accounts

seth@macrobyte.net — Fri, 01 Apr 2005 16:38:54 GMT

Gmail has increased the amount of storage space you can have for your email on their system. The old limit was one gigabyte. Today, on their birthday, they've increased the limit to, uh... well, they say they're going up to 2 GB, but it's not there yet. At the moment my account's limit is up to 1,447 MB.

They've added other new features, too. For example, if your browser supports it, you can use their "rich text" or "WYSIWYG" editor to add styles (bold, italics), lists (bulleted, numbered), different fonts, and various font sizes to your email.

That's not a new feature. Try replying to this post and you'll see something similar here on [tw]. I do like how they let you turn it on and off, though. Not sure why we haven't done that in Conversant, where our HTMLArea 3 plugin is either on or off for everybody. Duh. Now I have to fix that.

I have 50 Gmail invitations to give away. Big deal, so do a million other users. Tell me if you want one.

Maybe I should auction them instead of giving them away? ;-)

Mailing List Reminder

seth@macrobyte.net — Mon, 21 Mar 2005 19:03:11 GMT

It's been a couple of years since I mentioned this, I think, and I have a ton of new members and readers (RSS subscribers, mostly) who probably don't know anything about it, so...

This site (TruerWords) has a built-in mailing list. What's that mean? It means that if you really want to stay "current" with the goings-on here on my site (or in my life), you can "subscribe" to that mailing list. Every message posted here will automatically be sent to your email address.

It's interactive. If you respond to one of those email messages, the response is sent back to the site and automatically inserted into the discussion forum, so other people can read (and respond to) what you've written.

"The list," as it's usually referred to, is generally pretty quiet. Usually just two or three messages per day, or even none at all. Sometimes, if I'm feeling prolific, there will be a few more. Also, since messages can be posted by other people (any member can post a message to the discussion board), there may be a flurry of activity if I say something particularly intelligent, interesting, insightful, or partcularly stupid (meaning, I may say something that gets a bunch of other people to respond, so the messages won't always come from me).

There are complete insructions for subscribing and unsubscribing.

Note: I hate spam, but there's no perfect way to keep all "commercial messages" off of the list. In the last five years, I believe there have been six or seven spam messages posted to the mailing list. (Five of them were from the same company.) So, you may occasionally see a commercial junk mail from truerwords. I apologize in advance!

Dotster Keeps Rolling Out the Hits!

seth@macrobyte.net — Mon, 17 Jan 2005 14:18:58 GMT

Does anybody at Dotster actually test their "new features" before they roll them out?

They fixed the ridiculous bug with their consolidated domain reports (which were anything but consolidated). Today I received a single report listing all of the domains that are going to expire in the next sixty days, on which I am the owner or administrative contact. Very good!

Unfortunately, the email's recipient list also included anybody else who was listed as the owner or administrative contact of any of the domains mentioned in the newsletter. My company provides full domain hosting services. Think about that.

I'm lucky that I am the sole contact on my (or Macrobyte's) domains, and only a few clients have listed me as the admin contact on their domains. Still, those clients now have a list of most of my Dotster-registered domains, and now they all know about each other. (Good thing none of my clients are direct competitors...)

What will they try next, to top this hit? Maybe they'll send everybody's auto-renewal credit card info to everybody else, or do a courtesy mailing to their entire customer base with everybody's passwords. Yeah, that would be cool.

Six More Gmail Accounts

seth@macrobyte.net — Fri, 24 Dec 2004 15:25:11 GMT

I have ~~six~~ ~~five~~ ~~four~~ ~~three~~ ~~two~~ ~~two~~ ~~one~~ ZERO more Gmail account invitations. If you want (another) one, or know one of the last three people on the planet without one, let me know.

Update 2004/12/31: Bzzt. Sorry, they'e all gone now. Almost all of them went to people in other countries. Us wealthy americans need to share with the less fortunate when we can. ;-) (that's a joke! settle down.) One guy even told me he was "absolutely desperate" for a gmail address, and had been searching in vain for months. I teased him a little about that, but I still sent him the invitation.

Firefox 1.0 RC2 and Thunderbird 0.9

seth@macrobyte.net — Thu, 04 Nov 2004 09:46:56 GMT

Via the Burning Edge comes news that Firefox 1.0 RC2 has been released. This is expected to be the final release before 1.0 Final on November 9th. This update apparently includes a bunch of updates to the Mac's "Pinstripe" theme, though I haven't noticed anything different yet. All platforms have had some bugs fixed and minor feature enhancements, but they're specifically asked for feedback on the mac version.

The Rumbling Edge (hmm, I'm detecting a theme) reports that Thunderbird 0.9 has been released. The significant new features include message grouping and virtual folders (folders which always list messages matching some search criteria). That sounds handy! For a couple of years now I've been pushing BareBones to add that feature to Mailsmith.

My First "Sender Policy Framework" DNS Record

seth@macrobyte.net — Tue, 12 Oct 2004 09:29:42 GMT

This morning, I set up my first Sender Policy Framework (SPF) record for one of the domains hosted by Macrobyte.

SPF is a new type of DNS record which is designed to prevent (or at least limit) email spoofing. Basically, Server B is receiving an email from Server A. Server A says the mail is from yourname@yourdomain.com. Server B checks the DNS for yourdomain.com to see if Server A is allowed to send email for yourdomain.com. If it is, great. If not, it's considered a spoof (what happens from there is probably up to the administrator of Server B.)

The SPF wizard makes it easier than it would have been otherwise, but it's still a huge pain in the butt. It's made especially difficult by the fact that all of the domains we host have mail sent through the clients' home ISP's. The names of those ISP's � or anyone other domain through which a client might send a legitimate email that appears to come from the domain we host (got that?) � must all go into the SPF record. In other words, if I host your domain foo.com, and you occasionally send mail from yourname@yourdomain.com through your ISP's mail server (which Earthlink, for example, actually requires), then I have to list that ISP in the SPF record.

If Macrobyte's DNS servers are going to include SPF records for all of the domains they host, then we have two choices: talk to every single client and work out a list of domains through which they might send email for the domain we host, or we can create a form for them to fill out so they can basically do it themselves.

With the latter route, of course, we then need to contact most of the clients who use it to correct the mistakes they'll inevitably make. This isn't their fault, though... most people understand literally NOTHING about the domain name system. I can't even imagine how I could explain what I need from them in a general way which will "click" with the average joe, nevermind allow them to fill out a web form with the information I need.

In the end, I'm probably going to start with the most technically adept clients, and work from there. One client at a time. Perhaps it will become easier as I gain experience, and I'll find the right way to ask the right questions.

Unsolicited?

seth@macrobyte.net — Sun, 19 Sep 2004 15:43:19 GMT

This afternoon, I received an email from the mailing list attached to someone's weblog. I responded to the message. A few minutes later, someone wrote directly to me, asking me to take her out of my address book. The subject of her email was the same as what I had sent to the mailing list, so I knew what had happened: she thought I had sent it directly to her.

I explained that she was apparently subscribed to this mailing list and that she got a copy of my message because I had responded to another message on the list, she wasn't in my address book, and if she wanted to stop getting those messages she'd have to unsubscribe from the list. I even gave her instructions for doing so, and reminded her that I had given her the same instructions for the same reason a week earlier.

She, shall we say, "didn't appreciate" my response. She now believes that I simply enjoy sending her unsolicited email. She doesn't want to leave the list, but she only wants to see messages from the owner of the list, period. Apparently I'm the only person whose responses sully her inbox.

She done learned me real good. Next time ah wanna know how dem mail list thingies work, ah'll know jes who t'ask. Yup.

(There, how's that for a nice little vent, without hurting anyone's feelings?)

Want a gmail Account?

seth@macrobyte.net — Mon, 13 Sep 2004 06:56:13 GMT

Most of the invitations I've sent out in the last couple months for new gmail accounts have been ignored, so I'm not going to send out unsolicited invites anymore.

gmail is Google's free email service. It's still in beta, but it's pretty cool. It's also "by invitation only," so if you'd like one, send me an email.

Underwhelmed with Spam, but Impressed with Spammers

seth@macrobyte.net — Thu, 06 May 2004 07:27:03 GMT

Since installing SpamAssassin, I've been obsessively writing (and tweaking) custom rules, adjusting scores, and training the bayesian classifier.

Before it was installed I would get at least 100 UCE/UBE messages every night, and at least that many again throughout the day. SpamSieve did (and does) a nearly-perfect job of catching it all, but I really hated having to download that much useless mail in the first place. Also, Macrobyte's mail server hosts over 100 email accounts, and nearly all of them were getting some amount of spam: some a lot more than my account, some a lot less.

Last night, only five made it past SpamAssassin and into my mailbox. SpamSieve caught every one of them.

I have to admit, though, that I'm really impressed with the spammers' determination. There are some publicly available, custom rulesets for SA, that target certain types of spam. It wasn't long before the spammers started writing their messages to get around those rules. So, the rules were updated. The spammers clearly study the rules, because a little while later they started getting through again.

If you hadn't noticed, there are three tricks they've been using lately, quite effectively. First, they obfuscate words by putting punctuation be'tw_e!en the letters, or replacing letters with similar-looking characters: tests for specific words will fail, so they have to be rewritten to ignore punctuation and treat \\/ (two slashes) the same as 'v', ! the same as I, etc. Second, they've been filling the spam with random words (in html email they color those words the same as the background, so the reader doesn't see it but spamassassin does), or, even worse, lots of famous quotes. This confuses the bayesian classifier, which will see a few spam words but lots of generic words or "good" words. Third, some of them use almost no text at all, and instead just include an image with their message.

Weasels.

The filter-writers can work around these things, but it's not easy, and every additional test requires a few more CPU cycles. It's worth it, in the end, because fewer messages being delivered means the spammers make less money. Eventually, they'll stop trying.

That's the idea, anyway.

Custom SpamAssassin Rules

seth@macrobyte.net — Fri, 09 Apr 2004 08:22:36 GMT

Customizability is one of the nicer aspects of SpamAssassin. You can override the default scores for any test with just two words in a config file, and you can add your own set of tests almost as easily. Most tests consist of just three lines: one to define the test in perl, one to describe the test in English for the report added to the headers, and one to provide a score.

SpamAssassin is in the process of becoming an official Apache top-level project (which is excellent news, IMO). They're not there yet, but the Apache Foundation is hosting a wiki with lots of SpamAssassin goodies, including a small list of useful, custom rulesets (tests).

I've installed the first custom ruleset listed on that page, called antidrug.cf, which is intended to block pill-related spam but not legitimate medical email. It's well written, and highlights the way I think most spam filters should be done. These rulesets give low scores to messages that mention just one category of "pill" (like pain, sexual dysfunction, or diet medication), but higher scores when more types are mentioned. So, an email from your doctor that talks about five different types of pain meds won't be marked as spam, but spam that mentions pain medications, male performance enhancers, sleeping pills, anti-anxiety medication, and diet pills all in the same message will be incinerated on the spot. ;-).

It also gives higher scores to messages which use obfuscation when naming the medications, like two slashes \/ for the letter 'V' or the @ for the letter 'a'. (Actually, ti's a lot more thorough than that, but you get the idea.)

What's great about this sort of server-side, perl-based spam filter is that the rulesets are professionally maintained and so carefully considered. Virtually nobody takes the time to write these sorts of filters in their mail clients, as it's too much work for any one person to do well.

This is Fun

seth@macrobyte.net — Fri, 09 Apr 2004 13:04:35 GMT

Managing and tuning SpamAssassin on Macrobyte's server is turning out to be a lot of fun! I'm glad Brian talked me into this.

(I'm such a geek.)

SpamAssassin is Running

seth@macrobyte.net — Thu, 08 Apr 2004 17:25:46 GMT

Back in December I mentioned that Macrobyte was going to start running SpamAssassin on our mail server, in an attempt to cut down on the amount of spam it handles. Until a couple weeks ago, about 3/4 of all incoming messages were spam. That's ridiculous.

Brian A. set it up on the server months ago, and then a couple weeks ago I finally made time to configure and activate it. Since then, incoming spam has been cut in half, approximately. Server performance is about the same as it was before: the additional overhead of checking each incoming message for 'spamness' is balanced by there being less mail for the users to download from their mailboxes.

Last week a client in Georgia had me install SpamAssassin on their MacOS X Server box. This was a little challenging, as I don't run OS X Server and didn't have any experience with its mail server, Postfix. However, Apple themselves came to my rescue thanks to their long page of instructions for setting up SpamAssassin on MacOS X Server.

Still, there are some things to watch out for on that page:

The most difficult part of it, for me, was that they provided a procmail recipe to allow each user to have his/her own recipe also... but they don't mention that if you don't put a personal recipe file in every user's home directory, procmail will throw errors. (I don't know what the default setup looks like, but my client's OS X Server system doesn't have home directories for every mail account!)

My solution? Leave out the line beginning with "INCLUDERC"
The procmail recipe they listed uses a lock file when delivering mail to /dev/null.
```
    :0:
    * ^X-Spam-Level: \*\*\*\*\*\*\*\*\*\*\*\*\*\*
    /dev/null
    #trash all messages with a very high spam score
```
The second colon there on the first line, after the 0, is what tells it to use a lock file. That means only one thread at a time can send spam to the bit bucket, and on a high-throughput server this can turn into a serious performance problem. It doesn't make any sense, either: there's no danger of any sort of collision if it's just testing the message and sending it to /dev/null, so just leave out the second colon.

This client is already seeing a big difference with the amount of spam the users have to deal with, but now he's trying to get me to lower the cutoff point (the score at which a message is sent to /dev/null instead of just marked as [SPAM]) all the way down to the same score at which a message is first tagged. That would mean that any message SA thinks is spam will simply disappear, including false positives.

Instead, I'm talking them into some special addresses to train SA's bayesian filter (which is just one part of SA's scoring system), and I'll compromise by lowering the cutoff by a few points. A better trained bayesian filter combined with a lower cutoff point should result in a lot less spam.

Habeas Follow Up: I Spoke With Habeas, Inc.

seth@macrobyte.net — Wed, 14 Jan 2004 12:11:34 GMT

The Habeas idea seemed so good, it really bugs me that so much "Habeas spam" has been coming through lately with the Habeas headers. Three days of it now, with no sign of letting up.

Concerned that Habeas, Inc., might no longer be viable, I called and left a message on their voice mail. They called back about an hour later: yes, they are still there, and yes, they are quite aware of all the spam going out with the Habeas headers included.

After a rather confusing few minutes, I finally figured out that they were assuming the spam filtering software (like SpamSieve) were using the Habeas BlackList service. That service provides a technical block to habeas "infringers," to go along with the legal block: whenever a message with the Habeas headers is filtered by SpamSieve, it would check the Habeas black list (DNS) to make sure the sender is allowed to use the Habeas headers (which spammers never are).

Unfortunately, that would slow down the filtering process an awful lot. Also, without going through all the crazy lookups performed by a service like SpamCop, I'm not sure how SpamSieve (or the Habeas Blacklist) can be sure they're checking/blocking the right ip addresses.

Perhaps Michael would be so kind as to comment on this...?

Habeas Spam

seth@macrobyte.net — Tue, 13 Jan 2004 08:28:02 GMT

Yesterday I received a bunch of spam with Habeas headers. Those are special headers added to email that are supposed to gurantee that the message is NOT spam. The group that came up with the idea owns the trademark on those headers, and anybody who uses them without permission is suppsoed to be spanked by the courts until they can't sit down for at least a month.

SpamSieve, my spam filter of choice, has a preference setting to honor or ignore Habeas headers. I chose to honor them, hoping that they'd actually work. This tells SpamSieve that if the message has the special headers then it shouldn't bother checking to see if the mail is spam.

Unfortunately, the only mail with Habeas headers that I've ever received (that I know of) has been spam. Yesterday I got at least a dozen such messages, and yes some of them have been reported.

I hope the courts have an extra long paddle for these guys.

(Michael Tsai, SpamSieve's author, has a story about the Habeas problem, too.)

Less Spam Recently?

seth@macrobyte.net — Sun, 04 Jan 2004 13:35:38 GMT

I've been receiving a lot less spam in the last couple weeks. I know Brian has, too. Anybody else?

Perhaps spam is heaviest during the christmas shopping season? Is something else happening to cut it back?