It’s hard to benchmark anti-spam effectiveness

I’ve been following the mailing list for the Conference on Email and Anti-Spam (CEAS) Live Spam Challenge that took place last week in Mountain View, CA, since Commtouch had a couple licensing partners participating.

While I’m not involved on the technical level (lucky for me!) it didn’t take a software engineer to figure out something was wrong, with mailing list subjects like “Timeouts,” “Unplanned Restart,” “Not enough feedback,” and eventually “We Concede Defeat” and “Lessons Learned.”

I’m not going to get into all the technical details of why the test was problematic, however all agree that a lot was learned, primarily about how hard it is to test varying anti-spam techniques effectively. For example, the stream needs to be real-time, or it gives certain engines an advantage (and a delayed stream would give Commtouch, which is designed to work on real-life real-time spam, an unfair disadvantage). Feedback needs to be provided for those engines that “learn” (of course, Commtouch would be perfectly happy to eliminate this unneeded complexity from the test routine :). And a whole host of other issues that have been discussed at great length by the participants in the Challenge.

The one thing that was clear from the collegiality of the messages is that we are all in this together, trying to make the world a spam-free place. Hats off to those who participated, and contributed to making the next Challenge a better, fairer, and more useful benchmarking exercise.