Comment: Don't ban benchmarks

I recently read a license agreement for a suite of software development tools and discovered some interesting fine print: the agreement prohibits licensees from benchmarking the tools. Well, it doesn't prohibit benchmarking per se, but it prohibits disclosure of any results.

Although my company, BDTI, is a benchmarking company (and we benchmark tools as well as processors), we weren't planning on benchmarking these particular tools. We bought them for use on a software development project. But this kind of restriction disturbs me all the same, because it means that, should we be so inclined, we could not simply buy the tools, benchmark them, and publish the results. We would have to get the vendor's permission. And in our experience, vendors don't like having benchmark results published unless they know in advance the results will be positive, which when you think about it, somewhat defeats the purpose of independent benchmarking.

Can you imagine if car manufacturers would only sell cars under the condition that they couldn't be road-tested or if restaurants required that diners sign a pledge not to discuss their experiences? It would make you wonder what they were trying to hide, wouldn't it?

When I've asked tools vendors about benchmarking prohibitions in their license agreements, the most common rationale they've offered is that they don't want someone to do a lousy job of benchmarking, publish the results, and make them look worse than they are. We hate that too. But in fact, it rarely happens. I think the real reason behind such prohibitions is simply the fear that accurate information might be circulated that pointed to weaknesses in the product.

But let's be real. If the product has weaknesses (and every product does), customers will eventually figure that out. Better to let them know about weaknesses ahead of time, or even get the tool developers to fix them. People are much more likely to buy products when they believe that they have a solid grasp of what they are getting, even if they know that what they're getting isn't perfect. And people who get burned by hidden product weaknesses are unlikely to become repeat customers, especially if they suspect the vendor has not been forthright about their products.

Allowing benchmarking is one way for vendors to show good faith with their customers. It's good ethics, and it's good business.

About the author
Jeff Bier is the president of Berkeley Design Technology Inc., a benchmarking and consulting firm focusing on digital signal processing technology. Jennifer Eyre White of BDTI contributed to this column.