In addition to the drawings and operations being deliberately hand-picked to demonstrate new features, no direct comparison is performed at all between the [...]]]> Following on from the AutoCAD 2011 productivity study I critiqued earlier, there is now an LT version. Do the same credibility problems apply to this study too? Yes, and then some.

In addition to the drawings and operations being deliberately hand-picked to demonstrate new features, no direct comparison is performed at all between the two releases on the same platforms. Every single quoted “productivity improvement” figure includes, free of charge, three years of hardware and operating system progress and a more upmarket graphics card.

If you read business “news” sources that just reprint press releases, such as this Yahoo! Finance one (thanks, Carol Bartz), you won’t see this mentioned. Instead, you will see deceptive statements like these:

David S. Cohn, an independent consultant

Er, no, in this context he’s not independent, he’s an Autodesk consultant. Not that there’s anything wrong with that.

overall productivity gains of 44 percent for users moving from AutoCAD LT 2008 or earlier versions to AutoCAD LT 2011

…as long as you only ever perform certain carefully selected operations and upgrade your hardware and operating system. Like the other study, the 44% figure is totally meaningless and quoting it without qualification is downright deceptive.

Most users will be able to get more work done faster by upgrading to AutoCAD LT 2011

This statement is totally unsupported. There is no analysis of what “most users” do with the software, and no attempt to quantify the portion of time such users spend on these hand-picked operations. Neither is there any analysis performed on more common operations to see if the new releases introduced any detriment to productivity in those areas.

Improvements to the graphical user interface deliver a 43 percent productivity increase.

If that’s true, why do so many users of 2009 to 2011 immediately turn off the new user interface? Are they all stupid Luddites who have a burning desire to work much less efficiently? This study, like its non-LT counterpart, contains many unqualified statements about the Ribbon improving productivity and providing other benefits. I’d really like to see a proper independent study done into that.

To sum up, Autodesk is quite prepared to say misleading stuff about its products that will be regurgitated unquestioningly by those who don’t know any better, in the hope that it will be believed by those who do, and not exposed by those who care. But it’s not prepared to answer straightforward legitimate questions about its business, offering a pile of spin instead. This, supposedly because “management in publicly trade companies are forbidden by US laws and accounting regulations to discuss some topics”.

I think I’ll borrow a phrase from Deelip here, as it seems appropriate.

Bottom line. This is bullshit.

It just so happens that right now I’m in a no-bullshit mood. I’ve been exposed to more than enough of it lately. Unfortunate timing, Autodesk.

I know this sort of marketing device is nothing new, and maybe that’s the point. This kind of thing is so 20th century. In the good old days, negative commentary about stuff like this would be seen by few, and largely confined to company-controlled environments and one-way media such as printed magazines. Things aren’t like that any more. This sort of nonsense is being increasingly noticed, criticised and derided in blogs and social media. I have hope that the point will soon come when companies’ PR consultants work out that the negatives of spewing bullshit outweigh the positives. When that point is reached, the bullshit will stop. And won’t that be great?

According to a recent independent study, AutoCAD® 2011 can help you work up to 44% faster with the latest productivity enhancements.

I have a couple of problems with that sentence. First, it’s not an independent study. It’s a study conducted by [...]]]> Heidi Hewett just reported the following on her blog, about a productivity study:

According to a recent independent study, AutoCAD® 2011 can help you work up to 44% faster with the latest productivity enhancements.

I have a couple of problems with that sentence. First, it’s not an independent study. It’s a study conducted by long-time respected CAD figure David Cohn, but it was specified and paid for by Autodesk:

This productivity study was performed at the request of Autodesk Inc., which funded this work.

That’s not exactly independent then, is it? Second, the study does not state that AutoCAD 2011 is responsible for a 44% improvement. That’s a figure that combines both the effects of AutoCAD 2011 (over AutoCAD 2008), plus the effects of using a newer, faster PC. Just stating that figure wthout such a disclaimer is misleading.

Now to the study itself. Let me make it clear that I have no problem with David Cohn, who is respected, experienced and honest. I do not doubt that his study accurately describes his observations of the time taken to perform the chosen operations on the chosen drawings. The problem is that the study is designed to concentrate purely on a set of AutoCAD operations that benefit from the changes of the last three releases. In other words, the dice are very heavily loaded. To David’s credit, he states that very clearly in the study report:

Each drawing was chosen based on a number of criteria designed to showcase one or more features of the software that did not exist in AutoCAD 2008 but were added in subsequent releases. While each drawing could certainly be produced using the features and functions available in AutoCAD 2008, the advanced capabilities added in subsequent releases would likely enable a typical user to produce the drawing faster using AutoCAD 2011.

Since the premise of the test was to determine how much time could be saved by using a new feature, the test itself was already predisposed to show that using AutoCAD 2011 is more productive than using AutoCAD 2008.

A quick skim-read shows that there are several other problems with the study. For example, it doesn’t attempt to measure the productivity of those operations that are common to both releases, which are much more likely to be used in bulk by typical users. The report states that the Ribbon interface is likely to be more productive, but makes no attempt to justify that by comparing the exact same operations performed using the two interfaces.

In addition, both AutoCAD 2008 and 2011 are measured on a typical middle-age PC using XP, but only 2011 is measured on a modern PC running Windows 7. The report states that the latter tests were performed after the former tests, so the times will also be biased by familiarity with AutoCAD 2011, the drawings and the operations required. That’s where the 44% figure comes from, and it doesn’t mean anything.

What’s the point of studies like this, that are self-evidently designed to produce a good-looking outcome? Who are they supposed to fool?  Come on Autodesk, either do these things properly or don’t do them at all. Please.

Disallowing bug fixes for non-subscription customers is reprehensible, no matter what kind of spin is put [...]]]> I’m still looking for your questions about Autodesk Subscription and upgrade policies and pricing. No matter what you think about that, you have to admit that Autodesk’s current policies are less anti-customer than those inflicted on SolidWorks users.

Disallowing bug fixes for non-subscription customers is reprehensible, no matter what kind of spin is put on it. Not only that, it’s clueless. So you’re annoyed at Autodesk for whatever reason and are looking for alternative software from a company that doesn’t mistreat its customers? You know not to even bother looking at SolidWorks, don’t you?

Edit: more relevant links and customer comments from Devon Sowell and Matt Lombard’s blogs.