Of all the sources of comparitive reviews of consumer products, I hold Consumer Reports in very high regard. This is the company that essentially established the field, and they generally are careful and precise in their testing and reporting. I discovered when I was deeply involved in personal computer testing over the past 20 years that Consumer Reports did not always have the expertise or resources to test some technical products in depth, however, in ways that provided meaningful results for their readers.
As a result, I was discouraged but not completely surprised by a press release from Naked News this week that tracked the scores given various HDTV products by the magazine, and found some unexplained discrepencies. For example, one Samsung 32″ LCD scored 62 points in the recent December issue, but the same model received only 46 points in the issue published just one month earlier. The Naked Truth reported that seven different models had their ratings change in a just a one-month span.
If you’ve read some of my materials, you know that I have taken the position that it is impossible to adequately test HDTVs comprehensively and consistently, and this makes all comparitive reviews difficult to use. It is sad that Consumer Reports is the one to provide such strong evidence supporting my position on the subject, but quite frankly, if Consumer Reports can’t get it right, who do you think can? The bottom line is that you should not rely on product reviews to do much more than report the specifications; if you want meaningful and useful information to help you compare products, be prepared to gather that information yourself.