Get access to everything we publish when you sign up for Outside+.
Whenever a piece of equipment, accessory or other product is reviewed in a consumer magazine, manufacturers sit up and take notice: What company earned kudos? How was the review done? Will it affect sales? Wait, why didn’t our product win acclaim? Hey, how come our equipment wasn’t even in there when so-and-so’s was?
You know the drill. It’s one thing when a trade publication, like SNEWSÂ® reviews a product, or a general circulation magazine, like Shape or Men’s Journals, writes that it likes one. For example, SNEWSÂ®, with a cumulative review experience of decades, never compares or contrasts products, never claims to be scientific, and always discusses the positives and negatives of each piece compared to itself, rating it as a real user might see it. Other general circulation publications? Well, we can’t speak to how they all conduct reviews, although in 2003, we looked in-depth at Consumer Reports and other reviews in Runner’s World, as well as Prevention. Since then and partly due to our discussions with the Rodale management team, Runner’s World has significantly revamped its testing procedures. (See SNEWSÂ® stories, Feb. 17, 2003, “Taking a look at fitness equipment magazine reviewers,” and March 10, 2003, “Fitness equipment review story brings passionate response.”)
However, it’s a whole different story when one of the magazines with “consumer” in its name makes its picks. Be it Consumer Reports, Consumer Guide or Consumers Digest, the public sits up and takes notice since they sound so credible. People are known to waltz into stores with the reviews — right or wrong, credible or not — in hand. Manufacturers send out press releases. Retailers gladly hang tags on the equipment, if allowed, noting the acclaim.
Of course, if done incorrectly, a review can also harm a product. Indeed, one manufacturer told us recently he’d rather not be reviewed because of that potential.
“All these reviews are shaky,” he added, noting in particular the power that Consumer Reports wields. “You can overnight destroy a product or a company.”
Of course, no one has ever complained when the company got some top billing in a review that perhaps was comparing apples-to-oranges or otherwise based on incorrect information. Company execs know if a report appears with a good rating, particularly one in CR, they better scurry to increase forecasts and production — sometimes laughing all the way to the bank.
So what is the difference among the magazines? How does each go about the process? SNEWSÂ® has talked to each one to clarify the rather muddy waters:
Perhaps the grand-daddy of “consumer” magazines, Consumer Reports is the advertising-free publication run by its non-profit parent association, Consumers Union. At its headquarters in Yonkers, N.Y., the group has a sprawling test headquarters, with scientists, engineers, biomechanists and others all working in labs and test centers to put equipment and any kind of consumer product — from treadmills to shoes to coffee to hair dryers — to the test. They measure, prod, poke and run chosen equipment and gear into the ground, sometimes starting months and months ahead of what the public sees in print.
One difference between CR and others is that it never talks to a manufacturer to get items it reviews and never accepts donations. CR buys the equipment as any consumer might. Another is that it does all testing and reviewing in-house. CR usually calls companies at the end of the testing to fact-check models, prices and other facts, but otherwise declines contact with the companies and doesn’t reveal results ahead of publication.
In addition, its mission statement states the information it prints is only for the use of consumers as information, that none may be used in advertising or to promote a company product, that none of the published information may be used without written permission, and “if Consumers Union learns that this policy has been violated, it will take all steps necessary to prevent the misuse of its names or of any of its materials, including legal action where appropriate.”
“It’s hard to argue with our results because in many cases we are just giving data,” one CR director of testing has told us. “Everything we test goes through a rigorous procedure, and we are very objective.”
Only a small portion of the website is open without subscription, and if a company submits information for consideration, which is not necessarily encouraged, it will not offer acknowledgement or contact.
101 Truman Ave.
Yonkers, NY 10703-1057
SNEWSÂ® had some difficulty tracking down a real person at 40-year-old Consumer Guide. Its website has no contact information, only a smattering of background about the company, and only offered a general email for contact (although we note there are now a couple of 800 numbers). That said, when we emailed that “info@” contact, we got a pretty quick call back from its editorial director who was more than happy to explain the magazine, its mission and how it works.
Consumer Guide, which has paid advertising on its website, has several dozen freelancers it works with, each with specialties and expertise in different areas, including fitness equipment. That means one writer is assigned and is responsible for finding products — perhaps 10 or 15, we were told — and making up a list of what he or she will analyze in each. Company management then reviews the protocol to approve it, we were told. Then the freelancer uses the equipment and writes the reviews. Since fitness equipment can be large, that may mean going to area stores or to trade shows to use it. So “use” may only encompass a few minutes of actual testing. The freelancer is also responsible for giving unlimited ratings in three categories: Best Buy, Budget Buy and Recommended. The editorial director told us that they are considering moving away from these rankings since when you have so many products it’s difficult to truly call one “best,” because it often forces an apples-to-oranges comparison.
He was clear that these are not “awards,” but simply ratings and they are not currently based on any kind of numerical system but the freelancer chooses them based on his or her judgment only. In the future, they may indeed need to be based on a numerical rating that helps standardize them, he said, and they may become more like awards. If you go to the website, you notice paid advertisement and links. If you dig into an “About Us” section, you’ll find this disclaimer:
What is Paid Advertising?
Paid Advertising are ads from companies that pay Consumer Guide for placing their ads on this Web site. The companies are not paying Consumer Guide to review and rate their products. The companies are not paying Consumer Guide to place their products on this Web site. Consumer Guide’s editorial staff is completely independent from and uninfluenced by the efforts of our sales and marketing team. By clicking on Paid Advertising you will leave ConsumerGuide.com. The Web site you will go to is not endorsed by Consumer Guide.
What is interesting is that when the publication reviewed fitness products recently, it recommended consumers not buy products sold on TV, yet both Fitness Quest and Bowflex are paid advertisers. Ironic? Yes. In addition, those companies with product rated may use the information in its promotional materials any way it wants without payment.
“We try to do the best job we can,” the editorial director said, calling the ratings a “judgment” of that one product expert.
The website, which is free to consumers, states it will gladly look at material submitted about products for possible review.
Consumers Digest was a bit harder to reach. The website is nothing but one page — with an outdated address and a phone number that goes nowhere. A telephone number gleaned from 411 directory information got us nowhere. We finally sent off a message to a general email and got a voicemail back — quite promptly — with a phone number to call. Although that phone number went through, the extension we were given did not, so we had to wend our way through operators. When finally connected, however, the editor was also quite willing to talk about the magazine and what it does.
Consumers Digest has been around about 45 years, reviewing a wide variety of categories, including big items like automobiles and room air conditioners. Some are reviewed annually, some every three to five years, like fitness equipment. As Consumer Guide does, it also uses a stable of freelancers who are considered experts, the editor explained. Initial research for product to include is done in-house then a freelancer is assigned the review for which he or she looks at and uses the equipment. Again, in the case of large items like fitness equipment, he or she most likely ends up in stores or at trade shows and may contact the company itself to track down places to use it.
The sampling is done personally by the freelancer and is meshed with any additional information research unearthed, he explained. He said reviewers also speak with retailers and others about their experiences with the pieces to help form opinions. Like Consumer Guide, this is an experiential rating. Each category’s “best buy” ratings are based on criteria established by the freelancer and approved by magazine editors and may vary from category to category. These are also not “ladders” in ranking but simply editor picks as the best in three price ranges (premium, mid-range and entry-level).
Consumers Digest also accepts no advertising in its bi-monthly publication. But instead of being supported in part by a membership union like Consumer Reports, it is run from money paid by those who earn Best Buys who want to use the ratings in advertisements and marketing. Any company can send out a press release without paying the company, he said. However, if the company wants to use the Best Buy seal on packaging, in advertising, on its website or otherwise in promotional materials, it must pay the company a licensing fee that can range from about $25,000 to $35,000 a year.
“We don’t want people to confuse us with Consumer Reports, and we don’t want to imply we do lab testing,” the editor said, “but we do feel good about the real-world testing we bring.”
The editor told us he will gladly look at materials sent to him to consider products for review. Note this is a different address than the one still on the website. They moved a few months ago but haven’t changed it.
520 Lake Cook Road, Ste. 500
Deerfield, IL 60015
SNEWSÂ® View: All these “consumer” magazines sound so official and consumers take the “best buys” of such companies like gospel. But, as you can see, how they rate products is a bit of a free-for-all. Not to say that the “expert freelancers” the two companies use may not be good. They could be. But we question whether such a powerful “best buy” or top rating should fall on the shoulders of one person. And although the same two don’t claim to use “ladder” systems, we’ll bet consumers get confused, make an assumption the products are being ranked scientifically and assume that a “best buy” product is best for everybody, when it may not be at all. We also know that looking at a piece of equipment or even using it for a few minutes isn’t a true way to experience a product and form a solid judgment of its potential. The bottom line: Buyer bewareâ€¦and manufacturer too.