Integrity of Model Railroader Product Reviews

[rockythegoat]

Interesting topic over on the Atlas forum.

http://forum.atlasrr.com/forum/topic.asp?TOPIC_ID=17314

[Adelie]

I was chuckling a little when I read MR's review of the N-Scale Life Like 2-8-4 in the last issue. I don't really care either way, since it would not fit into the scheme my empire. I have a few Life Like diesels that run fine (the FA2/FB2s would pull a refrigerator), so I am in no way biased against them. But, to sum up the article:

It sure looks good and runs great except for the fact that it will only pull about 12 cars. Put it at the head of a small train.

If we were talking about a 0-4-0T, okay. A 600 or 1,000hp diesel, okay. Somehow, having to double head 2-8-4s to pull 20 cars, except maybe up a serious grade, seems to warrant a little more concern in the review. Maybe it's just me.

[simon1966]

I seem to remember reading that they get far more sent to them for review than they can possibly include. They probably avoid reviewing anything that is obviously bad and focus on more exciting well executed models. Certainly I have read reviews that made crticisim of paint quality and detail errors, but never a total trashing of a product.

[ndbprr]

No it isn't you. I go back to an era when they purchased the items they reviewed at hobby shops so they knew what the conumer was getting. Now the ethics have changed and they need to scoop the other magazine and placate their advertisers. Why I have no idea since they reach such a large audience and where else would they advertise? It is among many distrubing trends at Kalmbach IMHO.

[davekelly]

If MRR had to buy many of the items from a LHS the review would come out after the entire run was sold out.

[davekelly]

I do remember that when MRR reviewed some Walthers passenger car they really were critical about the shape of the roof.

[jrbernier]

Some of us are old enought to remember the old reviews - Most were luke warm, and made not to irate the manufacturer. Back in the 50's MR ran an artile about painting steam engines with 'stove polish'. Floquil got mad at MR and pulled their advertising fro a year! Back inthe early 70's Athearn was absent from MR for a numbe of years, but had large ads in RMC. In the mid 70's RMC made glowing reviews of a new steam engine kit. The actual production kits were terrible and they go many irate letters to the editor over the next couple of months(They later mentioned that they must of got 'special' preproduction kits).
The bottom line is it is a hard line to walk for product reviews when it can affect your adverising revenue. I am not saying the the editors have to cave in, but advertising revenue is a fact of life....

Jim

[Anonymous]

QUOTE: Originally posted by ndbprr

No it isn't you. I go back to an era when they purchased the items they reviewed at hobby shops so they knew what the conumer was getting. Now the ethics have changed and they need to scoop the other magazine and placate their advertisers. Why I have no idea since they reach such a large audience and where else would they advertise? It is among many distrubing trends at Kalmbach IMHO.

I humbly disagree with your use of the word "Scoop"

1- MR product reviews to me is one of the only Hobby Resources other than the internet that talks about the product at hand. I am not aware of any other reviews on a monthly schedule.

2- Perhaps there are other train magazines on the same rack as MR. But reviews? I look at MR. When I examine computer hardware I will view as many reviews as I can.

3- In support of #2 above, I also rely on the forums such as this one to understand a product better possibly before going to the store to buy it.

In regards to the 2-8-4 I think that if one had a small train improvements could be made in pulling power. I am biased because in HO I owned a 2-8-4 that was capable of about 12 cars up a 3% grade. Compare that to my BLI 4-8-2 that can get 24 of em up the same grade and haul everything I own on the flat. The limiting factor is quite literally the size of the layout in use.

Maybe one day the 2-8-4 will take it's rightful place as a strong puller when someone builds it just so.

MR Reviews to me has been quite consistent, perhaps I felt at times they may be rather diplomatic and brief when it comes to certain flaws. But hey... there are flaws in my equiptment so it really does not bother me very much.

I would like to see MR Reviews expanded to include more "Stuff" and possibly used in a layout setting where it may perform in a real world setting. I would like the good, bad and the ugly without any influence upon the reviewer.

The only way this works is if the reviewer is left alone with the train to examine it in peace, away from any outside influence.

[Anonymous]

I've seen some interesting reviews in recent years - Bachmann's original Class 37/4 would be a good example. When this was launched, it was panned by the web forum community for having assorted errors (mostly to do with the "look" of the loco - it looks fine to me and I bought one about a week after the launch). One of the magazines wrote an absolute paintstripper of a review (by this I mean really vicious) - Bachmann responded by threatening legal action (not sure if they followed this through - I only know the "bare bones" of the story and if anyone else knows more please say so!). The interesting reviews were in the more moderate magazines, one of which lauded the model when it first appeared then panned it when the retooled version arrived.

I tend to look at both MR and Continental Modeller's reviews. For example, for a while I was considering one of the Lionel HO UP turbines - one store was listing them cheaply, it had sound and other cool features, plus it was the sort of loco a museum might have. The MR review was favourable - their only complaint was that the twin motors had slightly different starting speeds. When Continental Modeller got hold of a review sample they commented on this, and the fact that not all wheels are driven (inexcusable in something this size and price), and that the detail was somewhat "heavy". As a result, I'd be more inclined to buy a BLI E7 or similar as I know them to be superb runners and with very fine detailing - all the reviews I've read agree on this. As with anything else, don't just base your decisions on one report.

Hey, where'd this soapbox come from...[:)]

[Adelie]

I agree with your HighIron's overall assessment. The reviewer should pretty much be left in peace to do the job. If the review is "foaming at the mouth," that's why they employ editors.

I mentioned the 2-8-4 because I thought it was funny. 12 cars on staight and level track was what the author stated. The solution of putting at the head of a short train made me chuckle. I wasn't suggesting he should get out Gallagher's sledge-o-matic, but it seemed worthy of more concern, considering this is a $250 retail locomotive. 12 cars on the straight and level means you almost need a pusher to get the locomotive alone up a 2% grade!
[soapbox]

The reviews aren't doing the reader any good if they get to a point where they require more interpretation of what is not being said than reading what is being said. Ultimately, when some poor slob plops down $180 (4 nscale's online price for this locomotive) and is disappointed, it doesn't do Life Like any good, either.

I've seen other respected publications go down this path in other fields, and it ultimately costs them their respect in product reviews, so I am sensitive to it. I really hope that is not happening to MR.

[Anonymous]

There was an earlier thread (within past ~6 mos.) where this issue came up, many of the same 'theories' were voiced, and one of the MR editors (Terry Thompson I think) came on and categorically denied that advertisers had any influence over product reviews.

I'll search for that thread, and post the link if I find it.

---edit---

OK guys I found the link, here it is:
http://www.trains.com/community/forum/topic.asp?TOPIC_ID=25748

** Better yet - here is the Entire Quote:

QUOTE: Originally posted by MReditor

Dear guys,

Thanks to all of you who have made constructive comments, even critical ones. We do appreciate them, and the thought that goes into them.

However, I have to throw the flag on some of the comments made about our reviews. We do not slant our reviews to please our advertisers. (If you don't believe me, read my review of the Lionel veranda turbine, or my upcoming review of the Trix NYC caboose.) We never have. We have lived by Al Kalmbach's model of "courageous editing" for 72 years, and we're not about to change that now. I keep that memo on my bulletin board, right next to my computer. (And yes, we get complaints from manufacturers who think we're too tough.)
Not only that, but what some of those who posted are suggesting isn't even possible, and it sure wouldn't be smart business. We don't even know who our advertisers for a given issue will be until long after the content is selected and written, so we can't play that game. And the "explanation of magazine economics" printed below is just plain wrong where MR is concerned. Well over half of our revenue comes from our readers. It would make no sense to jeopardize our main revenue stream for the sake of one ad, or ten for that matter.

Well, enough about that. Again, thanks to those of you who offered constructive thoughts.

Terry

[dknelson]

I must be missing the point to this thread. The review said the N scale 2-8-4 pulled just 12 cars. That seems to be the major flaw in the engine and it was mentioned. The bit about using it to pull only short trains seemed to drip with enough irony to make the point very clear. What is the problem I am supposed to perceive here?
Dave Nelson

[Anonymous]

There is definately a problem when a big steam engine cannot pull more than 12 cars without a helper uphill.

On the other extreme, a recent review stated that the BLI PRR 4-6-2 was capable of hauling 200+ cars (In HO) That is 50 pounds of train. (4 ounces / car)

One of these days one of those huge trains will shove the 4-6-2 off the downgrade curve and bury the hapless modeler in wreckage.

Then again I think somewhere there is a ghost that says if it can be done in model railroad a prototype exist for it.

For instance Unable to Pull **Ahem... UP Railroad passes my town on the main with 4 desiels and 8 cars at times. All of them roaring with power.

[selector]

As another view of all of this, the magazine testers most likely test their reviewed products on a standardized platform so that they can compare performance on a 'level playing field'. We in this forum don't do that. We take a train on faith (or because of its eye-candy appeal), and run it on our layouts, with all of the inherent track-laying screwups and wiring errors that creep into them. When the 'best loco since sliced bread' doesn't perform very well, we cry foul and assign blame where it should not necessarily be placed.

That said, the problem with advertising revenue and balanced reviews is duplicated in all walks of life. Doctors' offices get pharmacological freebies so that the product gets an advantage. Universities accept endowments from activists (remember Jane Fonda and her "Women's Studies" grant to Harvard a couple of years back? What do you suppose the results of research would have emanated from those august halls of learning? Free from bias?). And so on.

Bottom line, the magazine is a commercial entity, and must fly on its own revenue. Just as the auto magazines have to be careful about blading their clients' products, so does MR. It's always a balance between the huge adverstising dollars and the credibility of the editorial staff to keep a believing readership.

[jfugate]

It's possible to state shortcomings in a diplomatic way and not flame a product. Everything has its good points as well as its bad, and that seems to be how MR approaches its reviews.

If you get a review that really tears a product up, you can always choose to just not publish it so you don't antagonize a good advertiser. It would not surprise me if that has happened a few times over the years.

But basically I find MR's reviews useful and I expect them to shoot pretty straight, realizing of course, they will often elect to mention shortcomings tactfully and perhaps downplay them a bit. But that's not being dishonest, that's just being courteous and I wouldn't expect any less from MR.

[Adelie]

HighIron, the 4-6-2 pulling 200+ cars does have a practical use. It could be pulling on the head of the train and pushing on the rear at the same time! Saves the concern about matching the speed of the lead and the pusher.

Seems to me I saw a picture somewhere of about 6 or 8 diesels pulling a caboose once, too. At least that was not by necessity (I hope).
[:-,]

[MisterBeasley]

For an honest review, I look to these forums. Our fellow modellers have no financial interest one way or the other, and we know that they didn't get some special souped-up version from the manufacturer. "Ask the man who owns one" is still the best way to get the truth.

[CNJ831]

QUOTE: Originally posted by davekelly

I do remember that when MRR reviewed some Walthers passenger car they really were critical about the shape of the roof.

In fact, that review didn't appear until after the cars had begun to reach the hands of modelers. On the various forums posters all but eviscerated Walthers for issuing these swayback (or was it humpbacked?) cars and, in an associated discussion of the validity of published reviews, several posters down right dared MR to write a glowing report on such a faulty item. The review that shortly appeared in MR was clearly downbeat but it remained to be seen whether the posts had forced this truth or not.

A somewhat associated situation was that regarding Walthers more recent dynamometer car. Advertised for months in advance as having electronic readout, the production model appeared with a primative mechanical readout not unlike that offered by Devor 40 years earlier (!), without advising the potential buyer. The uproar all but forced the item from the marketplace, even before MR could review it.

CNJ831

[IRONROOSTER]

QUOTE: Originally posted by jfugate

It's possible to state shortcomings in a diplomatic way and not flame a product. Everything has its good points as well as its bad, and that seems to be how MR approaches its reviews.
....

I agree. I think of them as in depth informational pieces as opposed to a Consumer Reports comparison report with recommendations. Part of the problem may be what the reader perceives to be a problem. MR describes the product, it's up to you what kind of pulling power, etc. you feel you need. After all, that engine that pulls 200 cars may use only 10% of the throttle when pulling 20 cars and that may be a negative for some people who don't run long trains.

One of the things I like is how the reviewer describes any problems he found in assembling a kit.

Enjoy
Paul

[brothaslide]

It would be interesting to see an independent "Drudge Report" style of review for products. We have some thing like this on the forum. People do share their personal opinons about products etc. I have found the "product reviews" on this forum to be very helpful.

[davekelly]

Dave Nelson brought up an interesting and what I think is a valid point. MRR clearly stated that the engine pulled 12 cars. If a person regularly runs 20 cars I would hope that he wouldn't buy it and then complain that MRR didn't say it was a poor puller. I think that saying an engine pulls 12 cars is pretty much straightforward honest reporting. If the reviewer said "the engine is a strong/weak puller" I would be at a loss to decide whether or not to buy the thing as such a statement is kind of vague.

[Anonymous]

Well I personally don't place much faith in reviews, I look at them for information mostly. I never expect to see something really bad, even though it might be obvious.

Some time ago I did notice though that in the New Products photos there would always be one or more large photos of Walthers releases. And this was before Walthers became so prolific like these days. Each and every issue, there was something new from Walthers. Did no one else release worthy items?

I guess this is to be expected, as they are both in the same city and Walthers is or was the largest model railroad distributor, and they are one of the biggest advertisers.

Bob Boudreau

[Bob Hayes]

How many of you remember Tony Koester's review in RMC of Athearn's SD40-2 when it came out, and what Hal Carstens did afterward? As a result, I take RMC's reviews with a grain of salt. Just hope MR never drops to that level.
Bob Hayes

[davekelly]

Bob,

Fill us in on this.

Dave

[jfugate]

As I understand it, Mr Koester got his walking papers as a result of that review. I don't know if that's exactly all the reasons for Tony getting the axe, but it is true he suddenly disappeared as editor of RMC with no warning to the readership, and no explanation published.

It is troubling that it appears publishing a negative product review for a large advertiser's product could cost you your job -- if indeed that's what happened here.

[Bob Hayes]

Dave, Joe,
As I remember, Tony complained that the long hood was about a foot too wide and since Irv was a major advertiser, Tony got the axe. Don't know if there were any other reasons involved. As it has since turned out, I think we have all benefited from having him at Kalmbach.
Bob Hayes

[Anonymous]

When I got back into the hobby some 20 or more years ago, I bought up past issues of MR and RMC as a cheap and quick way to learn all about it. I liked the old RMC when Koester was the editor, especially the Letters to the Editor. He carried on convesations with readers, sort of like on this forum (a lot slower though!).

The letters to the editors in MR these days are mostly attaboys, gushing over some article or something else in the magazine, no real interaction with the readers.

Bob Boudreau

[davekelly]

I bet Hal is kicking himself for letting Tony go. while I don't always agree with Tony's view on model railroading, I can respect and admire his accomplishments in this hobby and his willingness to share with others. I also appreciate his ability to laugh at his own mistakes and inabilities (which contrary to what he may think are few). Truly one of the greats.

[Roadtrp]

I thought the review of the N-Scale Life Like 2-8-4 was very well written. It was a classic example of how you don't need to hit someone over the head to make your point. The number one thing I remember from that review is that the locomotive doesn't pull worth a darn. From other's comments here, it appears that message got through loud and clear to just about everyone.

[Anonymous]

QUOTE: We do not slant our reviews to please our advertisers. Terry Thompson

What does this mean exactly??
What are your review requirements,Policies etc.
What is to prevent a reviewer from getting under the table favors for a favorable review??
We all live in the real world, and we know it happens all the time,many times without knowledge from the actual company printing the review.

Do you have several reviewers but only 1 who writes the article?If not,
Would this not be a good idea if it is not currently done this way to avoid this possibility??

Perhaps the Kalmbach salaries are such that this wouldnt be tempting to an assocciate editor making huge bucks[;)]