What do reviews of role-playing game products do?

In this hobby blog, I have commented obliquely before on some gaming products, but I have not yet reviewed any gaming products. I was about to write my first review today but I found myself reflecting on what the point was, instead.

There are other bloggers (hello out there!) who regularly review role-playing game products. I consult them in stolen moments, looking for gems. I have even bought a few things on the spot after reading ecstatic reviews. I'd be willing to bet that the reviews of game books and modules by tenfootpole or Beyond Fomalhaut or Questing Beast or Refereeing & Reflection can make or break a new product's reception unless it has big corporate backing. A negative review can probably nullify a product's reach and market.

Blog reviews of role-playing games and modules are informal, and they are unlike other kinds of reviews in several ways.

What do I mean by other kinds of reviews? Well, there are a lot of kinds, each with its own economy behind it. In my day job, one of the things I sometimes do, on request, is to write reviews of scholarly books published by academic presses. Such reviews appear in journals where other researchers read them to learn whether they need to take that work into account in their own research. I also write reviews of book proposals and academic articles before publication, when presses ask me if I think they should publish them. These are not published openly and are anonymous to the author. Scholarship requires peer review. You do this usually without remuneration because you have the expertise and that's your job. In reviews of prospective publications, I give detailed comments, sometimes line by line, about method, argument, use of sources, use of modern scholarship, and even things like editing and presentation. Sometimes I endorse the publication of such materials, sometimes I say it doesn't deserve publication, and sometimes I say it should be published provided specified changes are made to remedy problems. Most scholarly publications are vetted by editors and then get two or three such reviews before going to print. Afterwards, still more reviews by other people follow, evaluating the finished product.

As I contemplate writing reviews of game products instead, I have to make a conscious effort to do things differently from that. Gaming products produced by small companies and by fans like me, as a labor of love, do not warrant that kind of criticism. Sixteen pages of gonzo dungeon filler and a map do not deserve the scrutiny warranted by a forty-page scholarly article on a topic like the changing economic organization of the Roman Empire in the third century.

Role-playing games comprise a much smaller industry than scholarship. Just compare the number of game designers in the world with the number of scientists, researchers, professors, and lecturers. The quantity of publications in all the sciences, all history and the humanities, and all the social sciences, dwarfs the table-top role-playing game industry by orders of magnitude. The amount of money involved in TTRPGs is minuscule by comparison with all of scholarship.

The big difference, though, is a different standard for what should be printed. The standard for RPG reviews is not "what's true?" but "what's fun?"

But are we even qualified to adjudicate what's fun for others?

When I am asked to give a review of a scholarly publication, it's because I am perceived, by my own record of publications, to have expertise in something addressed by that publication. If I don't believe I have that expertise, I refuse to review it and I recommend one or more specialists in the field concerned who I think would do the review better. When I see reviews by scholars about things that they don't really know about (uncommon, and it's usually obvious if that's the case), I take them less seriously.

The point is that you're supposed to be qualified to do a review.

I notice that the good people who regularly review role-playing game products distinctively lack one thing. They almost always lack experience using the products they review. This doesn't just go for today's blog reviewers. It goes back to reviews in the old days, in Dragon magazine (although many reviews there were based on actual use).

You would think that each review should start by saying, "I have (or have not) used this product in play." Mostly reviewers of a game product talk about how they might use it, or whether it looks cool, or if it adheres to ideal design principles of a general nature. There is a gap between those ideals and use in play.

The implicit idea is that the expertise required to review RPG products is based on experience playing in general, not actually using the product under review. Reviewers are supposed to have played a lot of different kinds of games, at least enough to know what they like, enough to justify their preferences. Even though it's a matter of taste, an odd aspect of the subculture is that gamers typically want to justify their personal preferences. That justification is implicit in an RPG product review. Gaming cliques form (like OSR or Story Games) on the basis of preferences and sometimes self-serving (typically false) histories of the hobby. Reviews play a little role in buttressing the coherence of RPG cliques.

As I reflect, though, the lack of practical experience with the specific product is what really sets RPG product reviews apart from reviews for just about anything else. It doesn't have to be Consumer Reports, where products are rigorously evaluated and tested and compared in effectiveness. Even if you read an Amazon buyer's review of some new gizmo, it will say how it worked when they actually used it. We don't get that for most reviews of role-playing game products.

Take music reviews to illustrate the problem. Whether a reviewer hears a music performance live or recorded, the reviewer has experienced the music. Reviewers of role-playing game products are at a disadvantage, because they need a group of people to invest hours of time before they have experience with the products under review. The best that most game product reviews can do is to imagine what it would be like to use it. And yet they sound convincing.

Reviewers can do this because imagining what it is like to use a product is typical of all gamers. I am pretty sure that most people who buy RPG products spend more time imagining what it would be like to use it rather than actually using it. You never squeeze all the potential out of an RPG module (let alone a game system) because there is an undefined variety of potential outcomes in use. That's what is so exciting about them.

That was my experience in the old days. There were more cool games than anybody could play. I still have my collection of early B/X, BECMI, and AD&D1e modules, and I never ran maybe half of them. They are just fun to look at. A kind of collector culture develops in this way based on imagining unrealized possibilities, even without any intention of realizing those possibilities. It's like the players who generate character after unused character because part of the fun is in the thrill of unrealized potential. Every character could be the basis of an epic, if only we had more time to play.

Part of RPG product reviews is to convey that "fun-to-look-at" feeling. Reviews of role-playing game products give something like the experience of getting to look over the shoulder of somebody as he flips through a book, hearing his comments, giddy or sneering or bland, as he turns the pages. It's like meeting a buddy who has just returned from the game store with an item he has not used, but it's fun to hear him talk about how he might use it. "I would totally run this adventure!" he says--even though probably neither of you ever will. You also experience unconscious peer pressure if he says enthusiastically that it's cool or if he says it's lame and stupid and hates the pictures.

Reviews of RPG products have this tentative character because there are far more products than there is time enough to play through them, and people rarely use them as they are written, but modify them at will.

Our time in actual play is precious and scarce. We can imagine far more cool stuff than we have time to share in play with friends. This creates a surplus of ideas that appear as modules that will never be used, or used by almost nobody.

We also get retro-reviews, those reviews of gaming products that came out forty-some years ago. Reviews of old-time products are full of nostalgia, which is not a normal motive to review anything else (except perhaps for classic films or student papers on classic works of literature).

Looking over somebody's shoulder at a new module or rule set (or at an old collector's item) can be pretty exciting. We want to know if the book is bound well and looks good. We want to know about layout and art. Does it inspire us to play? We want to know about the contents, as if we were flipping through the book lifted momentarily up from a game store rack--the way I used to do it a long time ago, when visiting actual stores in actual spaces was normal. Then we make the assessment: buy, save money to buy, or ignore? Would it be fun to play through this with your friends? But RPG product reviews are a lot more like videos of box openings than commentary on the practicability of the product. If they tell you whether it's usable, the assessment is based on a preconceived preferred style of play. Play styles are akin to fads.

In short, RPG product reviews are superficial out of necessity. They are rarely based on performance in use. To overcome this limitation, reviewers of RPG products establish themselves as representing a specific palette of tastes. People who regularly read a given reviewer's reviews know what to expect, because we know his tastes. If you feel your tastes are in line with his, you trust his reviews more.

From this, I suggest that RPG reviewers should say something explicit about the standards they are using. Without knowing those standards, you don't know what you are reading.

Those are some of the ways that these RPG product reviews work.

Also, where are the female reviewers?

So what do reviews of role-playing game products do?

Reviews of gaming products express fandom, they express preferences of style and favoritism towards specific designers or towards a specific design ethos. They prioritize general models of feature design over actual experience. Prioritizing general models puts a lot of weight on one's theoretical views about what makes for good play experience, and that makes reviews align with cliques.

All these ruminations should give some cheer to authors of badly reviewed game products. Nobody likes a bad review of their lovingly produced stuff, but if the reviewer didn't play it, how seriously can you take the review? Any game product can be good for something if it finds the right audience.

Then again, there is an awful lot of junk, isn't there? And the less work you have to do to adapt a module to your use at the table, the better you may think of it. In the end, I'll take the word of an experienced GM's review over the author's self-endorsement. They can answer some basic questions. That's why reviews of RPG products remain basically useful, even if they are limited in their value to people who actually use them in play.

But doesn't quality matter?

Yes, quality matters. How it is assessed is what makes me skeptical.

Here's a funny thing. My favorite adventure module of all time was B/X D&D's B10, Night's Dark Terror. Because I hardly ever played D&D with my players in the old days, I never met anybody else who knew anything about B10. I never spoke with anybody about it, except for the small group I ran it for once. Late last year, when I started to look around to see what was going on with role-playing games, I found retro-reviews and "best of" lists of old D&D adventures. These drew me in with nostalgia. B10 is consistently rated as one of the best. I don't think that's merely chance. Quality is real. Defining its makeup is the hard part. I am skeptical of ideal design principles, because skillful Referees can easily turn junk into wonder, and unskillful ones can turn great adventures into lame experiences. In the case of B10, retro-reviewers hedge about its "railroad" quality (a term I discuss here), but they can't help liking it a lot anyway. If you ask me, B10 provides a package of multilayered wondrous exploration, one of the main incentives of fantasy role-play. It's much harder to boil down that feeling into generic design principles. Or maybe the game philosophers of the blogosphere have been so busy barking up one theoretical tree that they ignored a bunch of other ones.

Another great series of modules was the Enemy Within campaign (1986-1989) for the original Warhammer Fantasy Roleplay. I ran the entire campaign from beginning to glorious end for high school and college students when it came out, including some all-night game sessions without any sleep and lots of caffeinated soda and fast food. The Enemy Within campaign (by the same authors as B10!) was fantastically fun. By today's prevailing outlook, it's nothing but a "railroad" for long stretches, the worst possible adventure structure. Why, then, did my players pursue the whole series with fanatical zeal, refusing to take breaks, sharing with me some of the funnest games I have ever experienced? If "railroading" is so cruel to players, as has been alleged, why did my players willingly ride that train with such exhilaration, throwing fuel in to go top speed? These were not inexperienced players, either.

It makes me think that dogmatic schools of adventure module design are missing something, like the Forge theorists who said that you if you mixed game system design goals, your game was incoherent. Maybe that missing something is connected with the lack of actually trying the products they review.

So what business would I have reviewing game products?

Answer: As much as the next reviewer does, I guess, because, apparently, the only qualification is having played a lot of role-playing games. If it's normal to write about modules you've never played, I have as much right as anybody else. Frankly, if you have actually played using a specific product (one that you did not design yourself!), you have more right to review it than any of the regular reviewers. You should do it, too. Tell us not whether the product checks off all the boxes in a grading rubric system, but whether it worked in real use, in real play, with your actual group. Was it fun for you? Why?