THE CAMPAIGN PLAYTEST

Playtesting can be tricky for sure. Over the years I feel like I've gotten a lot better though I'm still always learning and working to improve things. One lesson is that you can't just get by on discrete play tests of mechanics through combat and isolated scenarios, or simply crunching the probabilities. Those are good ways to get some data, and you need to do them, but you also have to run the game the way it is meant to be played. You need to run campaigns, otherwise you miss things. Something that looks broken on paper is often less of an issue in the context of a regular ongoing campaign, something that seems perfect on paper or in a discrete playtest can have hidden problems you don't see unless it comes up over the course of a series of adventures. Campaigns are done over the long haul, so you need to get that wider point of view to find issues. 

This is something I've seen again and again. Inevitably the mechanics that I end up regretting most are almost always the ones that didn't come up enough in campaign play tests (this is why I now make sure new mechanics and ideas come up plenty during campaigns). To give an example in our current project, Wandering Heroes of Ogre Gate, we initially had a different method for dealing with Grudge Encounters. In principle the idea we had worked, in shorter play testing it worked, but it didn't connect with how we were playing the game well enough once the ball got rolling. Finding this issue was important because it resulted in a much better Grudge system that flows naturally with events in the game. 

Another issue that you notice in campaign play tests but not so much in targeted play tests is drift between what's in the rules and what happens at the table. Sometimes the game play naturally shifts away from what you have written down, and it can even be possible to overlook the difference if you are not paying attention. When this happens I usually take it as a red flag because the drift occurs for a reason. It could be a reason that indicates a change is needed in the rules (i.e. you are drifting because what you have on the page doesn't make as much sense as what you started doing at the table) but in some cases it could simply be due to other factors (drifting away from what you have on the page and toward something you are more familiar with). In targeted play tests this tends to happen less because you are focused on playing the mechanics as written. A mechanic that feels fine when you are devoted to using it often becomes much more annoying in the middle of an ongoing campaign where you are juggling a bunch of other things. If you only playtest and never experience the drift, then you might experience it after the game is released and end up playing differently than what your book actually says. 

Without the campaign playtest you can easily miss the heart of the game. It is one thing to devise specific scenarios or combats for players to test, it is quite another to hold your players' interest week after week. This helps you understand why people might play your game, what it does well and what is fun about it. This is clear once you begin writing your Gamemaster chapter. If you can't intelligently write a GM section for you system and setting, then you probably haven't done enough campaign play testing. 

The campaign helps you catch key oversights. Over the course of a campaign things come up that you might overlook if you are just play testing scenarios. With GMing you often don't realize what's missing until you need it. If you are not play testing the campaign, it is easy to forget to include the obvious in the book. You'll still forget things, you just will be a lot less likely to forget the stuff that matters.

Fundamentally I think this is about experiencing the game the way it is meant to be played. If you're experience of your system is done in a way that has nothing to do with how people who buy it will play it, then there is going to be a huge disconnect between your ideas and their implementation. You do need to do things that GMs and players probably won't (you will want spread sheets with data, probabilities hashed out, etc), that is still important. However the most important thing in my mind is to actually run the thing continuously over a long term campaign.