I've run a lot of horror RPGs over the years but I have also played in many Horror campaigns. Ravenloft, Call of Cthulu, All Flesh Must Be Eaten, World of Darkness and even Ororsh from Torg are some of the games that stand out in my mind. They came with advice for running effective horror, often the advice was taken as gospel by new GMs or carried with vetern GMs for years. As a die-hard Ravenlodt GM I very much embraced what I call the Vincent Price approach to Horror GMing (lengthy and evocative descriptions, heavy on the melodrama and adjectives, etc). Over time however, I began to realize this approach wasn't always the most effective. In the hands of some GMs it backfired, performed before certain players it provoked giggles rather than shudder of fear. Part of the problem with this kind of advice is it focuses on what to do, it is a list of things to make a point of doing as you describe the game to your players. But it often fails to account for the things that make horror difficult to achieve in a role playing game. When we developed Horror Show, I paid close attention to the hurdles GMs face when trying to spook their players. Here is a partial list of my observations in no particular order.
GM ISN'T "THERE": We talk a lot about player immersion and how this can produce a deeper experience of the game, but we don't talk about GM immersion that much. This makes some sense, since the GM plays the role of the world and its inhabitants, he doesn't play a single character like players do. But when you describe a scene to your group, if you don't see it in your head, if you don't feel like your there it loses something (especially in horror game). Being there is an important part of bringing the horror setting to life and pointing out the vital details (not the needless ones) that inspire fear.
PLAYERS ARE TOO GUARDED OR CYNICAL: Have you ever noticed that some people respond to horror movies with fear and others with laughter? To feel fear at a horror movie you have to give in a little, allow it to frighten you. The same is true of a horror rpg. It isn't about being tough or wimpy, an intellectual or a fool; it is about giving into the thrill of being scared. To do that, you can't play with a sense of detachment, you need to be immersed in the experience. This involves immersion like the above hurdle, but it is more than that. It is about lowering your guard. Ultimately whether players are willing to give into the experience or not is beyond the GM's control, but this is a good thing to be aware of.
GM ISN'T OBSERVING PLAYER CUES: It is important to know what is working and what isn't in a horror game. Not paying attention to minor cues like expressions and body posture is a sure way to miss this. Players who are affraid, appear more tense and interested. They lean forward as you describe the action. Players who are not affraid appear bored, disinterested or even confident.
BAD ATMOSPHERE: Have you ever tried watching a horror movie in the day time with a bunch of friends or with the lights on? Have you noticed it is a lot more scary witht the lights out and when you are alone? The same thing with horror RPGs. Nothing kills a good game of Cthulu or Ravenloft like bright lights, Ipads and cheese doodles. Keep the distractions away, these are horror mood killers.
GM TRIES TOO HARD: Remember the Ravenloft advice I mentioned earlier? If it happens to fit your natural style, by all means go for it, but when a GM forces himself into a style that isn't his own in an effort to be scary, it usually doesn't work.
SCRIPT IMMUNITY: When a horror GM protects player characters from death he murders some part of the world. This is one of my few unbending rules of horror: death must be on the table. Players know when they have script immunity and providing it really does cheapen the threat. Even worse: only killing the PCs when they make a "stupid decision". This has actually been official advice in a number of gaming products and needs to come to an end in any horror campaign worth its salt. If no one ever dies, you are doing horror with kid gloves.
GM OVERKILL: By the same token, mercilessly slaughtering PCs or, worse, scripting their deaths ahead of time is just as inneffective in horror. Death should be on the table but it needs to be fair and determined by the dice (not GM fiat).
Well, those are the major hurdles I kept noticing. If anyone else has any to share, feel free to comment.