In a recent game at my table, the cleric wanted to use his shield to pin a ghoul against the wall in order to stop it hitting people and make it easier to hit.

The player was thinking of it like a grapple or pin, but those mechanics didn't make sense -- the grapple only stops a creature, but doesn't really inhibit its ability to attack or make it easier to hit; and the pin is only available with a feat the player doesn't have.

But it was a good story beat that I didn't want to refuse, so in the moment I just mechanically called for a shove to knock the ghoul prone, and in the narrative he had the ghoul pinned against the wall with his shield instead of actually prone. It wasn't exactly what we wanted, but it was the closest thing I could come up with.

Is there a better way to mechanically represent crushing your enemy against a surface?

Prone was an excellent choice

You said that the cleric wanted "to stop [the ghoul from] hitting people and make it easier to hit." The prone condition has the following two applicable features:

• The creature has disadvantage on attack rolls.


• An attack roll against the creature has advantage
  if the attacker is within 5 feet of the creature. Otherwise, the attack roll has disadvantage.

It might seem like the "prone" condition doesn't apply because the ghoul is not splayed out on the ground. But keep in mind that "prone" does not always mean knocked down: it's a spectrum of conditions that make movement awkward and inconvenience a creature at short range (while making it harder to hit at a distance). As an example, consider the following rules on Flying Movement (PHB, p. 191):

If a flying creature is knocked prone, has its speed reduced to 0, or is otherwise deprived of the ability to move, the creature falls unless it has the ability to hover or it is being held aloft by magic, such as by the fly spell.

This text indicates that a flying creature can be "prone" while still (supernaturally) flying, which wouldn't make much sense if "prone" only had its traditional meaning.

Jeremy Crawford has described the various ways that the Shove action can be interpreted as something other than a "shove" (like "a throw, slamming into the other person, and so on.") Similarly, "prone" condition could cover a number of conditions that are analogous without actually requiring the creature to be lying down on the ground.

There are other options

If the Cleric was trying ensure the ghoul was not only inconvenienced but also trapped, he could have tried to both Shove the ghoul prone and Grapple it. For example, if he had the Shield Master Feat, he could use the Attack action to grapple the ghoul (with a free hand), and then a bonus action to Shove the ghoul into the wall with your permission that this would mechanically make it prone. This would stop the ghoul from simply using half its movement remove the prone condition. Or if the cleric had the Grappler Feat, he could attempt to "pin" the ghoul to the wall, if it was already grappled by him (again, with a free hand), causing both him and the ghoul to become restrained.

But if the cleric didn't have a Feat or special ability that would permit him to render the ghoul restrained via an Action, I'd be hesitant to let him do so. You'd essentially be giving the cleric the ability to do with one attack what other characters can only do with several (or a Feat). If the cleric had only one attack available (as most classes of cleric do), I think that your decision to declare the ghoul to be "prone" was completely appropriate.

Grappled was the right choice

Grappled:

• A grappled creature’s speed becomes 0, and it can’t benefit from any
  bonus to its speed.


• The condition ends if the grappler is incapacitated (see the condition).


• The condition also ends if an effect removes the grappled creature from the reach of
  the grappler or grappling effect, such as when a creature is hurled away by the
  thunderwave spell.

Using official rules, you can take an action to grapple an opponent:

When you want to grab a creature or wrestle with it, you can use the
Attack action to make a special melee attack, a grapple. If you’re
able to make multiple attacks with the Attack action, this attack
replaces one of them. (...) Using at least one free hand, you try to seize the target
by making a grapple check instead of an attack roll: a Strength
(Athletics) check contested by the target’s Strength (Athletics) or
Dexterity (Acrobatics) check

In addition to grappling, you can give the ghoul disadvantage on its attacks, as it is pinned against the wall, and give attacks against it disadvantage, as the cleric is in the way. In the rules (thanks David Coffron), it says:

The GM can also decide that circumstances influence a roll in one
direction or the other and grant advantage or impose disadvantage as a
result.

This covers all the effects you were intending, follows all the rules, and is balanced.