My party (lvl 7) is investigating a demon portal, and we try to obtain as much info as possible before entering it. We learned that it is kept open by the magical force of something or someone, and it will probably close when we destroy that person or object.
We've come to believe that the object might be the heart of an NPC. To make a long story short, she was cursed to do stuff for an evil God, and probably summoned some demons right before we broke the curse. Those demons opened the gate, but the NPC doesn't remember specifics of the time she was cursed, probably to repress all the horrible things she did with undead and stuff.

We felt her pulse, and she has none. But still, my party is in doubt - it might be because she has no heart, or it might be that she has a heart but it is just not beating.

If our dwarf cleric uses her locate object spell to locate the nearest human heart, that should be able to determine whether the NPC has a heart if she is the closest human, right? Or would the cleric have to have seen an actual human heart?

RAW, no this would not work. However...

...your DM may rule otherwise because it's clever and cool.

Remember that spells only do what they say they do, and Locate Object says:

Describe or name an object that is familiar to you. You sense the direction to the object's location, as long as that object is within 1,000 feet of you. If the object is in motion, you know the direction of its movement.

The spell can locate a specific object known to you, as long as you have seen it up close--within 30 feet--at least once. Alternatively, the spell can locate the nearest object of a particular kind, such as a certain kind of apparel, jewelry, furniture, tool, or weapon.

This spell can't locate an object if any thickness of lead, even a thin sheet, blocks a direct path between you and the object.

So we have to meet the conditions for the spell to work, and the big one here is object.

In D&D 5E, an object is explicitly defined as (emphasis mine):

...a discrete, inanimate item like a window, door, sword, book, table, chair, or stone, not a building or a vehicle that is composed of many other objects.

So even if the heart in question WAS an inanimate object (which it doesn't sound like it is), it might be hard to pull off since it's part of a greater whole (the NPC).

Using the spell the way you describe also undermines the 4th-level spell Locate Creature, which is explicitly intended to locate living things.

However...

All that said, it seems like the scenario you find yourselves in lends itself to using the Locate Object spell in this way, and it may have been your DM's intention for you to do this. Depending on the circumstances, I might "rule of cool" that your suggested use of the spell might work, but I'd make the DC high, since we aren't meeting the various other conditions like the caster having seen the heart, etc.

IF the heart is separate, however, you might be able to pull this off, but your caster would still have to have a pretty good idea of what the object looks like.

So ultimately, it will come down to your DM, but RAW, no, this does not work if the heart is inside the NPC.

An example where this might work: if the heart is artificial, like made of stone, AND if your caster is familiar with it, it may be a valid target for this spell.

It's not that easy to use the locate object for this purpose, but it might just work.

The spell can be used in two ways:

The spell can locate a specific object known to you, as long as you have seen it up close—within 30 feet—at least once. Alternatively, the spell can locate the nearest object of a particular kind, such as a certain kind of apparel, jewelry, furniture, tool, or weapon.

Since you haven't seen that specific heart up close, you are limited to the second use case, i. e. finding the nearest object of a kind. There can be some discussion on how exact can you be with specififying the kind but for this to work best, you would want to go with something like "find the closest [NPC's name] heart". But I don't think that's a reasonable kind of object.

What would be a useful (and reasonable) kind of object for this purpose? Exactly what you suggest – [NPC's race] heart (a human heart). For this to work, the person casting the spell would have to be of a different race (in this case non-human, otherwise the heart found would just be the caster's one). So the dwarf is a good choice here. For best results, have another member of the same race (the NPC's race, so a human) close by, but in other direction (unless you actually expect that the heart is kept nearby and could be located with the same casting of the spell).

Is human heart even an object, though? This is tricky. If you view the heart (as your DM might) as an inseparable part of a living thing, it is probably not a valid target for the spell. This may especially be the case if you compare locate object with locate creature (thanks Carcer for mentioning this), as using locate object (a 2^nd level spell) to locate a heart (or brain or whatever else) of a specific creature would allow it to entirely replace a 4^th level spell. So ask your DM to be sure.

But not all is lost. If a human heart is not a valid target while in a living human, it might certainly be a valid target if it's kept separately. So in your case, if the heart is kept somewhere for the purposes of gatekeeping, you will find it (in fact it's probably going to be the only targetable exemplar of human heart). But only if it is within 1000 feet (and not on the other side of your demon portal).

Keep in mind that your efforts can still be foiled with a even thin sheet of lead.