mjhjmtu

In the campaign I run, it is frequently the case that one player will make an attack against an unsurprised enemy to start combat. For example:

DM: "At the end of the long hall, you see a large door, flanked by two
hobgoblins. They are alerted to your presence but do not move."

Ranger: "I shoot at the one on the left with my longbow."

In these situations, no one is surprised, and a party member has specifically stated that they will perform the action that brings us into combat and cues initiative order. But does this affect initiative order?

If everyone just rolls for initiative, then there is a very good chance that someone, maybe even an enemy, will have a higher initiative than the inciting adventurer. But that doesn't make a whole lot of sense when the combat wouldn't happen at all (not yet, anyway) if that adventurer didn't make an attack. And then the player who is playing the ranger feels like his cool idea was ignored when he rolls a crappy initiative.

I realize that those who roll a higher initiative than the inciting adventurer could take a non-combat option, such as Dodge, Help, or Dash, but I don't think the party would be a big fan of reducing the party's attack quantity for the first round..

I was thinking about making a house rule that gives such a player a +5 bonus on their initiative roll or that allows players to choose their initiative as long as it isn't higher than what they rolled or something like that, but is there already a mechanic in place for this kind of situation?

The initiative and surprise mechanics already cover this situation in full.

Surprise

The GM determines who might be surprised. If neither side tries to be stealthy, they
automatically notice each other.

In your example, both parties are clearly aware of each other. Though they may not be instantly hostile, the fact that strangers are advancing on them with weapons drawn (else how could the Ranger shoot one of the hobgoblins) means that they're going to at least be ready for action.

Initiative

Initiative determines the order of turns during combat. When combat starts, every participant makes a Dexterity check to determine their place in the initiative order. The GM makes one roll for an entire group of identical creatures, so each member of the group acts at the same time. The GM ranks the combatants in order from the one with the highest Dexterity check total to the one with the lowest. This is the order (called the initiative order) in which they act during each round.

It is possible that - though the ranger is the one who decided to kick off the action by aiming at a hobgoblin - their reactions are fast enough that they react to his hostile intent and start moving or counterattack.

There's no reason to believe combat wouldn't start without an arrow actually being fired. In a tense situation like the one described, the first sign of aggression - especially on the part of the party advancing on a guarded location - is sufficient to get everyone moving.

No special mechanic exists

Oftentimes with questions like these, it is helpful to remember that although initiative is a mechanic to simulate combat, presumably everything happens within the same 6 seconds.

So for your specific situation, let's assume the ranger decides he is going to shoot at the hobgoblin and kick off combat. Since no one is surprised, everyone can act. The hobgoblin rolls an 18 and the ranger rolls a 12, so the hobgoblin goes first.

Because everything happens simultaneously, the hobgoblin sees that the ranger is in the process of grabbing an arrow, raising their bow, or some other threatening thing and is reacting to that. Whether that reaction is to run forward and attack, flip the table to duck and cover, or something else is irrelevant; the hobgoblin is simply reacting to the threat of attack that was triggered by the ranger's intent to attack.

Once the ranger's turn comes around, they can then react to the hobgoblin's action. Perhaps running to get a better attack angle or something else, but from a RAW perspective, the ranger goes after the hobgoblin.

It is not the act of releasing the arrow that triggers the initiative roll, but rather the Ranger's intention to fire it.

In situations like this, when one player tries to perform a combat action, the usual way to handle it is to simply have everyone roll initiative. Anyone rolling better than the Ranger simply reacted quickly enough that they saw the Ranger readying their bow and nocking the arrow, and get to respond before he has time to loose.

It might make sense to allow the Ranger to roll his Initiative check with Advantage for being the one trying to trigger the fight, but since nobody is surprised it seems perfectly reasonable that the Hobgoblins (or the Ranger's allies) understand his intentions and happen to be just a bit quicker in their acting.