mjhjmtu

A word for the unexpected reaction of laughing in a situation of complete hopelessness or laughing with sad undertones

Gallows humor is "humor that treats serious, frightening, or painful subject matter in a light or satirical way." It has the connotation of someone laughing at their own execution.

Urban Dictionary actually has a good summary:

gallows humor

Sense of humor that arises from stressful, traumatic or
life-threatening situations where death seems to be an inevitability.
Different from black humor in that comments of this sort are made by
the person/persons affected (i.e. about to die).

It is not always used to mean that the person is facing literal death, but the idea is the same. They are expressing amusement or using humor in the face of their own inevitable fate.

I think that generally the kind of laughter you're talking about could be called nervous laughter or a nervous chuckle. It looks like "nervous laughter" is an actual term that psychiatrists use to describe the kind of laugh people give when they're put in a stressful situation. There is even an entire article on Wikipedia dedicated specifically to this expression:

Unhealthy or "nervous" laughter comes from the throat. This nervous laughter is not true laughter, but an expression of tension and anxiety. Instead of relaxing a person, nervous laughter tightens them up even further. Much of this nervous laughter is produced in times of high emotional stress, especially during times where an individual is afraid they might harm another person in various ways, such as a person's feelings or even physically.

According to Oxford Living Dictionary (https://en.oxforddictionaries.com/)

it's
MIRTHLESSNESS.

This noun is formed from an adjective 'mirthless':

(of a smile or laugh)

: lacking real amusement and typically expressing irony.

// ‘he gave a short, mirthless laugh.'

hysteria

Psychiatry - a psychological disorder (not now regarded as a single definite condition) whose symptoms include conversion of psychological stress into physical symptoms (somatization), selective amnesia, shallow volatile emotions, and overdramatic or attention-seeking behavior. The term has a controversial history as it was formerly regarded as a disease specific to women. – Google

1a : a psychoneurosis marked by emotional excitability and disturbances of the psychic, sensory, vasomotor, and visceral functions without an organic basis. b : a similar condition in domestic animals. 2 : behavior exhibiting overwhelming or unmanageable fear or emotional excess. – MW

"hysterical laughter", in quotes, has almost half a million results.