Universe B does not (yet) have magic mirrors.

Universe A magic mirrors do not connect with a corresponding Universe B magic mirror. Sort of like In Mary Poppins, Bert's sidewalk chalk painting does not connect with a sidewalk chalk painting at the destination - Bert, Mary and the kids enter the chalk painting and emerge in the countryside of the alternate dimension. The cartoon residents of that dimension cannot come into our dimension the same way. Your mirrors connect similarly - some way such that people come thru and go back the way they came, but not vice versa. B people cannot use this route to leave B.

In Mary Poppins they were pulled back to their own world when the chalk painting was destroyed by rain. Maybe something has to happen to the mirror in A to pull the travelers back through - like the lights go out. It would be a good story way to ensure you were not trapped - your magic mirror is set in the garden, and you have until sundown. Unless there are artificial lights...

-

Later in your story the Universe B people, realizing that magic mirrors are possible, eventually make their own. They too are one way and they are more Universe B; different, possibly sinister.

Because everyone knows they would die

In Universe A these mirrors are well known and studied by some very smart people. Even though the average Joe cannot easily obtain a mirror, it is well known to the average UA resident that if you try to bring anything living from Universe B back into UA it will die. Everyone knows this in UA, it's a simple fact that everyone has heard. I mean have you ever met anyone on the street not from this universe?

If anyone from UB asks to come with the characters back to UB, the character from UA will have to tell them that unfortunately it just is isn't possible. As everyone in UA knows, a UB'er will not survive the trip into UA. You can even add horrible details of how painful they heard the death was.

But....

Through a series of events someone from UB either decides to risk it, accidentally falls into the mirror, or is faced with a dire situation anyways, so hey why not. Much to their surprise they open their eyes to UA, and conveniently for them they are still breathing.

Turns out those smart people had a reason to misinform people, and due to the rarity of the mirrors, and the high cost of proving them wrong those smart people were successful.

The level of ambient magic in Universe B could simply not be high enough for its natives to pass through a portal. Natives of Universe A are not creating a connection when traveling back to Universe A, but are just dropping their current connection.

Perhaps the existance of the portals is actually bleeding magic from Universe A into Universe B, and eventually, the ambient magic of Universe B will rise high enough to allow B-to-A travel.

Disease.

Before travelling from Universe A to Universe B, you have to get a set of vaccines to prevent you from contracting common Universe B illnesses. If you do show signs of contagion, there are procedures in place for returning to the Universe A medical center and getting treatment in quarantine. All Universe A citizens are well informed that allowing a Universe B resident to travel into Universe A has the potential to trigger a plague that could kill off millions of people.

Later, you could have someone "find a cure" for the disease (or determine Universe A residents couldn't contract it in the first place), retracting the travel ban. Or you could determine that it's only actually contagious if you show certain signs within the last three weeks, or a Universe B resident stole a mirror and released the plague already, sending all Universe A citizens running to the doctor for a vaccine they wouldn't normally bother to get. At this point, why bother closing the barn door after the cows have gotten out?

Mirrors are linked to their travelers

Any one can safely travel through a Mirror from World A (I'll refer to all worlds merely by letter from now on) to B, and thence, back through that mirror return to A. However, Travelling through a Mirror from B will send the traveler to C. Unfortunately C is desolate, inhospitable and unpopulated. Normally anyone who travels there perishes since there are no mirrors to return to B through. Since no one ever comes back from B everyone just assumes it's lethal to travel by mirror originating from B. Eventually someone does bring a mirror through from B to C after which it's discovered that mirror travel from C to A is possible, and travelers from B can return from C to B through the one mirror that was brought there.

Universe B is quite literally the opposite version of Universe A. So much so, that everything in Universe B is in fact made of antimatter.

Antimatter is perfectly harmless with itself and behaves exactly like ordinary matter (other than the fact that they have positrons and antiprotons instead of electrons and protons). However, we all know that matter and antimatter don't mix: when such particles interact, they violently transform into pure energy.

So how can people from Universe A use the mirrors to explore Universe B? Well, the mirrors are more than mere portals between universes. A better analogy would be an airlock. Crossing the mirror from A to B applies a magical "antimatter shielding" to each individual particle, such that they behave exactly like antimatter particles.

When returning to Universe A, the shielding is stripped away and the ordinary matter behaves ordinarily once again.

Unfortunately, the mirrors were developed with this explicit "A $rightarrow$ B = wax on", "B $rightarrow$ A = wax off" rule. Therefore, if anyone or anything from Universe B tries to cross into Universe A, the mirror won't be able to place "matter shielding" on them... to horrendous effect.

Now, you can choose how secure the mirrors are regarding Universe B objects.

• They can be dangerously unsecure, such that objects and people CAN cross, but there are just rules forbidding it for the simple reason that if anything were to cross, it'd be the equivalent of multiple nuclear bombs the moment they crossed.¹ This would be the equivalent of the mirror going "oh, this object has no shielding to wax off, so ignore it".


• They can be secure, forbidding non-shielded (anti)matter from crossing back into Universe A. You can also choose how severe this security feature is:
  
  
  • It may be harmless to those attempting to cross: the mirror momentarily shuts down as soon as the first non-shielded molecule attempts to cross¹
  
  
  • It may be harmful: the mirror allows the molecules to enter,² but upon detecting that they are unshielded, destroys them. So if someone from Universe B puts their hand through the mirror, they'll no longer have a hand.
  
  
  • It may be lethal: the mirror allows entire objects/people³ to enter before determining whether they are unshielded. If so, the entire object/person is destroyed.
  
  

Later on, this problem is somehow overcome, such that people from Universe B can cross over to Universe A.

As others have suggested, this may be by people from Universe B developing their own mirrors. This would imply that members of each universe must use different mirrors.

Another option is that someone figures out how to "hack" the original mirrors to allow passage. You can once again decide how easy or hard this is:

• They just had to change the mirror's behavior from "A $rightarrow$ B = wax on", "B $rightarrow$ A = wax off" to "unshielded = wax on", "shielded = wax off". This implies that the antimatter shielding for ordinary matter can be trivially repurposed as ordinary-matter shielding for antimatter.


• They had to develop shielding suitable to make antimatter behave like ordinary matter, and then make the mirror's "code" more complex: "if A-matter, A $rightarrow$ B = wax on antimatter shielding, B $rightarrow$ A = wax off antimatter shielding; if B-matter, B $rightarrow$ A = wax on ordinary matter shielding, A $rightarrow$ B = wax off ordinary matter shielding".

¹ (yes, this footnote appears twice above) Some magic handwaving will be needed to explain why random air molecules from Universe B aren't a problem (either generating massive heat on the Universe A side or causing the mirror to constantly shut down). Or just ignore this point.

² Note the word here is "enter", not "cross". This implies that the mirror isn't a door, but a bridge between the two dimensions, such that objects actually don't go A $rightarrow$ B (or vice versa), but A $rightarrow$ mirror bridge $rightarrow$ B. The travel between both ends can be virtually instantaneous as far as anyone is concerned, but this bridge is the "safe zone" where the unauthorized unshielded antimatter can be safely destroyed before it can enter Universe A.

³ Some more magic handwaving (or authorial ignoring) will be needed to explain how the mirror identifies what constitutes an "entire object/person".

Universe A is 'higher' on the planar scale than Universe B

This means that, just like falling vs climbing, one-way travel is easier.

But things are attracted to their plane!

Meaning that, a person going from A to B creates an effect like a rubber band being stretched. So, when they want to return, this rubber band is enough to overcome the difference in 'height'.

So how is this resolved?

One of 3 ways:

• The rubberband effect allows stowaways. Someone realizes that passing a sufficiently heavy object (either literally heavy - a lead casket - or metaphysically - a trinket from a material more attuned to its home plane than most) from A to B, allows someone to grab on/get in and go from B to A. Their pull towards B and 'planar weight' is far too insignificant compared to their vessel's pull towards A.


• The planes are moving. People might be concerned to discover that world A is moving downwards compared to world B. The difference becomes small enough that B->A is possible at some point, and one day A->B might become impossible (and later yet, the same will happen to going back to B from A).


• The difference is not that big. Someone, through accident or study, discovers that placing a mirror at a point of sufficiently high altitude (literal or planar) in plane B and its exit in a place of low altitude in plane A can allow B->A as well, or perhaps even invert the limitations. Perhaps this only works now and not earlier because the planes are in fact moving, and this is just the first symptom of the above scenario.

The planar travel is possible only if some particular restraints are met.

Let's say that in the universe B there is a big abundance of metals in the environment, so that the majority of its inhabitants are slightly poisoned by heavy metals (think of mercury poisoning: it could also give an explanation of why the world B is a dystopic version of A).

Planar travel is possible only if the person has "low content" of metals, otherwise he/she would simply slam against the glass; people of A have no problem, but people of B would simply contain too much of forbidden substances to cross the mirror.
People of A, while in the universe B, would need to give big attention to what they eat, avoiding food that could contain metal, otherwise they couldn't come home. This would be a stimulus for the wizards of world A to improve the magic of the mirrors, reducing the limits that can prevent the travel, to the point that eventually even people of B will be able to cross the mirrors.

The Evil One

Once, long ago, a great evil arose in Universe A. After a Great War, in which many heroic deeds were done and many horrible things happened, the mages of Universe A united in one last, desperate attempt to defeat The Evil One (whose name shall not be mentioned). They were unable to kill this being, but managed to imprison him in his home realm, Universe B. This prison is secured with seven great seals, kept hidden lest they be discovered and destroyed, freeing The Evil One to wreak havoc upon Universe A once more.

Now, millenia later, the Great War is but a distant memory, and few people actually believe it happened, or that there ever was an Evil One. Dramatic progress has been made into dimensional travel, but the strange fact that only one-way travel to Universe B is not entirely understood. Experts suspect that it is caused by a magical shield, which is anchored somehow by physical objects, but they are not entirely sure how or why it was created. To solve this dilemma and to allow free passage and trade with the disadvantaged people of Universe B, they find an unlikely hero and his ragtag band of misfits, and task them with a great quest: to find a way to unite the peoples of Universe A and Universe B in harmony and great justice.

Meanwhile, a Great Evil broods and watches, yearning to once more spread his malign influence over the delicious feast once denied him...

One such possibility is that the laws of physics are different between the universes, and universe B biology is incompatible with universe A physics.

For example, suppose that Universe A does not have quantum tunnelling. Thus, the lifeforms there don't need it. If they travel to Universe B, they are largely unaffected. However, The lifeforms in Universe B depend on quantum tunneling. If they were to travel over, they would die, or face medical problems, as you decide.

This gets somewhat easier to justify if magic is involved. There could be incompatible magics, much like incompatible blood types.

Racism

Humans in universe B have all existing portals to universe A.

They refuse to travel to Universe A because they let all these subhuman species mingle together, also, they aren't terribly keen on letting their resident non-humans escape oppression via mass exodus through these mirrors, so the mirrors to Universe A are kept under close observation.

Likewise, Universe A isn't too keen on being a destination for refugees from this backwater hell hole of a universe - all the humans are bigoted jerks, and if they started taking in the non-humans it'd get to be a serious immigration problem. Similarly, if any of their own people went over, their either weirdo human supremacists, or they were insane enough to travel to a world where their own kind is totally oppressed, so in general Universe A isn't terribly keen on letting anyone from Universe B pop on over.

Balance between worlds

Balance is a common theme in fantasy writing. People expect it. The swing of the scales between good and evil, physical strength vs magical power, etc. Of course mirrors have to be powerful artifacts to counter this. Powerful enough that only the people in A have really mastered it.

One enters, one leaves

Mirrors can allow the balance to shift temporarily. When someone (or more specifically, up to X people, depending on the power of the mirror) portals A->B, it can't be used again until it portals X people back B->A.

The plot thickens

This means people have to be careful with mirrors. If you portal from A, the last thing you want is for people to know you're from A. You want to get in, do what you've got to do, and get out. B is a dangerous place, and the last thing you want is to be murdered for your mirror.

So this means it's not illegal or impossible for people to portal B->A, just hard and plot-dependent. Choose your own adventure from here :)

Disease Control

A long time ago, a B creature went to the A universe, but he carried a disease with him they are asymptomatic carriers, millions of persons died by the plague. Since then, no people from the B universe is allowed to travel to A universe.

People from A who travel to B universe are allowed to travel back after spending time on quarantine. If they don't die, it means they didn't get sick by the plague.

Obviously, this is just a legal restriction if you really want you could travel illegally, or make a vaccine.

Energy Levels

The universe A is at a higher level of energy compared to universe B.

Travel from A to B is free, since you already have the energy. Instead, travel from B to A requires an extraordinary amount of energy/magic.

When a traveller from A universe who is in B universe and want to come back to its original universe (A) also travel free, since he already has in himself the necessary level of energy. Maybe, when an A person travel to B, all the energy released due the lowering of its energy state is stored, so when they want to come back to its home they just must spend that already store energy.

If a B habitant wants to go to A he will need to spend huge amounts of energy/magic.

Think of high altitude acclimatization of mountain climbers, or people such as Nepali natives being accustomed to the oxygen deficiency (and their bodies able to compensate for it) far better than tourists who usually stay at sea level.

Universe A is a world where the ambient magic quality is far different from Universe B, which maybe started off similar but then went through an apocalypse that caused mana depletion.

When someone travels A -> B, they might find it harder to replenish magic from surroundings (though crystals etc. will continue to work). However, innately the living beings in A regulate the amount of magic they absorb from their environs so it doesn't overflow; while B natives have long lost the need and habit of doing so (because, there just wasn't enough to absorb).

Now, if someone from B travels to A they would literally die by overdosing on mana from the atmosphere.

 this restriction would need to be removed at some point 

You can later enable inter-universe travel from B to A in different ways, depending on how common & uncontrolled you want it:

1. Acclimatization chamber (on B) or decompression chamber (on A) : Facilities are built for B's residents to get slowly accustomed to higher pressure/purity of ambient mana so their bodies can adjust to it. These might be expensive and controlled by powerful factions.




2. Spells/ skills to improve self-regulation: Denizens of A learn to balance the mana inside & outside their bodies naturally, while growing up. Someone comes up with a practice regime for people in B to learn those skills before they travel to A.

They are missing a spacial dimension, such as in the tale of Flatland or Interstellar (2014 movie).

Members of Universe A can interact with Universe B by ignoring their extra spacial dimension(s), but members of B cannot gain the dimensions that would allow them to enter or perhaps even properly perceive A.

Explanation:

The "magical lock" is actually placed on mirrors in Universe B by a secret order of humans tasked with preventing the other "savage" non-human races from traveling over to Universe A and obtaining technology that would be able to obliterate humans.

Source of Magic:

The source of magic in each universe is the combined wisdom and magical ability of all magic-wielding inhabitants. Universe A has many more highly adept magic users, owing to its relatively stable and functional society.

The Lock:

The strength of the lock is directly proportional to the magic abilities of the lock creator(s) and maintainer(s). In the case of Universe B, the human lock builders were very powerful and the secret order has maintained knowledge of and skill with magics that have been otherwise lost to the rest of the inhabitants. There has to this point never been an individual or group of magic users with enough ability to override a lock (many have tried). Killing one or more of the secret order diminishes the strength of the locks until/unless the order is replenished.

Travel:

Travelers from A->B maintain a magical connection back to the point of origin that allows them to return from any mirror in Universe B. In all but the rarest of cases, they do need to keep a talisman/device on their person that taps into the source of Universe A magic. The talisman temporarily unlocks (but does not destroy) any Universe B lock. The talisman itself is magic- and gene-locked, which means it cannot open a mirror portal without a willing, living magic user (i.e., no chopping off a hand or holding a gun to the talisman-holder's head).

Resolutions:

• A human from the secret order goes haywire and (magically or otherwise) forces the other members to release one or more locks.


• A human from the secret order has a child with a non-human and over time, diplomacy prevails and the multiple species become allies. The locks are released.


• A human from the secret order falls in love with a non-human and gives them a way to "overthrow" the humans.


• An inhabitant of Universe A has a child with an inhabitant of Universe B. The child grows up with a connection to the source of magic in both universes. Whenever someone travels from A->B, the child is able to tap into the vast magic reserves of Universe A. Over time, they become the most powerful magic user in Universe B.


• An inhabitant of Universe A has a child with an inhabitant of Universe B. The mother or father or both are killed at some point, leaving behind a talisman. The child's magic "profile" and genes match those of its parent from Universe A, allowing it to use the talisman.

Anyway, variations on a theme. I hope something in here helps you.

The great interdict

Years ago, a world traveler from universe B brought a plague to Universe A.
Smallpox, the black death or something more exotic.

It wiped out a large fraction of the population, including the family of some of the post powerful mages.

To ensure it never happened again, a great and powerful ritual was enacted to block worldwalkers from entering universe A.

The ritual has a keystone that can be destroyed or individuals can be keyed into it or it can be otherwise bypassed by someone who knows about it and has access to the keystone.

Universe A might have a restrictive immigration policy

Universe A has had the magic mirrors for some time and has had several incidents of people coming from other universes already. For whatever reason, the government in Universe A has decided that it would like to heavily regulate or restrict immigration or tourism from other universes.

There are many potential reasons why any given polity would adopt a restrictive immigration policy. In no particular order:

1. Economic reasons (they're worried that they can't support more people / that them foreigners will took their jerbs!)




2. Public health concerns (does Universe B have diseases that nobody in Universe A has immunity to or expertise curing?)




3. Racism




4. Concerns about assimilation or culture (e.g. will people from Universe B speak the language and be able to find work?)

Alternatively, Universe B could have a restrictive emigration policy

Universe B's city sounds pretty awful, like it's run by a totalitarian regime of sorts. Totalitarian regimes often ban emigration to keep people from living somewhere better. Maybe they have a policy that prevents people from leaving their universe? Maybe the mirrors are totally illegal their and ownership of one is punishable by death or something horrific.

The (non-scientific) energy level of Universe A is higher than Universe B.

When someone from Universe A travels to Universe B either they have some sort of super power or their extra energy is locked in some manner.

Someone (or something) from Universe B cannot travel to Universe A unless something fills them with enough energy to exist (not just survive) in Universe A. They also have to survive the energy influx.

The mirrors could be tools for projection into another universe, rather than strictly "traveling". This could happen behind the scenes, or could be an explicit part of the magic involved in mirrors.

But for whatever reason, the no magic can truly connect two dimensions. They can only project things from the universe they're created in into another universe! Thus someone could walk into a mirror and appear on the other side - but there wouldn't really be a way to "get back" other than triggering the magic involved in the mirror's projection to cancel it.

This could imply that all traveling is temporary, and is broken, with the traveling person killed or returned, when the mirror is broken. Or the mirror's mechanics could involve projection through enchantment or some other special means so that characters traveling stay in the Universe B even when the mirror is broken.

Whichever way the specifics are chosen, this implies the mirrors only work to transport goods from the dimension they originate in to some other dimension. Then being able to travel from Universe A to Universe B could be as simple as "only Universe A's magic supports creating these mirrors".

Evolution of Conceptualization

That an item can be enchanted at all is a concept that requires people to evolve spiritually and mentally enough to enable the enchantment.

"A" people have spent countless generations becoming aware of this concept and gradually attained the ability at a societal level to easily enchant an item for travel to "B".

"B" people are in a younger universe and haven't yet evolved to that level of conceptualization needed to make the enchantment work for them... or at least they hadn't until plot progression event X occurred.

Perhaps event X was simply some "C" people that were tired of waiting for "A" people to next conceptualize travel to "C" and nudged things along. After all, all "C's" know that A + B tranposed by X = C.

Space is opaque in Universe B

Matter in Universe B does not allow the transmission of photons. All inhabitants of Universe B are blind. They have no mirrors and cannot use them.

Mirror travel is actually a magical tether.

In Stargate, to dial from A to B didn't mean you knew how to dial back from B to A. You needed the coordinates of where you were going and the symbol of origin for where you were coming from. In the original movie, the team was trapped for a while because Daniel didn't know what to punch in for the return trip.

So I think a reasonable explanation would be something like this:

Someone somewhere came up with this mirror device and has them programmed specifically with the coordinates to get to B. To facilitate a return trip, it also provides a tether for that specific object to then return from B back to A. However, an object going from B to A without this tether will require its own mirror, with its own magical formula -- the "coordinates" of getting to A. Until someone on B figures all this out and develops such a formula, there is simply no way for an untethered object to get from B to A. They don't have the coordinates. There is no B to A link.

This could be handy for some future plot devices too. Big Bad can't figure out how to dial from B to A but maybe he figures out how to steal someone's tether. Now that person is stuck in B and Big Bad gets a ticket to A.

The biochemistry of Universe A is non-chiral, but otherwise similar enough that someone from UA can live on food in UB. A person from UB (our universe, with a lot of chiral chemistry) can visit UA but either must bring food with him, or slowly starve.

https://en.m.wikipedia.org/wiki/Chirality_(chemistry)

You can make this more general -- UA people can eat nearly anything that can combine with oxygen to produce energy. UB people cannot.