mjhjmtu

The author of a manuscript I'm editing is using the medcup macro in the MWE below. We've noticed the scaling it totally off if used inside the text command. Can anyone explain why?

documentclass[a4paper]memoir
usepackageamsmath%,scalerel}
usepackagegraphicx

defmedcup%
 operatornamewithlimits%
 mathchoicevcenterhboxscalebox1.2$bigcup$
 vcenterhbox$bigcup$%
 bigcup%
 bigcup%
 %


begindocument

[
 textInside cstext: $ medcup_n = 1^infty $
]

[
 textNot inside cstext: medcup_n = 1^infty
]

enddocument

When medcup is used in a display equation, it normally produces a scaled version of a text style bigcup. What is happening in your first equation, however, is that a scaled version of the display style bigcup is used instead because text causes a displaystyle command to be inserted in $bigcup$.

The text command is designed so that any equations in its argument are typeset in the current math style, which is displaystyle in this case.
Its definition contains a mathcoice, whose displaystyle branch is effectively given by

hboxeverymathdisplaystyleletf@sizef@sizeselectfont #1

Because this sets everymath=displaystyle, displaystyle will be inserted at the start of every equation inside the argument of text.
This includes the first $bigcup$ in the definition of medcup.
Your first equation therefore effectively becomes

[
 hboxInside cstext: 
 $displaystyle
 operatornamewithlimits
 vcenterhboxscalebox1.2$displaystylebigcup$
 _n = 1^infty
 $
 
]

and thus produces a scaled version of the display style version of bigcup, rather than the text style version.

To ensure that the text style version of bigcup is always used, you can explicitly insert a textstyle in the definition of medcup:

newcommand*medcup%
 operatorname*%
 mathchoicevcenterhboxscalebox1.2$textstylebigcup$%
 bigcup%
 bigcup%
 bigcup%
 %

(I also replaced the deprecated operatnamewithlimits by operatorname* and simplified the textstyle branch, as suggested by egreg in this comment.)

It has to do with the fact that [text$mathchoiceDTSs$] produces Displaystyle and not Textstyle.

Thus, adding textstyle in the definition is needed to overcome this behavior.

documentclass[a4paper]memoir
usepackageamsmath%,scalerel}
usepackagegraphicx

defmedcup%
 operatornamewithlimits%
 mathchoicevcenterhboxscalebox1.2$bigcup$
 vcenterhbox$bigcup$%
 bigcup%
 bigcup%
 %


begindocument

[
 textInside cstext: $textstyle medcup_n = 1^infty $
]

[
 textNot inside cstext: medcup_n = 1^infty
]

[text$mathchoiceDTSs$]
[bigcupscalebox1.2$bigcup$text$scalebox1.2$bigcup$$]
[bigcupscalebox1.2$bigcup$text$scalebox1.2$textstylebigcup$$]

defmedcup%
 operatornamewithlimits%
 mathchoicevcenterhboxscalebox1.2$textstylebigcup$
 vcenterhbox$textstylebigcup$%
 bigcup%
 bigcup%
 %


[
 textInside cstext: $textstyle medcup_n = 1^infty $
]

[
 textNot inside cstext: medcup_n = 1^infty
]
enddocument

The first two lines of output are from the OP's original definition. Then I demonstrate the display/text-style issue with the "D" line.

Then I show on the next line how the 3rd bigcup should be the same as the 2nd but isn't because of this behavior.

The next line fixes that by explicitly invoking textstyle.

Finally, I change the OP's definition, explicitly adding textstyle, to fix the overall issue.