The Cavalier fighter (Xanathar's Guide to Everything, p. 30) gets the Unwavering Mark ability which lets you mark a creature and states (emphasis mine):

In addition, if a creature marked by you deals damage to anyone other
than you, you can make a special melee weapon attack against the
marked creature as a bonus action on your next turn. [...]

Regardless of the number of creatures you mark, you can make this
special attack a number of times equal to your Strength modifier
(minimum of once), and you regain all expended uses of it when you
finish a long rest.

In most cases, it would be fairly easy to figure out how many of these special attacks the Cavalier can make, because the Cavalier's Strength usually does not fluctuate. But a lot of things can happen during a busy adventuring day, for example, Strength damage by some nasty monster, some kind of transformation magic or the ingestion of mystical potions. All these things could temporarily alter the Cavalier's Strength. So my question is:

If the Cavalier's Strength changes, does the number of special attacks against marked creatures change as well?

I assume the answer is Yes, but I'm not sure how to handle it if the Cavalier's Strength increases or decreases for just a little while.

Yes, your number of uses would change

As I would interpret this, if your Strength mod goes up you get extra uses, and if the Strength mod goes down, they disappear.

The way to handle this with temporary changes is that as long as the changes last, your changed Strength modifier determines the number of uses you have between long rests.

Example of a changing Strength Mod

If you normally have a +3 mod and you have used it twice, and then your mod goes down to +2, you can't use it until your mod goes up again.

A bit later, a magical effect causes your Strength mod to go up to +4 - you can make 2 more uses.

If you use those 2, and then later your mod goes back to +3 (magical effect wears off) you can't use it as you have already used it 4 times.

If your mod goes back up to +4 again you still cannot use it as you have already used it 4 times since your last long rest.

If you want an extra use you need to get your Strength mod boosted to +5.

This would work similarly to changing your maximum hitpoints because of a changing Constitution mod:

If your Constitution modifier changes, your hit point maximum changes as well, as though you had the new modifier from 1st level. For example, if you raise your Constitution score when you reach 4th level and your Constitution modifier increases from +1 to +2, you adjust your hit point maximum as though the modifier had always been +2. So you add 3 Hit Points for your first three levels, and then roll your Hit Points for 4th level using your new modifier. Or if you’re 7th level and some effect lowers your Constitution score so as to reduce your Constitution modifier by 1, your hit point maximum is reduced by 7. (Basic Rules, p. 61).