mjhjmtu

Why do we interpret the accelerated expansion of the universe as the proof for the existence of dark energy?

The accelerated expansion only tells us that the Einstein field equation must contain a cosmological constant, but I can put the constant on either side of the equation. Usually we put it on the right hand side and interpret it as an additional contribution to energy, namely dark energy. But if I put it in the left hand side instead, I would have a theory of gravity that explains both the "small scale" success of general relativity and the large scale expansion of the universe, without needing an element like dark energy.

So why do we keep saying that there is this dark energy and trying to identify it as the vacuum energy while we could more elegantly (in my opinion) use a theory of gravity with a cosmological constant and explain all observations?

The accelerated expansion of the universe is not direct evidence for dark energy, i.e. a perfect fluid contribution to the stress-energy tensor with $w = -1$. Dark energy is just by far the simplest thing that fits the data well.

It's simple to cosmologists because they are used to dealing with matter in the form of perfect fluids, and dark energy is just another one. And it's simple to particle physicists because it can be sourced by a constant term in the Lagrangian, the simplest possible term.

At the classical level, the distinction you're making is not really important. A cosmological constant, which you call a "modification of gravity", amounts to adding a constant term to the Lagrangian. But the standard description of dark energy also amounts to adding a constant term to the Lagrangian. They're the exact same thing -- a constant is a constant, it doesn't come with a little tag saying if it's "from" gravity or something else. Neither is more elegant because functionally all the equations come out the same. It's a philosophical difference, not a real difference.

But the situation changes dramatically when you account for quantum effects. That's because we know that QFT generically produces vacuum energy, i.e. sources a constant term in the Lagrangian, whether or not we put that term in classically or not. So even if you do explain the accelerated expansion by some other mechanism, you have to explain why this one isn't in effect. This is a difficult argument to make, because the contribution from QFT is already too big even if you only trust it up to tiny energy scales like $1 text eV$!

Of course, there is room to work on alternative theories, such as quintessence and phantom energy; these are functionally different because they correspond to a perfect fluid with $w neq -1$. At present, observational constraints show that the acceleration can only be explained by one additional perfect fluid if that component has $w = -1$ to within about 20% accuracy. If you wish, you can think of these theories as a modification of gravity by just moving their contributions to the other side of the Einstein field equation.

My impression is that these theories simply ignore the QFT vacuum energy without explanation. That's the real elephant in the room, and probably the reason so few physicists work on explaining the accelerated expansion. We have an automatic mechanism to explain the expansion, and that mechanism appears to be $10^120$ times more powerful than it should be. It seems premature to start to consider additional mechanisms before understanding this one first, and there seems to be no possible understanding of it now except via the much-hated anthropic principle. The ultimate explanation of the expansion will be a job for physicists of another millenium.