mjhjmtu

I have defined the commutator between two matrices as

comm[A_,B_]:=A.B-B.A

I have defined the nested commutator as

nestcomm[A_,B_,n_]:= ToExpression[ 
 StringRepeat["comm[a,", n] <> "b" <> StringRepeat["]", n] 
 ]

where $n$ indicates how many time the commutator must be nested.

Does exists a more simple way to define an $n$-time nested operation between two elements $A$ and $B$? I tried to use the Nest function but without success.

Redefine your comm function to take in and put out a pair of matrices, then Nest can be called easily:

comm[A_, B_] := A, A.B - B.A

Then just invoke Nest with the number of terms in the final position. For n=2:

Rest[Nest[comm, a, b, 2]]
a.(a.b - b.a) - (a.b - b.a).a

Would something like the following work?

comm[A_,B_,1] := A.B-B.A
comm[A_, B_, n_Integer?Positive] := comm[comm[A, B, n-1], B, 1]

For example:

comm[A, B, 2] //TensorExpand
comm[A, B, 3] //TensorExpand

A.B.B - 2 B.A.B + B.B.A

A.B.B.B - 3 B.A.B.B + 3 B.B.A.B - B.B.B.A

Use:

comm[A_, B_, n_Integer?Positive] := comm[A, comm[A, B, n-1], 1]

to obtain the same definition as in @bill's answer.