[D] In cross-attention, why is Q taken from decoder, and K taken from the encoders output respectively?

I looked up in so many places but couldn't find an answer. What happens if we switch Q and K to be from the encoder and decoder respectively? Would it make any difference?

attention decoder difference encoder machinelearning