What does Relay Key Reject do in 5G?

It tells the UE that the relay key request failed and provides the cause for that failure.

Is Relay Key Reject part of ordinary registration?

No. It belongs to a specialized relay security procedure rather than the standard registration path.

What should engineers inspect first?

Start with the relay transaction identity and the 5GMM cause.