What does Security Mode Reject do in 5G NAS?

It tells the network that the UE cannot accept the selected NAS security configuration from Security Mode Command.

What usually happens after Security Mode Reject?

The current procedure typically fails, moves into status or reject handling, or must be retried with a fresh context.

Is Security Mode Reject the same as Authentication Failure?

No. Authentication Failure belongs to the authentication challenge stage, while Security Mode Reject belongs to the later NAS security activation stage.