5G NAS - Security Mode Reject Explained

Security Mode Reject is the NAS message the UE sends when it cannot accept the security configuration selected by the network in Security Mode Command. It is the failure branch of the NAS security procedure.

For beginners, the simple meaning is: the UE is telling the network that it cannot use the requested NAS security settings.
For engineers, this message is the clean breakpoint that separates successful authentication from failed NAS security activation.

What is Security Mode Reject in simple terms?

The network told the UE which NAS security settings to use. The UE could not accept those settings, so it sends Security Mode Reject instead of Security Mode Complete.

Why Security Mode Reject matters

This message matters because it tells you that the procedure did not fail in the identity stage or the authentication stage. It failed when the UE tried to activate the selected NAS security configuration.

That makes Security Mode Reject one of the most valuable branch markers in registration troubleshooting.

Where Security Mode Reject appears in the call flow

UE                              gNB / AMF
|<-- Authentication Request -----|
|--- Authentication Response ---->|
|<-- Security Mode Command ------|
|--- Security Mode Reject ------->|
|<-- Registration Reject / abort-|

It most commonly appears in registration or other 5GMM procedures where the network is activating or refreshing NAS security.

Transport characteristics

  • Direction: UE to AMF
  • Interface: N1
  • Transport on access side: commonly via UL Information Transfer
  • Security expectation: it belongs to the NAS security activation branch, so it should always be read together with the earlier Security Mode Command

What Security Mode Reject means operationally

Operationally, Security Mode Reject tells engineers that the UE refused to move into the selected protected NAS state.

The first useful checks are:

  • what 5GMM cause the UE reported
  • what algorithms and key context the network selected
  • whether the network replayed UE security capabilities correctly

Important Information Elements

IEWhy it matters
5GMM causeExplains why the UE rejected the NAS security activation and is the first field engineers should inspect.

Example message dump

Security Mode Reject
  Extended Protocol Discriminator: 5G Mobility Management
  Security Header Type: Plain NAS
  Message Type: Security Mode Reject
  5GMM Cause: UE security capabilities mismatch

How to read this dump

  • Start with the 5GMM Cause.
  • Then go back to the earlier Security Mode Command and compare the cause with the selected algorithms, ngKSI, and replayed UE capabilities.
  • After that, follow the later branch to see whether the procedure ends in Registration Reject, status handling, or a retry.

What to check in logs

  • verify that Security Mode Reject immediately follows Security Mode Command
  • inspect the 5GMM cause first
  • compare the reject with the selected NAS algorithms and key context
  • check for mismatch between actual UE support and replayed UE capabilities
  • correlate the rejection with the later procedure failure path

FAQ

What does Security Mode Reject do in 5G NAS?

It tells the network that the UE cannot accept the selected NAS security configuration from Security Mode Command.

What usually happens after Security Mode Reject?

The current procedure typically fails, moves into status or reject handling, or must be retried with a fresh context.

Is Security Mode Reject the same as Authentication Failure?

No. Authentication Failure belongs to the authentication challenge stage, while Security Mode Reject belongs to the later NAS security activation stage.

Summary

Security Mode Reject is the NAS message the UE sends when it cannot accept the NAS security configuration selected by the AMF in Security Mode Command.