What does Security Mode Command do in 5G NAS?

It tells the UE which NAS ciphering and integrity algorithms to activate so the procedure can continue securely.

Is NAS Security Mode Command the same as RRC Security Mode Command?

No. NAS Security Mode Command is sent between the AMF and UE over NAS, while the RRC version is an access-stratum message between the gNB and UE.

What usually comes after Security Mode Command?

The usual next message is Security Mode Complete, followed by later registration or service messages such as Registration Accept.