Most of the time I see admins preventing or granting access only using SSHd config. For example using the AllowGroups directive. But that is only half the story. If you have a valid user, local or in a remote directory (LDAP), you can still login via the console, over serial, or using 'su'.

To prevent this, you can allow access to only specific users, groups and locations by configuring it in the /etc/security/access.conf file and modify /etc/pam.d/system-auth.

• Make sure you have the pam_access.so module enabled. You can configure this with authconfig.
• Add the configuration to /etc/security/access.conf. See examples below:

  # vim /etc/security/access.conf
  
  # Everyone in the group sysaccess can login from all locations.
  + : sysaccess : ALL
  
  # Only allow local root login.
  + : root : LOCAL
  
  # Allow local accounts to login local
  + : ALL : LOCAL
  
  # Deny everyone else
  - : ALL : ALL