If you are still using 2012R2 for ADFS and not in a position to leverage Access Control Policies you can leverage the Group SID setting in order to allow the specific group of users you want to authenticate to the ...
Open Group Policy Management Console ( Start > Run > gpmc.msc). Locate Domain Controllers OU and find Default Domain Controllers Policy. Edit Default Domain Controllers Policy. Expand Computer Configuration > Policies > Windows Settings – Security Settings > User Rights ...