If you are still using 2012R2 for ADFS and not in a position to leverage Access Control Policies you can leverage the Group SID setting in order to allow the specific group of users you want to authenticate to the IDP.
Directions:
- Right click and select “Properties” of the IDP you want to edit.
- Select the “Issuance Authorization Rules” tab
- Select add new rule and select ” Group SID (Browse)”, choose the group you want to use.
- If exists, remove permit all rule thereafter
While this does work, existing SSO/SAML sessions may take a few hours to expire if this was just implemented. Access Control Policies are easier to use and can be combined with various conditions without needing to resort to complex syntax operations.