Hi Anthony,
To meet your requirement, you can do the following steps:
1. Remove the external user from the Team Site and Sub-Site. Please make sure that the external user is removed completely.
2. Cancelled inherit permissions. As shown in the following figures:
3. Set up new groups:
Select the Checkbox next to Create a new group:
4. Share Site:
5. Sent invitation to external user:
6: Invitation accepted:
Best regards,
Jerry Cen