Yes, all Time Tracker Team Users must necessarily be Contacts in your Salesforce Org. Before creating Time Tracker Users add all Users as Contacts. Then when you create Time Tracker Users, link them to the appropriate Contact records.
Also, each contact should have his/her own unique email ID. We use the email Id to send an OTP when the User has forgotten his/her Time Tracker password.