Hi Gstyl,
When opening the Excel file in SharePoint Online with local Excel application, the behavior of requiring your credential is expected. You can just type your credential in order to open the file.
However, based on my test with Office 365 p1 plan in Windows 8, IE10, Office Professional Plus, the Excel files can be opened in local Excel application. Also, I have run Office desktop setup from the Downloads page in Office 365 Portal.
To narrow down the root cause of the issue, you can refer to the following steps:
1. Check if the other Excel files or other Office documents in your Team Site cannot be opened in local application normally.
2. Add the URL of Team Site to Trusted Sites of IE10, and set Security level to Medium.
3. Sign out from your Team Site, and clear cache data and cookies of IE10, then try again.
Thanks,
Cherry Wang