In Sharepoint 2010 when anonymous user tries to open xlsx file the following error may occur:
At the same time old xsl files are opened well. Several solutions for this problems are mentioned there: http://support.microsoft.com/kb/2498047. However if they are not good for your scenario, you may use solution described below. The reason of the problem is the following: for several Excel files (by default they are xlsx, xlsb and xls) Sharepoint generates special link which points to the handler /_layouts/xlviewer.aspx?id=… instead of the link to original file. And when anonymous user tries to open xslx file, we get 401 Unauthorized exception because /_layouts/xlviewer.aspx page is not available for anonymous by default. In order to fix the problem we may do the following.
Excel files for which Sharepoint creates dynamic links are specified in serverfilesExcelServer.xml file which is located in C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\TEMPLATE\XML folder. By default it looks like this:
After we will comment line with xlsx extension, Sharepoint will use regular links which will point to files themselves:
And after that problem should disappear. Hope it will help you in your work.