One of our sites was migrated from Sharepoint 2007 to Sharepoint 2013. In 2007 it used FBA with custom login page. After migration user noticed strange behavior: sometime Sharepoint returned empty page or HTTP 500 Internal server error. During checking Sharepoint logs the following error was found there:
Exception of type 'System.ArgumentException' was thrown. Parameter name: encodedValue
at Microsoft.SharePoint.Administration.Claims.SPClaimEncodingManager.DecodeClaimFromFormsSuffix(String encodedValue)
at Microsoft.SharePoint.Administration.Claims.SPClaimProviderManager.GetProviderUserKey(IClaimsIdentity claimsIdentity, String encodedIdentityClaimSuffix)
at Microsoft.SharePoint.Administration.Claims.SPClaimProviderManager.GetProviderUserKey(String encodedIdentityClaimSuffix)
at Microsoft.SharePoint.Utilities.SPUtility.GetFullUserKeyFromLoginName(String loginName)
at Microsoft.SharePoint.SPGlobal.CreateSPRequestAndSetIdentity(SPSite site, String name, Boolean bNotGlobalAdminCode, String strUrl, Boolean bNotAddToContext, Byte UserToken, SPAppPrincipalToken appPrincipalToken, String userName, Boolean bIgnoreTokenTimeout, Boolean bAsAnonymous)
at Microsoft.SharePoint.WebControls.SPControl.EnsureSPWebRequest(SPWeb web)
at Microsoft.SharePoint.WebControls.SPControl.SPWebEnsureSPControl(HttpContext context)
at Microsoft.SharePoint.ApplicationRuntime.BaseApplication.Application_PreRequestHandlerExecute(Object sender, EventArgs e)
at Microsoft.SharePoint.ApplicationRuntime.SPRequestModule.PreRequestExecuteAppHandler(Object oSender, EventArgs ea)
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
When web application was migrated to 2013 it started to use claims-based authentication. Custom login page was re-implemented using the following authentication logic (there are many examples of custom login page for FBA when claims-based authentication is used. Most of them use more or less the same logic):
On lines 9-21 we authenticate user via FBA claims authentication provider (SPIisSettings.FormsClaimsAuthenticationProvider). If provided user credentials are incorrect, call to SPSecurityContext.SecurityTokenForFormsAuthentication() method will throw exception. If credentials are correct it will set FedAuth authentication cookies.
On line 23 there is call to FormsAuthentication.SetAuthCookie() method which sets previously used .ASPXAUTH cookies used by ASP.Net FBA authentication. Developer who implemented login page told that it was left there for backward compatibility, but was not able to remember which exact components required this cookies.
And as it turned out presence of both FedAuth and .ASPXAUTH cookies caused mentioned exception in SPClaimEncodingManager.DecodeClaimFromFormsSuffix(). When call to FormsAuthentication.SetAuthCookie() was removed, error disappeared. Hope that information will help someone.