Saturday, June 21, 2014

FBA users management with pagination on SQL server side in Sharepoint. Part 1

Although in this article I describe FBA in relation with Sharepoint most of described ideas and concepts will be also useful if you work with different applications, e.g. with common ASP.Net web applications.

Forms based authentication (FBA) is one of the authentication types available for web applications (concept of forms authentication when users identify themselves by providing username and password is applicable in different types of applications, but in this post I will mostly refer to web applications). With FBA users physically may be located in different storages. Most frequent storage type is SQL Server, although they may be located e.g. in AD (and accessible via LDAP) or in custom storage (e.g. in the file on file system). For SQL server there are ready for use database script prepared by MS, which creates db with all necessary tables for FBA, and .Net classes SqlMembershipProvider and SqlRoleProvider for working with this database from the code (for AD scenario classes are ActiveDirectoryMembershipProvider and AuthorizationStoreRoleProvider). Here is the schema of aspnetdb:

image

This model is quite flexible and may be used in lot of scenarios including those applications which work with very big number of users. However in this case you may encounter with performance problems e.g. when list all users in user management administration. It is possible to use paging of course but if you will do it in-memory (i.e. inside of your application, when all users are retrieved from the database and logic of calculating and displaying correct page is implemented in the code), there won’t be real performance improvement. You may add caching, but first load anyway may be very slow.

Problem become even worse if you use FBA in more complex applications which have own user management system like Sharepoint and you need to add additional info to each user retrieved from SQL Server from this system as well. In this part I will show how to solve 1st problem by using paging on SQL Server side and in next part 2nd part will be handled on example with Sharepoint (update 2014-06-22: second part is published here).

SQL Server side paging is not new idea. Nowadays SQL Server handles it with ROW_NUMBER T-SQL function which became available from 2005 version of SQL Server. However SqlMembershipProvider doesn’t use it (instead it uses temp tables) and those can be used in earlier versions of the SQL Server. Let’s see how we may use it in our FBA application for improving performance.

First of all let’s see interface of MembershipProvider class which is base class for SqlMembershipProvider and all other providers (it can be also used as base class for FBA with custom storage. In this case you need to implement all its abstract methods by yourself):

image

If we want to optimize user listing we are interested in the following methods:

   1: public abstract MembershipUserCollection FindUsersByEmail(string emailToMatch,
   2:     int pageIndex, int pageSize, out int totalRecords);
   3:  
   4: public abstract MembershipUserCollection FindUsersByName(string usernameToMatch,
   5:     int pageIndex, int pageSize, out int totalRecords);
   6:  
   7: public abstract MembershipUserCollection GetAllUsers(int pageIndex,
   8:     int pageSize, out int totalRecords);

As you can see they have 3 parameters which are used for pagination:

- pageIndex
- pageSize
- totalRecords (output parameter)

In SqlMembershipProvider class these methods use the following stored procedures:

- aspnet_Membership_FindUsersByEmail
- aspnet_Membership_FindUsersByName
- aspnet_Membership_GetAllUsers

They are quite similar, let’s see how aspnet_Membership_GetAllUsers is implemented:

   1: CREATE PROCEDURE [dbo].[aspnet_Membership_GetAllUsers]
   2:     @ApplicationName       nvarchar(256),
   3:     @PageIndex             int,
   4:     @PageSize              int
   5: AS
   6: BEGIN
   7:     DECLARE @ApplicationId uniqueidentifier
   8:     SELECT  @ApplicationId = NULL
   9:     SELECT  @ApplicationId = ApplicationId FROM dbo.aspnet_Applications
  10:         WHERE LOWER(@ApplicationName) = LoweredApplicationName
  11:     IF (@ApplicationId IS NULL)
  12:         RETURN 0
  13:  
  14:  
  15:     -- Set the page bounds
  16:     DECLARE @PageLowerBound int
  17:     DECLARE @PageUpperBound int
  18:     DECLARE @TotalRecords   int
  19:     SET @PageLowerBound = @PageSize * @PageIndex
  20:     SET @PageUpperBound = @PageSize - 1 + @PageLowerBound
  21:  
  22:     -- Create a temp table TO store the select results
  23:     CREATE TABLE #PageIndexForUsers
  24:     (
  25:         IndexId int IDENTITY (0, 1) NOT NULL,
  26:         UserId uniqueidentifier
  27:     )
  28:  
  29:     -- Insert into our temp table
  30:     INSERT INTO #PageIndexForUsers (UserId)
  31:     SELECT u.UserId
  32:     FROM   dbo.aspnet_Membership m, dbo.aspnet_Users u
  33:     WHERE  u.ApplicationId = @ApplicationId AND u.UserId = m.UserId
  34:     ORDER BY u.UserName
  35:  
  36:     SELECT @TotalRecords = @@ROWCOUNT
  37:  
  38:     SELECT u.UserName, m.Email, m.PasswordQuestion, m.Comment, m.IsApproved,
  39:             m.CreateDate,
  40:             m.LastLoginDate,
  41:             u.LastActivityDate,
  42:             m.LastPasswordChangedDate,
  43:             u.UserId, m.IsLockedOut,
  44:             m.LastLockoutDate
  45:     FROM   dbo.aspnet_Membership m, dbo.aspnet_Users u, #PageIndexForUsers p
  46:     WHERE  u.UserId = p.UserId AND u.UserId = m.UserId AND
  47:            p.IndexId >= @PageLowerBound AND p.IndexId <= @PageUpperBound
  48:     ORDER BY u.UserName
  49:     RETURN @TotalRecords
  50: END
  51: GO

The idea is very simple here: it creates temporary table with auto incremented identity index column and userId (lines 22-27), then stores all userIds into this temp table ordered by user name (lines 29-34). As result we have own index table where first column contains number of row where appropriate user is stored. Last step is quite simple: initialize total records by number of affecting users (line 36) and return those users which correspond to the passed page index and page size by making cross-table query to database tables and built temporary table (lines 38-49). Stored procedures aspnet_Membership_FindUsersByEmail and aspnet_Membership_FindUsersByName use the same idea with additional filtering login by user name and email.

Now when we know how it works in backend, let’s prepare front end. In Sharepoint for displaying users list we may use convenient SPGridView class (in ASP.Net case you may use GridView control). For enabling paging we need to set AllowPaging property to true on the page where control is added and override its CreateDataSourceSelectArguments() method. Overriding can be done in custom class:

   1: public class CustomGridView : SPGridView
   2: {
   3:     protected override DataSourceSelectArguments CreateDataSourceSelectArguments()
   4:     {
   5:         int startRow = this.PageSize * this.PageIndex;
   6:         return new DataSourceSelectArguments(this.SortExpression, startRow,
   7:             this.PageSize);
   8:     }
   9: }

Here we only provide correct values for start row and page size. Without it SPGridView always passes 0 in start row. Then we need to implement DataSourceControl and DataSourceView classes which can be done like that:

   1: public class CustomFBADataSource : DataSourceControl
   2: {
   3:     private DataSourceView view;
   4:      
   5:     public CustomFBADataSource()
   6:     {
   7:         this.view = null;
   8:     }
   9:     protected override DataSourceView GetView(string viewName)
  10:     {
  11:         if (this.view == null)
  12:         {
  13:             this.view = new CustomFBAUsersView(this, viewName);
  14:         }
  15:         return this.view;
  16:     }
  17: }

For DataSourceView’s inheritor implementation is more complex:

   1: public class CustomFBAUsersView : DataSourceView
   2: {
   3:     public CustomFBAUsersView(IDataSource owner, string viewName):
   4:         base(owner, viewName)
   5:     {
   6:     }
   7:  
   8:     protected override IEnumerable ExecuteSelect(DataSourceSelectArguments args)
   9:     {
  10:         // get site details
  11:         var site = SPContext.Current.Site;
  12:         var settings = CodeFiles.Utils.GetFBAIisSettings(site);
  13:         if (settings == null)
  14:             return null;
  15:  
  16:         if (args.MaximumRows == 0)
  17:         {
  18:             return null;
  19:         }
  20:         // get paged users from sql database
  21:         int pageNumber = args.StartRowIndex/args.MaximumRows;
  22:         int totalRecords;
  23:         var membershipUsers = Membership.GetAllUsers(pageNumber, args.MaximumRows,
  24:             out totalRecords);
  25:         if (membershipUsers.Count == 0)
  26:         {
  27:             return null;
  28:         }
  29:  
  30:         var users = new DataTable();
  31:         users.Columns.Add("Name");
  32:         users.Columns.Add("Email");
  33:         users.Columns.Add("Active");
  34:  
  35:         foreach (MembershipUser membershipUser in membershipUsers)
  36:         {
  37:             var row = users.NewRow();
  38:             row["Name"] = membershipUser.UserName;
  39:             row["Email"] = membershipUser.Email;
  40:             row["Active"] = membershipUser.IsApproved ? "Yes" : "No";
  41:             users.Rows.Add(row);
  42:         }
  43:  
  44:         // add fake rows to allow switch pages
  45:         if (args.MaximumRows < totalRecords)
  46:         {
  47:             // fill previous pages
  48:             for (int i = 0; i < pageNumber; i++)
  49:             {
  50:                 for (int j = 0; j < args.MaximumRows; j++)
  51:                 {
  52:                     var row = users.NewRow();
  53:                     users.Rows.InsertAt(row, 0);
  54:                 }
  55:             }
  56:  
  57:             // fill following pages
  58:             var totalPages =
  59: Math.Ceiling(((decimal) totalRecords)/((decimal) args.MaximumRows));
  60:             for (int i = pageNumber + 1; i < totalPages; i++)
  61:             {
  62:                 for (int j = 0; j < args.MaximumRows; j++)
  63:                 {
  64:                     var row = users.NewRow();
  65:                     users.Rows.Add(row);
  66:                 }
  67:             }
  68:         }
  69:  
  70:         var dataView = new DataView(users);
  71:         return dataView;
  72:     }
  73: }

First of all it contains necessary logic for retrieving paged users information from SQL database (lines 20-42). It became possible after implementing of CreateDataSourceSelectArguments() method in inheritor of SPGridView control above, because after that argument passed in ExecuteSelect() method in DataSourceView got correct values for start row and maximum rows. But this is only half of all work. In order to be able to switch between pages we also need to make grid view “think” that there are also other pages, and amount of these pages (both preceding and subsequent related to the current page) should be exactly as in SQL Server. In order to do it we add fake empty rows to the resulting DataView object (lines 45-68).

After these manipulations we will get quite fast users listing which will work with good performance even if we have many users in our FBA database. In next part we will add information to each returned user from Sharepoint which will be done with Camlex library – open source library for building dynamic CAML queries via C# lambda expressions. With such tasks flexibility of Camlex is revealed on its full power. See you on the next part (update 2014-06-22: second part is published here).

Thursday, May 29, 2014

Add preview for pages with friendly urls in search results in Sharepoint 2013

Search preview is one of the nice features which was introduced in Sharepoint 2013. It gives possibility to see the target page without redirecting on that page. Preview may work for basic html pages or for documents (e.g. content of Word, Excel, Powerpoint documents can be shown in preview window). Some time ago we faced with one strange problem, which is platform bug from my point of view: for pages with friendly urls search preview doesn’t work. I.e. if for pages which have regular structural urls (http://example.com/pages/foo.aspx) preview panel looked like this:

image

then for pages with friendly urls defined in managed metadata term store (e.g. http://example.com/foo) it looked like this:

image

The key for solving this problem is in display template used for the page. Or to be more accurate, display template for hover panel used for the item. As you probably know for most items there are 2 display templates:

  1. for rendering item in search results;
  2. for hover panel for rendered item.

First “basic” display template has reference on second “supporting” display template in its code. By default Sharepoint uses standard Item_CommonItem_Body.html display template for items in search result (it may be overridden for different content types by search result types). Some time ago I wrote about how to modify this display template for fixing another Sharepoint problem, see Problem with cut titles in search results in Sharepoint 2013. In this article I will continue to work with the same display template, however you may use this approach with any other display template.

First of all let’s see from where problem with missing search preview for pages with friendly urls comes from. In order to do this we will need to check the code of Item_CommonItem_Body.html. First of all here is how javascript callback function is defined (this callback function is called when you click on the item in search result, see below):

   1:  
   2: var showHoverPanelCallback = ctx.currentItem_ShowHoverPanelCallback;
   3: if (Srch.U.n(showHoverPanelCallback)) {
   4:     var itemId = id + Srch.U.Ids.item;
   5:     var hoverId = id + Srch.U.Ids.hover;
   6:     var hoverUrl = "~sitecollection/_catalogs/masterpage/" +
   7:         "Display Templates/Search/Item_Default_HoverPanel.js";
   8:     showHoverPanelCallback = Srch.U.getShowHoverPanelCallback(itemId, hoverId,
   9:         hoverUrl);
  10: }

Later in the code it is used like this:

   1: <div id="_#= $htmlEncode(id + Srch.U.Ids.body) =#_" class="ms-srch-item-body" onclick="_#= showHoverPanelCallback =#_">
   2:     ...
   3: </div>

i.e. in onclick attrbiute of the item’s div. Second place where it is used is a link with item’s title:

   1: var titleHtml = String.format('<a clicktype="{0}" id="{1}" href="{2}"' +
   2:     'class="ms-srch-item-link" title="{3}" onfocus="{4}" {5}>{6}</a>',
   3:     $htmlEncode(clickType), $htmlEncode(id + Srch.U.Ids.titleLink),
   4:     $urlHtmlEncode(url), $htmlEncode(ctx.CurrentItem.Title), 
   5:     showHoverPanelCallback, appAttribs,
   6:     Srch.U.trimTitle(title, maxTitleLengthInChars, termsToUse));

Here it is used in onfocus attribute for the link which points to the target page.

Let’s check javascript code which gets callback function again. At first it retrieves it from ctx.currentItem_ShowHoverPanelCallback property and stores it to showHoverPanelCallback variable. Then it passes showHoverPanelCallback to Srch.U.n() function which is defined in Search.ClientControls.js like this:

   1: Srch.U.n = function Srch_U$n(obj) {
   2:     return SP.ScriptUtility.isNullOrUndefined(obj);
   3: }

and if result is true (i.e. if callback in ctx.currentItem_ShowHoverPanelCallback is null or undefined), it creates callback by itself. And it uses the following display template for the hover panel: ~sitecollection/_catalogs/masterpage/Display Templates/Search/Item_Default_HoverPanel.js.

Item_Default_HoverPanel.js is automatically generated javascript file from OTB display template Item_Default_HoverPanel.html. The problem is that this display template doesn’t support preview, i.e. it doesn’t have necessary code which loads the page into iframe and shows it. So if it is used, it technically can’t show preview. In order to show preview for pages another display template should be used: Item_WebPage_HoverPanel.html. It includes all necessary code and iframe for displaying the page like shown on the picture in the beginning of the article.

My first thought was just to replace Item_Default_HoverPanel.js on Item_WebPage_HoverPanel.js in the code above. However as it turned out code, which constructs callback function, is not called for most of the items (moreover it has a bug, because itemId variable defined as itemId = id + Srch.U.Ids.item will be incorrect. It will end with “_item_item” and will point to not-existent html element). It means that Srch.U.n() function returned false, i.e. callback was defined in ctx.currentItem_ShowHoverPanelCallback for most of the items. I added tracing with console.log() for ctx.currentItem_ShowHoverPanelCallback and found the following:

for pages with regular structural urls it contained something like this:

   1:  
   2: EnsureScriptParams('SearchUI.js', 'HP.Show',
   3: 'ctl00_SPWebPartManager1_g_2dd8e1bb_d0f9_486a_8daa_0e775d6c05cc_csr2_item', 
   4: 'ctl00_SPWebPartManager1_g_2dd8e1bb_d0f9_486a_8daa_0e775d6c05cc_csr2_hover',
   5: '~sitecollection\u002f_catalogs\u002fmasterpage\u002fDisplay Templates\u002f' +
   6:     'Search\u002fItem_WebPage_HoverPanel.js', false);

while for pages with friendly urls it shows this:

   1: EnsureScriptParams('SearchUI.js', 'HP.Show',
   2: 'ctl00_SPWebPartManager1_g_2dd8e1bb_d0f9_486a_8daa_0e775d6c05cc_csr2_item',
   3: 'ctl00_SPWebPartManager1_g_2dd8e1bb_d0f9_486a_8daa_0e775d6c05cc_csr2_hover',
   4: '~sitecollection\u002f_catalogs\u002fmasterpage\u002fDisplay Templates\u002f' +
   5:     'Search\u002fItem_Default_HoverPanel.js', false);

The key difference is in display templates used for hover panel: in first case Item_WebPage_HoverPanel.js is used which supports previews, while in second case Item_Default_HoverPanel.js is used where previews are not supported. After that another solution came to my mind: we need to construct callback function by ourselves for all items using Item_WebPage_HoverPanel.js display template. But how to do it? As callback is defined in ctx.currentItem_ShowHoverPanelCallback (which technically may be done on server side or in earlier steps in call stack on the client side), it may be hard to intercept the code where it is created. Much simpler to do it directly in display template.

Here is how it can be done. Instead of code shown above we need to use another code:

   1: var showHoverPanelCallback = ctx.currentItem_ShowHoverPanelCallback;
   2:  
   3: // change hover callback to WebPage for all items for showing preview
   4: try {
   5:     var idPrefix = id;
   6:     var suffix = "_item";
   7:     var idx = idPrefix.indexOf(suffix, idPrefix.length - suffix.length);
   8:     if (idx != -1) {
   9:         idPrefix = idPrefix.substring(0, idx);
  10:     }
  11:     var itemId = idPrefix + Srch.U.Ids.item;
  12:     var hoverId = idPrefix + Srch.U.Ids.hover;
  13:     var hoverUrl = "~sitecollection/_catalogs/masterpage/Display Templates/" +
  14:         "Search/Item_WebPage_HoverPanel.js";
  15:     showHoverPanelCallback = Srch.U.getShowHoverPanelCallback(itemId, hoverId,
  16:         hoverUrl);
  17:  
  18:     // we can access top-level div only asynchronously
  19:     AddPostRenderCallback(ctx, function(){
  20:         var itemContainer = document.getElementById(itemId);
  21:         itemContainer.setAttribute("onmouseover", showHoverPanelCallback)
  22:     });
  23: } catch (ex) {
  24:     console.log("Error occured in common item display template:" + ex);
  25: }

First of all we create correct ids of the divs for item itself and hover panel (lines 5-12). As I wrote above, code in OTB display template is incorrect: id variable which is used as a base already contains “_item” prefix, so we don’t need to add “_item” and “_hover” to it directly. After that for hoverUrl we use Item_WebPage_HoverPanel.js display template which supports previews. The last interesting part is how we attach created callback to the parent div, which is used as container for the item in search results. We can’t do it synchronously in the code, because this DOM element is not ready on the moment when the code is called. So we add it asynchronously by adding custom post render callback (lines 19-21).

After that you need to reupload Item_CommonItem_Body.html display template to /_catalogs/masterpage/Display templates/Search folder and publish it with major version. If you have several site collections you need to do it on all site collections separately. And the last thing: if you don’t want to change OTB display template, but instead want to create custom based on it and change this custom template, then in order to ensure that this display template is used on your search results page, you need to assign it to the ItemBodyTemplateId property of the ResultScriptWebPart on the target page for search requests:

   1: <property name="ItemBodyTemplateId" type="string">
   2:     ~sitecollection/_catalogs/masterpage/Display Templates/Search/Item_CustomCommonItem_Body.js
   3: </property>

After that it should use your custom display template for the search items.

That is all which I wanted to write about fixing problem with search previews for pages with friendly urls in Sharepoint 2013. In future articles I will also show how to enable search preview for list items in search results, which is also not available by default.

Friday, May 23, 2014

Disable sleep mode and hibernate on Windows 8.1

In order to disable sleep mode on Windows 8.1 you need to open Power Options window (you may just click new Start button in the left bottom corner and type it in common Search box). By default you will have Balanced plan selected:

image

After that click “Change plan settings”. You will see the following window:

image

On this window you need to set “Put the computer to sleep” setting to Never (on example above it is already set to it). If editing will be disabled you will need to click “Change settings that are currently unavailable” link. You may change setting both for “On battery” and “Plugged in” options or only for one of them. Also you may disable turning off the display here. After that click “Save changes” button.

Many people stop on this step, but that’s not all. After saving changes click “Change advanced power settings” link and expand Sleep tree node:

image

Here you also may need to click “Change settings that are currently unavailable” link before it will be possible to change settings. Ensure that in “Sleep after” both settings are set to Never and in “Hibernate after” also both settings should have Never. By default in “Hibernate after’ in “Plugged in” it will have 180 minutes. You need to decrement it to 0 using up-down control, it will show Never when you will reach 0. Also check that under Hard drive “Turn off hard disk after” is also set to Never.

After that save all changes. Your Windows 8.1 computer should not go to sleep or hibernate anymore.

Friday, May 16, 2014

Fix “Unable to process Put message” error when add mapping to user profile property to AD attribute

In one of my previous posts I showed how to create custom AD attribute and map it to user profile property via PowerShell: Create custom AD attribute and map it to Sharepoint user profile property. Regardless of whether you work with custom or standard AD attribute when you add mapping you may get “Unable to process Put message” error. In order to fix it go to Central Administration > Services on Server and restart User profile service and User profile synchronization service ((stop and then start)):

image

Note that in order to restart User profile synchronization service you will need credentials used by this service for synchronizing profiles. So ensure that you have them before to stop it. After that run your script again, it should run without errors now.

Saturday, May 3, 2014

Camlex 3.6 and Camlex.Client 1.5 are released

Today I released new 3.6 version of Camlex open source library (and Camlex.Client 1.5 for managed client applications). For those who doesn’t know about this project yet, it is library which helps to build dynamic CAML queries (i.e. queries which are built in runtime based on some parameters) via C# lambda expressions and fluent interfaces. You may see a lot of examples on this page. Also in my blog you may see other related articles about Camlex.

In new release support for explicit cast boolean expressions was added. I.e. if before you have to write expression like this:

   1: string caml = Camlex.Query().Where(x => (bool)x["Foo"] == true).ToString();

which in turn produces the following CAML:

   1: <Where>
   2:   <Eq>
   3:     <FieldRef Name="Foo" />
   4:     <Value Type="Boolean">1</Value>
   5:   </Eq>
   6: </Where>

then now you may write in simpler way without comparing with true or false explicitly:

   1: string caml = Camlex.Query().Where(x => (bool)x["Foo"]).ToString();

This record is shorter and won’t cause problems with ReSharper which may suggest you to remove not needed from its point of view comparison (in previous version it would cause exception).

In order to produce CAML with comparing with false value, use not (!) operator:

   1: string caml = Camlex.Query().Where(x => !(bool)x["Foo"]).ToString();

which will produce:

   1: <Where>
   2:   <Eq>
   3:     <FieldRef Name="Foo" />
   4:     <Value Type="Boolean">0</Value>
   5:   </Eq>
   6: </Where>

The same simplified syntax is available in complex conditions with && and || operations:

   1: string caml = Camlex.Query().Where(x => ((bool)x["foo1"] && !(bool)x["foo2"]) ||
   2:     (bool)x["foo3"]).ToString();

will produce:

   1: <Where>
   2:   <Or>
   3:     <And>
   4:       <Eq>
   5:         <FieldRef Name="Foo1" />
   6:         <Value Type="Boolean">1</Value>
   7:       </Eq>
   8:       <Eq>
   9:         <FieldRef Name="Foo2" />
  10:         <Value Type="Boolean">0</Value>
  11:       </Eq>
  12:     </And>
  13:     <Eq>
  14:       <FieldRef Name="Foo3" />
  15:       <Value Type="Boolean">1</Value>
  16:     </Eq>
  17:   </Or>
  18: </Where>

Also appropriate change was added to reverse engineering feature which allows to get C# representation of CAML expression with Camlex library (i.e. opposite transformation comparing with what is shown above). I.e. if you will enter the last CAML query above in Camlex online service, you will also get short version of boolean cast expressions without explicit comparison with true or false. Camlex online service was created exactly for simplifying using of Camlex.Net library for developers which worked with regular string CAML queries before.

New packages can be downloaded from Codeplex project site or from NuGet with the following commands:

Install-Package Camlex.NET.dll
Install-Package Camlex.Client.dll

If you have any comments or ideas for further improvements don’t hesitate to post them on discussions page on project site.