Saturday, August 30, 2014

Assign values to managed metadata fields programmatically in sandbox solutions in Sharepoint

Assigning values to managed metadata fields programmatically in farm solution, which have full trust and access to Microsoft.SharePoint.Taxonomy namespace, is well known currently and don’t cause much problems:

   1: var field = (TaxonomyField)item.Fields["Foo"];
   2: var session = new TaxonomySession(site);
   3: var term = session.GetTerm(termId);
   4: field.SetFieldValue(item, term);

In this code we use assumption that term guid is already known (in farm solution it can be retrieved from terms collection by term label).

Also it is possible to set value to managed metadata field via client object model, but instead of Microsoft.SharePoint.Taxonomy assembly we should use Microsoft.SharePoint.Client.Taxonomy assembly:

   1: var ctx = new ClientContext(url);
   2: var list = ctx.Web.Lists.GetByTitle("MyList");
   3: var fields = list.Fields;
   4: var field = fields.GetByTitle("Foo");
   6: var query = new CamlQuery();
   7: string queryXml = string.Format(
   8: "<view>" +
   9: "  <query>" +
  10: "    <where>" +
  11: "      <eq>" +
  12: "        <fieldref name=\"ID\" />" +
  13: "        <value type=\"Integer\">{0}</value>" +
  14: "      </eq>" +
  15: "    </where>" +
  16: "  </query>" +
  17: "</view>", id);
  18: query.ViewXml = queryXml;
  19: var listItems = list.GetItems(query);
  21: ctx.Load(listItems, items => items.Include(i => i["Foo"]));
  22: ctx.Load(fields);
  23: ctx.Load(field);
  24: ctx.ExecuteQuery();
  26: if (listItems.Count != 1)
  27: {
  28:     return;
  29: }
  30: var item = listItems[0];
  32: var txField = ctx.CastTo<TaxonomyField>(field);
  33: var termValue = new TaxonomyFieldValue();
  34: termValue.Label = label;
  35: termValue.TermGuid = termId;
  36: termValue.WssId = -1;
  37: txField.SetFieldValueByValue(item, termValue);
  38: item.Update();
  39: ctx.Load(item);
  40: ctx.ExecuteQuery();

In this code we also assume that we know term guid – it is needed for proper creation of TaxonomyFieldValue object (lines 33-36). In case of client object model it is more complicated to get it. But e.g. if you need to copy value of one list item to another, you may get term guid from source item by casting it to string, which looks like this: “{wssId};#{label}|{guid}”.

The remaining question is to how set value of managed metadata field programmatically in sandbox solution? We know that sandbox solutions which contain code are now deprecated, but it is still may be needed in the real life. The problem is that sandbox solutions are partially trusted and don’t have access neither to Microsoft.SharePoint.Taxonomy.dll nor to Microsoft.SharePoint.Client.Taxonomy.dll. During investigations I didn’t find standard ways to do it, and only found one trick (or even hack) which works in sandbox. This trick is based on the fact that for each managed metadata field in Sharepoint there is hidden Note field (see e.g. Problem with not crawled managed metadata fields in Sharepoint 2013 for some details) which also contains string representation of the taxonomy value, but not in the form used in the managed metadata field itself (see above). In the Note field value is stored in the following form: “{label}|{guid}”. And the trick is that if we have source list item with specified value in managed metadata field, we may build string representation of taxonomy value in the form “-1;#{label}|{guid}” (we used “-1” for wssId here, which would mean that term is not in the TaxonomyHiddenList list yet. Even if it exists there I didn’t found problems with this approach – see How to Work with Managed Metadata Columns by Using the SharePoint Client Object Model) and then assign this string directly to managed metadata field. Also for safety it is better to assign Note field value itself:

   1: var val = sourceItem["FooTaxHTField"];
   2: if (val != null)
   3: {
   4:     targetItem["Foo"] = string.Format("-1;#{0}", val);
   5: }
   6: else
   7: {
   8:     targetItem["Foo"] = null;
   9: }
  10: targetItem["FooTaxHTField"] = val;
  11: targetItem.Update();

This approach may be problematic if you don’t have source item from which you need to copy the taxonomy value, because you will need to get term guid somehow and you can’t get it from terms collection in sandbox code. But if you have static structure of the terms you may provision them with known ids and then use these ids in the code. Anyway I hope that this information will be useful for you and will help in your work.

Wednesday, August 13, 2014

Internal details of Document ID feature activation or why Document ID is not added to content type after feature activation – Part 2

This is the 2nd part of series about internal details of Document ID activation, part 1 is here. We finished on the fact that you need to wait at least 0,5h or change system time before to run “Document ID enable/disable” job. Let’s continue investigation and will check what happens after that.

After activation of “Document ID Service” feature, Document ID settings page becomes available in Site settings page. After feature activation it will look like this:


As mentioned in part 1 now we will temporary change system time: add 1 hour to the current time and reset Sharepoint Timer service. After that go to CA > Monitoring > Job definitions > “Document ID enable/disable” job (for appropriate web application) and click Run now. If you will click on Running jobs after that you should see progress bar for "Document ID enable/disable" job (if you would run it before time specified in DeliveryDate column you would only see in Jobs history that "Document ID enable/disable job" ran 00:00:00, i.e. exited immediately after started):


If you will execute the same Sql query which we used in part 1 after job will be finished it should not return any rows:

   1: declare @enable uniqueidentifier
   2: set @enable = cast('749FED41-4F86-4277-8ECE-289FBF18884F' as uniqueidentifier)
   3: select * from [dbo].[ScheduledWorkItems] where [Type] = @enable

At this moment if you will check Document ID settings page it will look like this:


I.e. now it will have value in "Begin IDs with the following characters" field. But also it shows the warning:

Configuration of the Document ID feature is scheduled to be completed by an automated process.

It means that feature activation still not finished. It happens because “Document ID enable/disable” job creates another work item with type A83644F5-78DB-4F95-9A9D-25238862048C. In order to check it execute the following Sql query:

   1: declare @assign uniqueidentifier
   2: set @assign = cast('A83644F5-78DB-4F95-9A9D-25238862048C' as uniqueidentifier)
   3: select * from [dbo].[ScheduledWorkItems] where [Type] = @assign


It runs with the following params which is specified in TextPayload column of ScheduledWorkItems table:

   1: <?xml version="1.0" encoding="utf-16"?>
   2: <DocIdWorkitem
   3: xmlns:xsd=""
   4: xmlns:xsi="">
   5:   <fOverwriteExisting>false</fOverwriteExisting>
   6:   <Cookie>1</Cookie>
   7: </DocIdWorkitem>

This work item is processed by "Document ID assignment" job. This item is added by "Document ID enable/disable" job and sets values in Document ID field for all existing documents on the site. Here we also should pay attention on value in DeliveryDate column. It should be near the time when you forced "Document ID enable/disable" job on previous step. As we didn't change time back yet in order to execute this work item just force it in Central administration. You should see "Document ID assignment" job progress in Running jobs:


and after some time work item should be removed from ScheduledWorkItems table (you can check it by running Sql query shown above).

After that you may change system time back and reset Sharepoint Timer service. If you will now go to Site settings > Document ID settings it should not show red warning text anymore.

But "Begin IDs with the following characters" field will still contain the same random string:


If you need to change Document ID prefix to your own value, you will need to go through whole process again. After you will click Ok work item with type 749FED41-4F86-4277-8ECE-289FBF18884F with the following parameters:

   1: <?xml version="1.0" encoding="utf-16"?>
   2: <DocIdEnableWorkitem
   3: xmlns:xsd=""
   4: xmlns:xsi="">
   5:   <fEnable>true</fEnable>
   6:   <fScheduleAssignment>true</fScheduleAssignment>
   7:   <fOverwriteExisting>false</fOverwriteExisting>
   8:   <fDocsetCtUpdateOnly>false</fDocsetCtUpdateOnly>
   9:   <prefix>TEST</prefix>
  10: </DocIdEnableWorkitem>

(In this example TEST prefix is used). And work item with type A83644F5-78DB-4F95-9A9D-25238862048C and params:

   1: <?xml version="1.0" encoding="utf-16"?>
   2: <DocIdWorkitem
   3: xmlns:xsd=""
   4: xmlns:xsi="">
   5:   <fOverwriteExisting>false</fOverwriteExisting>
   6:   <Cookie>2</Cookie>
   7: </DocIdWorkitem>

After both jobs will be finished, add some document into document library. When you will click View properties, to should see Document ID field – link with text “TEST-1-1” which points to url:


I.e. now you can use OTB Document ID field in document content types. This is the end of the series. Hope that this information will help you in the work.

Internal details of Document ID feature activation or why Document ID is not added to content type after feature activation – Part 1

Document ID Service is OTB feature which adds Document ID field to OTB Document content type. All custom content types which inherit Document content type also will have it after correct feature activation. The main problem is that this field is added asynchronously via timer job daily (see below) and it causes a lot of confusions for developers and administrators. On Sharepoint Online it is problematic to force the process, but for on premise environments it is possible and in this article I will show how to do it and on what mechanisms it is based on. Also note that on Sharepoint Online DocId feature is only available with Enterprise plan.

Before to continue reading this article I highly recommend to check another my post Internal details of SPWorkItemJobDefinition or several reasons of why SPWorkItemJobDefinition doesn’t work, which describes details of internal implementation of work items in Sharepoint. It is needed because Document ID feature actively uses work items and we need to know how they work before to investigate Document ID feature itself. Work items which are used in the process of activation of Document ID feature are processed by the following timer jobs (you may find them in Central administration > Monitoring > Job definitions):

Job title: Document ID enable/disable
Job type: Microsoft.Office.DocumentManagement.Internal.DocIdEnableWorkItemJobDefinition
Work item type: 749FED41-4F86-4277-8ECE-289FBF18884F
Description: Work item that propagates content type changes across all sites when the Document ID feature is reconfigured.
Default schedule: Daily 21:30-21:45

Job title: Document ID assignment
Job type: Microsoft.Office.DocumentManagement.Internal.DocIdWorkItemJobDefinition
Work item type: A83644F5-78DB-4F95-9A9D-25238862048C
Description: Work item that assigns Document ID to all items in the site collection.
Default schedule: Daily 22:00-22:30

Both jobs are provisioned by DocumentManagement feature (Id = "3A4CE811-6FE0-4e97-A6AE-675470282CF2") of WebApplication scope. First job adds fields to content types, second – sets the value in added fields for existing documents. If you don't see these jobs in Central administration > Monitoring > Job definitions, activate this feature with force flag. Pay attention on work item types (749FED41-4F86-4277-8ECE-289FBF18884F and A83644F5-78DB-4F95-9A9D-25238862048C) – we will need them below.

First of all we need to activate "Document ID Service" feature in Site settings > Site collection features. Lets check what happens in feature receiver Microsoft.Office.DocumentManagement.DocIdFeatureReceiver during activation and deactivation:

   1: public override void FeatureActivated(SPFeatureReceiverProperties
   2: receiverProperties)
   3: {
   4:     FeatureActivateDeactivate(receiverProperties, true);
   5: }
   7: public override void FeatureDeactivating(SPFeatureReceiverProperties
   8: receiverProperties)
   9: {
  10:     FeatureActivateDeactivate(receiverProperties, false);
  11: }

Internal FeatureActivateDeactivate() method is shown below:

   1: private static void FeatureActivateDeactivate(SPFeatureReceiverProperties
   2: receiverProperties, bool fActivate)
   3: {
   4:     SPSite parent = receiverProperties.Feature.Parent as SPSite;
   5:     ULS.SendTraceTag(0x61326d37, ULSCat.msoulscat_DLC_DM, ULSTraceLevel.High,
   6:         "Document ID/FeatureActivateDeactivate: Entering  with fActivate = {0}",
   7:         new object[] { fActivate });
   8:     DocumentId.EnableAssignment(parent, null, fActivate, fActivate, false, false);
   9:     ULS.SendTraceTag(0x61326d38, ULSCat.msoulscat_DLC_DM, ULSTraceLevel.High,
  10:         "Document ID/FeatureActivateDeactivate: Leaving");
  11: }

Let’s check DocumentId.EnableAssignment() method:

   1: public static void EnableAssignment(SPSite site, string prefix, bool fEnable,
   2: bool fScheduleAssignment, bool  fOverwriteExistingIds, bool fDocsetCtUpdateOnly)
   3: {
   4:     if (site == null)
   5:     {
   6:         throw new ArgumentNullException("site");
   7:     }
   8:     if ((site.RootWeb == null) || (site.RootWeb.ContentTypes == null))
   9:     {
  10:         throw new ArgumentException("site.RootWeb.ContentTypes");
  11:     }
  12:     if (fEnable && !IsFeatureEnabled(site))
  13:     {
  14:         throw new InvalidOperationException(
  15: "Assignment cannot be enabled with the feature deactivated");
  16:     }
  17:     if (DocIdHelpers.IsSiteTooBig(site))
  18:     {
  19:         DocIdEnableWorkitem.ScheduleWorkitem(site, prefix, fEnable,
  20:             fScheduleAssignment, fOverwriteExistingIds, fDocsetCtUpdateOnly);
  21:     }
  22:     else
  23:     {
  24:         DocIdEnableWorkItemJobDefinition.EnableAssignment(site, prefix, fEnable,
  25:             fScheduleAssignment, fOverwriteExistingIds,  fDocsetCtUpdateOnly);
  26:     }
  27:     new DocIdUiSettings(fEnable, prefix).Save(site);
  28: }

So it check whether current site collection “too big” by using DocIdHelpers.IsSiteTooBig() method. If yes, it creates new work item which will add fields to content types and will create another work item which will assign values in these fields for existing documents (see below). If site is not “too big”, it will make changes in content types synchronously and then also will create work item for filling fields for existing documents. I.e. this second work item will be created in any case, but we will return to it later. Now let’s check what means “site is too big”. Here is how DocIdHelpers.IsSiteTooBig() method is implemented:

   1: public static bool IsSiteTooBig(SPSite site)
   2: {
   3:     return IsSiteTooBig(site, 1, 40, 20);
   4: }
   6: public static bool IsSiteTooBig(SPSite site, int maxWebs, int maxListsPerWeb,
   7:     int maxDoclibsPerWeb)
   8: {
   9:     ...
  10: }

I.e. site is “too big” when it contains more that 1 SPWeb, more that 40 lists or more than 20 document libraries. If one of these conditions is true for your site, content types will be changed asynchronously in work item. It is created in DocIdEnableWorkitem.ScheduleWorkitem() method as shown above. Here is how it is implemented:

   1: public static void ScheduleWorkitem(SPSite site, string Prefix, bool Enable,
   2:     bool ScheduleAssignment, bool OverwriteExisting, 
   3: bool DocsetCtUpdateOnly)
   4: {
   5:     if (site == null)
   6:     {
   7:         throw new ArgumentNullException("site");
   8:     }
   9:     string url = site.Url;
  10:     ULS.SendTraceTag(0x636e7037, ULSCat.msoulscat_DLC_DM, ULSTraceLevel.Medium,
  11: "Document ID Enable:ScheduleWorkitem for '{0}':enable={1},schedule={2},overwrite={3}",
  12:         new object[] { url, Enable, ScheduleAssignment, OverwriteExisting });
  13:     ULS.SendTraceTag(0x636e7038, ULSCat.msoulscat_DLC_DM, ULSTraceLevel.Medium,
  14: "Document ID Enable: ScheduleWorkitem called for site '{0}' by this stack: {1}",
  15:         new object[] { url, Environment.StackTrace });
  16:     XmlSerializer serializer = new XmlSerializer(typeof(DocIdEnableWorkitem));
  17:     StringWriter writer = new StringWriter(CultureInfo.InvariantCulture);
  18:     DocIdEnableWorkitem o = new DocIdEnableWorkitem();
  19:     o.fEnable = Enable;
  20:     o.fScheduleAssignment = ScheduleAssignment;
  21:     o.fOverwriteExisting = OverwriteExisting;
  22:     o.fDocsetCtUpdateOnly = DocsetCtUpdateOnly;
  23:     o.prefix = Prefix;
  24:     serializer.Serialize((TextWriter) writer, o);
  25:     string strTextPayload = writer.ToString();
  26:     writer.Dispose();
  27:     site.AddWorkItem(Guid.Empty, DateTime.Now.AddMinutes(30.0).ToUniversalTime(), 
  28: DocIdEnableWorkItemJobDefinition.DocIdEnableWIType, site.RootWeb.ID, site.ID, 1,
  29: false, Guid.Empty, Guid.Empty, site.RootWeb.CurrentUser.ID, null, strTextPayload,
  30: Guid.Empty);
  31:     ULS.SendTraceTag(0x636e7039, ULSCat.msoulscat_DLC_DM, ULSTraceLevel.Medium,
  32: "Document ID Enable: ScheduleWorkitem for '{0}' leaving", new object[] { url });
  33: }

Work item is added on lines 27-30 by calling SPSite.AddWorkItem() method. It adds work item which with type 749FED41-4F86-4277-8ECE-289FBF18884F which is processed by “Document ID enable/disable” job (see above). This work item propagates new Documend ID field. More accurately it propagates 2 fields are created and added to OTB Document CT: Document ID (SPFieldUrlValue) and Document ID Value (string). Both of these fields are not shown neither in Document content type details page in Site settings > Content types nor in Documents doclib columns in library settings. You will see it only when will upload some document to Documents doclib (or other doclib which uses OTB Document content type or custom content type which inherits it) and then will click View properties. Document ID will be there and it will look like link which points to{docId}.

If you read the article which I mentioned in the beginning you already know that work items are stored in ScheduledWorkItems table in content database. In order to check it execute the following Sql query:

   1: declare @enable uniqueidentifier
   2: set @enable = cast('749FED41-4F86-4277-8ECE-289FBF18884F' as uniqueidentifier)
   3: select * from [dbo].[ScheduledWorkItems] where [Type] = @enable

Query should return 1 row:


It has the following params specified in TextPayload column:

   1: <?xml version="1.0" encoding="utf-16"?>
   2: <DocIdEnableWorkitem
   3: xmlns:xsd=""
   4: xmlns:xsi="">
   5:   <fEnable>true</fEnable>
   6:   <fScheduleAssignment>true</fScheduleAssignment>
   7:   <fOverwriteExisting>false</fOverwriteExisting>
   8:   <fDocsetCtUpdateOnly>false</fDocsetCtUpdateOnly>
   9: </DocIdEnableWorkitem>

Pay attention on DeliveryDate column. It contains the earliest time in UTC format (add 3 hours in
order to get local time) when job can be executed. But if you will try to force job running (by clicking Run now in CA) before the time specified in DeliveryDate column, nothing will happen. This is the most often reason of confusion: force “Document ID enable/disable” job run immediately after feature activation. But as you can see from listing of DocIdEnableWorkitem.ScheduleWorkitem() method, DeliveryDate is set to DateTime.Now.AddMinutes(30.0).ToUniversalTime(). Which means that you have to wait at least 0,5h or change system time, restart timer server, run the job and then change time back and restart timer service again (to be more accurate you will need to run also “Document ID assignment” job before to change system time back right after “Document ID enable/disable” job is finished. See part 2 for details).

The reason of this behavior is in [dbo].[proc_GetRunnableWorkItems] stored procedure:

   1: CREATE PROCEDURE [dbo].[proc_GetRunnableWorkItems] (
   2:         @ProcessingId          uniqueidentifier,
   3:         @SiteId                uniqueidentifier,
   4:         @WorkItemType          uniqueidentifier,
   5:         @BatchId               uniqueidentifier,
   6:         @MaxFetchSize          int = 1000,
   7:         @ThrottleThreshold     int = 0,
   8:         @RequestGuid           uniqueidentifier = NULL OUTPUT     
   9:         )
  10: AS
  11:     SET NOCOUNT ON
  12:     IF (dbo.fn_IsOverQuotaOrWriteLocked(@SiteId) >= 1)
  13:     BEGIN
  14:         RETURN 0
  15:     END
  16:     DECLARE @iRet int
  17:     SET @iRet = 0
  18:     DECLARE @oldTranCount int
  19:     SET @oldTranCount = @@TRANCOUNT
  20:     DECLARE @Now datetime
  21:     SET @Now = dbo.fn_RoundDateToNearestSecond(GETUTCDATE())
  22:     DECLARE @InProgressCount int
  23:     DECLARE @ThrottledFetch int
  24:     DECLARE @ReturnWorkItems bit
  25:     SET @ReturnWorkItems = 0
  26:     BEGIN TRAN
  27:     SET @InProgressCount = 0
  28:     SET @ThrottledFetch = 0
  29:     SET @ThrottleThreshold = @ThrottleThreshold + 1
  30:     IF @ThrottleThreshold > 1
  31:     BEGIN
  32:         SET ROWCOUNT @ThrottleThreshold
  33:         SELECT 
  34:             @InProgressCount = COUNT(DISTINCT BatchId)
  35:         FROM
  36:             dbo.ScheduledWorkItems WITH (NOLOCK)
  37:         WHERE
  38:             Type = @WorkItemType AND
  39:             DeliveryDate <= @Now AND
  40:             (InternalState & (1 | 16)) = (1 | 16)
  41:     END
  42:     IF @BatchId IS NOT NULL
  43:     BEGIN
  44:         SET @ThrottledFetch = 16
  45:     END
  46:     IF @InProgressCount < @ThrottleThreshold
  47:     BEGIN
  48:         UPDATE
  49:             dbo.ScheduledWorkItems
  50:         SET
  51:             InternalState = InternalState | 1 | @ThrottledFetch,
  52:             ProcessingId = @ProcessingId
  53:         FROM
  54:             (
  55:             SELECT TOP(@MaxFetchSize)
  56:                 Id
  57:             FROM
  58:                 dbo.ScheduledWorkItems
  59:             WHERE
  60:                 Type = @WorkItemType AND
  61:                 DeliveryDate <= @Now AND
  62:                 (@SiteId IS NULL OR 
  63:                     SiteId = @SiteId) AND
  64:                 (@BatchId IS NULL OR
  65:                     BatchId = @BatchId) AND
  66:                 (InternalState & ((1 | 2))) = 0
  67:             ORDER BY
  68:               DeliveryDate
  69:             ) AS work
  70:         WHERE
  71:             ScheduledWorkItems.Id = work.Id
  72:         SET @InProgressCount = @@ROWCOUNT
  73:         SET ROWCOUNT 0            
  74:         IF @InProgressCount <> 0
  75:         BEGIN
  76:           EXEC @iRet = proc_AddFailOver @ProcessingId, NULL, NULL, 20, 0
  77:         END
  78:         SET @ReturnWorkItems = 1
  79:     END
  80: CLEANUP:
  81:         SET ROWCOUNT 0
  82:         IF @iRet <> 0
  83:         BEGIN
  84:             IF @@TRANCOUNT = @oldTranCount + 1
  85:             BEGIN
  86:                 ROLLBACK TRAN
  87:             END
  88:         END
  89:         ELSE
  90:         BEGIN
  91:             COMMIT TRAN
  92:             IF @InProgressCount <> 0
  93:                AND @InProgressCount <> @MaxFetchSize 
  94:                AND @WorkItemType = 'BDEADF09-C265-11d0-BCED-00A0C90AB50F'
  95:                AND @BatchId IS NOT NULL AND @SiteId IS NOT NULL
  96:             BEGIN
  97:                 UPDATE
  98:                     dbo.Workflow
  99:                 SET
 100:                     InternalState = InternalState & ~(1024)
 101:                 WHERE
 102:                     SiteId = @SiteId AND
 103:                     Id = @BatchId    
 104:             END            
 105:             IF @ReturnWorkItems = 1
 106:             BEGIN
 107:                 SELECT ALL
 108: DeliveryDate, Type, ProcessMachineId as SubType, Id,
 109: SiteId, ParentId, ItemId, BatchId, ItemGuid, WebId, UserId, Created,
 110: BinaryPayload, TextPayload,
 111:                     InternalState
 112:                 FROM
 113:                     dbo.ScheduledWorkItems
 114:                 WHERE
 115:                     Type = @WorkItemType AND
 116:                     DeliveryDate <= @Now AND
 117:                     ProcessingId = @ProcessingId
 118:                 ORDER BY
 119:                     DeliveryDate
 120:                 IF @@ROWCOUNT <> 0
 121:                 BEGIN
 122:                     EXEC @iRet = proc_UpdateFailOver @ProcessingId, NULL, 20
 123:                 END
 124:             END
 125:         END
 126:         RETURN @iRet
 128: GO

As you can see it returns items which have DeliveryDate <= @Now. That’s why we need to change system time in order to process work items.

You may stop here if you wanted to have quick solution for the problem of why Document ID field is not added. But if you would like to know more internal details, you should check part 2.