Monday, June 28, 2021

Use SQL LIKE operator in CosmosDB

One of known pain of developers who works with Azure table storage is lack of SQL LIKE operator which allows to perform queries with StartsWith, EndsWith or Contains conditions. There is another NoSQL database available in Azure - CosmosDB. It is not free (although Table storage is also not free but is very cheap) but with some notes it supports LIKE operator.

The first thing you should do when create new CosmosDB instance is to select underlying API:

At the moment of writing this post the following APIs were available:
  • Core (SQL)
  • MongoDB
  • Cassandra
  • Azure table
  • Gremlin

If you will create CosmosDB with Azure table API - it will look the same as native table storage and will have the same 6 operations available for queries: <, <=, >, >=, =, <>. I.e. no LIKE operator still:


But if you will create it with Core (SQL) API it will look different (it will work with "documents" instead of tables and rows):

and which is more important will allow to use SQL-like API and queries from Microsoft.Azure.Cosmos nuget package. E.g. if we have Groups table we may fetch groups which have some user as member using the following query (here for simplicity we assume that members are stored in JSON serialized string):

string userName = ...;
string sqlQueryText = $"SELECT * FROM c WHERE c.members like '%{userName}%'";

var queryDefinition = new QueryDefinition(sqlQueryText);
var queryResultSetIterator = this.container.GetItemQueryIterator<Group>(queryDefinition);
var items = new List<Group>();
while (queryResultSetIterator.HasMoreResults)
{
    var currentResultSet = await queryResultSetIterator.ReadNextAsync();
    foreach (var item in currentResultSet)
    {
        items.Add(item);
    }
}

So if you need NoSQL database with support of LIKE operator consider using CosmosDB with Core (SQL) API.

Thursday, June 24, 2021

Compare performance of Azure CosmosDB (with Azure Table API) vs Table storage

As you probably know Table storage is basic NoSQL db option available in Azure. It is very cheap and used in many Azure components internally (e.g. in Azure functions or bots). From other side CosmosDB is powerful NoSQL db engine available in Azure which support several different APIs. You need to select this API when create CosmosDB instance and it can't be changed after that:

  • Core (SQL) 
  • MondoDB
  • Cassandra
  • Azure Table
  • Gremlin

I.e. if you have existing application which already uses Table storage - you may create CosmosDB with Azure Table API and switch to it without rewriting application code (although this is in theory. E.g. if you app targets .Net Framework you will most probably need to retarget it to .Net Standard or .Net Core. But this is topic of different post).

In our product we use Table storage and decided to compare performance of CosmosDB (with Azure table API) and native Table storage. For testing the following data was used:

  • table with about 100K rows
  • query against system PartitionKey field: PartitionKey eq {guid} (this is fast query itself because PartitionKey is indexed field)
  • each query returned about 300 rows from overall 100K rows
  • execution time was measured

Same test had been ran 10 times in order to get average execution time - for CosmosDB and Table storage separately. Here are results:

Test run# CosmosDB (msec) Table storage (msec)
1 3639 2825
2 3249 2008
3 3626 1613
4 3016 1813
5 2764 1942
6 2840 1713
7 4704 1776
8 2975 2036
9 3353 1808
10 2833 1699
Avg (msec) 3299.9 1923.3

I.e. average execution time for native Table storage was less than for CosmosDB with Azure Table API.

Friday, June 4, 2021

Issue with PnP JS sp.site.exists() call when page url contains hashes

Today we faced with interesting problem when tried to use sp.site.exists(url) method from PnP JS for determining whether site collection with specified url exists or not: all calls returned 403 with System.UnauthorizedAccessException:

During analysis I've found that it tried to make HTTP POST calls to the following url for all sites: https://{tenant}.sharepoint.com/sites/_api/contextinfo which is definitely not correct because https://{tenant}.sharepoint.com/sites is not correct url but managed path. Then we noticed that page where this code was executed contains #/ in url:

https://{tenant}.sharepoint.com/sites/mysitecol#/favorites

and it seems like PnP JS parses it incorrectly in this case. As workaround we checked which API is used inside sp.site.exists(url) call (/_api/SP.Site.Exists) and rewrote code with explicit API call instead:

let promise = context.spHttpClient.post(context.pageContext.site.absoluteUrl + "/_api/SP.Site.Exists",
  SPHttpClient.configurations.v1, {
  headers: {
	"accept": "application/json;"
  },
  body: JSON.stringify({
	url: url
  })
}).then((response: SPHttpClientResponse) => {
  response.json().then((exists: any) => {
	if (exists.value) {
	  // site exists
	} else {
	  // site doesn't exist
	}
  });
});

Hope that PnP JS will fix this issue at some point.

Update 2021-07-05: issue inside pnp js was fixed by adding initialization call:

sp.setup({
  spfxContext: this.context
});

See this discussion for more details.

Wednesday, May 26, 2021

Latest Az 6.0.0 module conflicts with PnP.PowerShell 1.3.0

Today new Az 6.0.0 module was released (we wait for this release since it should contain fix for this issue: Webapp:Set-AzWebApp doesn't update app settings of App service). However it introduced own problem: it conflicts with PnP.PowerShell 1.3.0. It is important in each order these 2 modules are imported: if PnP.PowerShell is imported before Az then call to Connect-AzAccount will throw the following error:

Connect-AzAccount : InteractiveBrowserCredential authentication failed:
Method not found: 'Void Microsoft.Identity.Client.Extensions.Msal.MsalCacheHelper.RegisterCache(Microsoft.Identity.Client.ITokenCache)'.
+ Connect-AzAccount
+ ~~~~~~~~~~~~~~~~~
    + CategoryInfo          : CloseError: (:) [Connect-AzAccount], AuthenticationFailedException
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand

Here is the minimal PowerShell code which allows to reproduce the issue:

Import-Module "PnP.PowerShell" -MinimumVersion "1.3.0"
Import-Module "Az" -MinimumVersion "6.0.0"
Connect-AzAccount

In order to avoid the error Az module should be imported before PnP.PowerShell:

Import-Module "Az" -MinimumVersion "6.0.0"
Import-Module "PnP.PowerShell" -MinimumVersion "1.3.0"
Connect-AzAccount


Tuesday, May 25, 2021

Camlex 5.3 and Camlex.Client 4.2 released: support for StorageTZ attribute for DateTime values

Today new versions of Camlex library (both for server side and CSOM) have been released: Camlex 5.3 and Camlex.Client 4.2. For client object model separate packages are available for Sharepoint online and on-premise:

Package Version Description
Camlex.NET.dll 5.3.0 Server object model (on-prem)
Camlex.Client.dll 4.2.0 Client object model (SP online)
Camlex.Client.2013 4.2.0 Client object model (SP 2013 on-prem)
Camlex.Client.2016 4.2.0 Client object model (SP 2016 on-prem)
Camlex.Client.2019 4.2.0 Client object model (SP 2019 on-prem)

In this release possibility to specify StorageTZ attribute for DateTime values was added. Main credits for this release go to Ivan Russo who implemented basic part of the new feature. So now when you create CAML query for DateTime values you may pass "true" into IncludeTimeValue() method which then will add StorageTZ="True" attribute for Value tag:

var now = new DateTime(2021, 5, 18, 17, 31, 18);
string caml = Camlex.Query().Where(x => (DateTime)x["Created"] > now.IncludeTimeValue(true)).ToString();

will generate the following CAML:

<Where>
  <Gt>
      <FieldRef Name="Created" />
      <Value Type="DateTime" IncludeTimeValue="True" StorageTZ="True">2021-05-18T17:31:18Z</Value>
  </Gt>
</Where>

Thank you for using Camlex and as usual, if you have idea for its further improvement you may post it here.

Thursday, May 20, 2021

One problem with bundling SPFx solution

If you run "gulp bundle" for your SPFx project and face with the following error:

No such file or directory "node_modules\@microsoft\gulp-core-build-sass\node_modules\node-sass\vendor"

Try to run the following command first:

node node_modules\@microsoft\gulp-core-build-sass\node_modules\node-sass\scripts\install.js

It will create "vendor" subfolder under "node_modules\@microsoft\gulp-core-build-sass\node_modules\node-sass" and will download necessary binaries there:

Downloading binary from https://github.com/sass/node-sass/releases/download/v4.12.0/win32-x64-64_binding.node                                                 
Download complete                                                                                                                                             
Binary saved to ...\node_modules\@microsoft\gulp-core-build-sass\node_modules\node-sass\vendor\win32-x64-64\binding.node

After that "gulp bundle" command should work.

Tuesday, May 11, 2021

New task group is not visible in edit Azure DevOps pipeline window

If you have created new task group in Azure DevOps and want to add it to your pipeline you may face with the problem that this task group won't be visible in Add task window (I'm talking now about using visual designer when edit pipeline, not by edit yaml file). Even if you stop edit and edit pipeline again - it won't appear. In this case try to click Refresh link on the top - it will fetch latest results from DevOps and your task should appear after that. Note that you should search by task group name: