Friday, March 18, 2022

Several ways to reduce JSON response size from Web API

If you have Web API which returns some data in JSON format at some point you may need to optimize its performance by reducing response size. It doesn't important on which technology Web API is implemented (ASP.Net Web API, .Net Azure Functions or something else). For this article the only important thing is that it is implemented on .Net stack.

Let's assume that we have an endpoint which returns array of objects of the following class which have many public properties:

public class Foo
{
    public string PropA { get; set; }
    public string PropB { get; set; }
    public string PropC { get; set; }
    public string PropD { get; set; }
    ...
}

By default all these properties will be returned from our API (including those properties which contain nulls):

[{
        "PropA": "test1",
        "PropB": null,
        "PropC": null,
        "PropD": null,
        ...
    }, {
        "PropA": "test2",
        "PropB": null,
        "PropC": null,
        "PropD": null,
        ...
    },
    ...
]

As you can see these properties with nulls still add many bytes to the response. If we want to exclude properties which contain nulls from response (which in turn may significantly reduce response size) we may add special class-level and property-level attributes to our class. Note however that it will help only if you serialize response with Json.Net lib (Newtonsoft.Json):

[JsonObject(MemberSerialization.OptIn)]
public class Foo
{
    [JsonProperty(NullValueHandling = NullValueHandling.Ignore)]
    public string PropA { get; set; }
    [JsonProperty(NullValueHandling = NullValueHandling.Ignore)]
    public string PropB { get; set; }
    [JsonProperty(NullValueHandling = NullValueHandling.Ignore)]
    public string PropC { get; set; }
    [JsonProperty(NullValueHandling = NullValueHandling.Ignore)]
    public string PropD { get; set; }
}

Attribute JsonObject(MemberSerialization.OptIn) means that it should serialize only those properties which are explicitly decorated by JsonProperty attribute. After that response will look like that:

[{
        "PropA": "test1",
    }, {
        "PropA": "test2",
    },
	...
]

i.e. it will only contain those properties which have not-null value.

As you can see response size got significantly reduced. The drawback of this approach is that it will work only with Json.Net lib (JsonObject and JsonProperty attributes classes are defined in Newtonsoft.Json assembly). If we don't use Newtonsoft.Json we need to use another approach which is based on .Net reflection and on the fact that Dictionary is serialized to JSON in the same way as object with the properties. Here is the code which does it:

// get array of items of Foo class
var items = ...;

// map function for removing null values
var propertiesMembers = new List<PropertyInfo>(typeof(Foo).GetProperties(BindingFlags.Instance | BindingFlags.Public));
Func<Foo, Dictionary<string, object>> map = (item) =>
{
    if (item == null)
    {
        return new Dictionary<string, object>();
    }
    var dict = new Dictionary<string, object>();
    foreach (string prop in selectProperties)
    {
        if (string.IsNullOrEmpty(prop) || !propertiesMembers.Any(p => p.Name == prop))
        {
            continue;
        }
		
        var val = item.GetPublicInstancePropertyValue(prop);
        if (val == null)
        {
            // skip properties with null
            continue;
        }
        dict.Add(prop, val);
    }
    return dict;
};

// convert original array to array of Dictionary objects each of which contains only not null properties
return JsonConvert.SerializeObject(items.Select(i => map(i)));

At first we get original array of items of Foo class. Then we define map function which creates Dictionary from Foo object and this dictionary contains only those properties which don't contain null value (it is done via reflection). Here I used helper method GetPublicInstancePropertyValue from PnP.Framework but it is quite easy to implement by yourself also:

public static Object GetPublicInstancePropertyValue(this object source, string propertyName)
{
    return (source?.GetType()?.GetProperty(propertyName,
            System.Reflection.BindingFlags.Instance |
            System.Reflection.BindingFlags.Public |
            System.Reflection.BindingFlags.IgnoreCase)?
        .GetValue(source));
}

With this approach we will also get reduced response which will contain only not-null properties and it will work also without Newtonsoft.Json:

[{
        "PropA": "test1",
    }, {
        "PropA": "test2",
    },
	...
]


Tuesday, March 15, 2022

Handle errors when load images on a web page

Regular html images (<img> tag) have quite powerfull mechanism of error handling and practice shows that it is not that well-known. E.g. if we have image on some web page (it may be any html web page regardless of underlying technology which rendered it) defined like that:

<img src="http://example.com/image.png" />

Now suppose that browser couldn't fetch this image because of some reason: e.g. image could not be found in specified url location (404 Not Found) or current user doesn't have permissions to this location (403 Forbidden). How to handle such situations and add graceful fallback logic (e.g. show some predefined image placeholder or use more advanced technique to fetch the image)?

There is standard html mechanism which allows to handle errors which may occur during loading of the images. Within img tag we may define error handler like that:

<img src="http://example.com/image.png" onerror="imageOnError()" />

In this case imageOnError() function will be called when error will occur during loading of the image. Knowing this technique we may implement graceful fallback for some scenarios when initial image was not successfully loaded because of some reason. E.g. some time ago I wrote an article where showed how to fetch image from Sharepoint site collection where current user doesn't have access: Return image stored in Sharepoint Online doclib from Azure function and show it in SPFx web part. If we will combine these 2 posts we may implement logic which by default shows images from predefined location (e.g. from Sharepoint doclib). But if current user doesn't have access to this doclib we may use onerror handler and fetch image via Azure function and app permissions and then show it in base64 encoded format. This is only one example how described technique may help to achieve more user friendly experience and provide functionality for end users which otherwise wouldn't be available.

Thursday, March 10, 2022

Disable PnP telemetry

PnP components send telemetry data to PnP team which helps to get more statistics for making various technical and architectural decisions. However sometimes you may want to disable it for the customer (e.g. due to GDPR or related topics). In this case you need to disable telemetry in the following components:

  • PnP.PowerShell
  • PnP.Framework
  • PnP.Core
  • pnp js

For disabling telemetry in PnP.PowerShell run the following command:

$env:PNPPOWERSHELL_DISABLETELEMETRY = $true

Note that it will disable PnP.PowerShell telemetry only in the current PowerShell session.

PnP.Framework itself doesn't send telemetry but it uses PnP.Core in some scenarios (e.g. modern pages API) which in turn sends telemetry. I.e. if you use PnP.Framework (and not create PnP.Core context by yourself) there is currently no option to disable it. However PnP team is working over it and probably soon there will be update about it. For disabling telemetry in PnP.Core you need to set property DisableTelemetry to false during creation of the context e.g. like this:

public class Startup : FunctionsStartup
{
    public override void Configure(IFunctionsHostBuilder builder)
    {
        var config = builder.GetContext().Configuration;
        var settings = new Settings();
        config.Bind(settings);
        builder.Services.AddPnPCore(options =>
        {
            options.DisableTelemetry = true;
            ...
        });
    }
}

For disabling telemetry in pnp js use the following code:

import PnPTelemetry from "@pnp/telemetry-js";

const telemetry = PnPTelemetry.getInstance();
telemetry.optOut();

Tuesday, March 1, 2022

Return image stored in Sharepoint Online doclib from Azure function and show it in SPFx web part

Imagine that we need to display image which is stored e.g. in Style library doclib of Sharepoint Online site collection (SiteA) on another site collection (SiteB) and that users from SiteB may not have permissions on SiteA. One solution is to return this image in binary form from Azure function (which in turn will read it via CSOM and app permissions) and display in SPFx web part in base64 format. With this approach way we can avoid SPO permissions limitation (assuming that Azure functions are secured via AAD: Call Azure AD secured Azure functions from C#).

At first we need to implement http-triggered Azure function (C#) which will return requested image (we will send image url in query string param). It may look like this (for simplicity I removed errors handling):

[FunctionName("GetImage")]
public static async Task<HttpResponseMessage> Run([HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = null)]HttpRequestMessage req, TraceWriter log)
{
	string url = req.GetQueryNameValuePairs().FirstOrDefault(q => string.Compare(q.Key, "url", true) == 0).Value;
	var bytes = ImageHelper.GetImage(url);
	
	var content = new StreamContent(new MemoryStream(bytes));
	content.Headers.ContentType = new MediaTypeHeaderValue(ImageHelper.GetMediaTypeByFileUrl(url));

	return new HttpResponseMessage(HttpStatusCode.OK)
	{
		Content = content
	};
}

Here 2 helper functions are used: one which gets actual image as bytes array and another which returns media type based on file extension:

public static class ImageHelper
{
	public static byte[] GetImage(string url)
	{
		using (var ctx = ...) // get ClientContext
		{
			var web = ctx.Web;
			var file = web.GetFileByServerRelativeUrl(new Uri(url).AbsolutePath);
			ctx.Load(file);
			
			var fileStream = file.OpenBinaryStream();
			ctx.ExecuteQuery();
			
			byte[] bytes = new byte[fileStream.Value.Length];
			fileStream.Value.Read(bytes, 0, (int)fileStream.Value.Length);
			return bytes;
		}
	}

	public static string GetMediaTypeByFileUrl(string url)
	{
		string ext = url.Substring(url.LastIndexOf(".") + 1);
		switch (ext.ToLower())
		{
			case "jpg":
				return "image/jpeg";
			case "png":
				return "image/png";
			... // enum all supported media types here
			default:
				return string.Empty;
		}
	}
}

Now we can call this AF from SPFx:

return await this.httpClient.get(url, SPHttpClient.configurations.v1,
	{
		headers: ..., // add necessary headers to request
		method: "get"
	}
).then(async (result: SPHttpClientResponse) => {
	if (result.ok) {
		let binaryResult = await result.arrayBuffer();
		return new Promise((resolve) => {
			resolve({ data: binaryResult, type: result.headers.get("Content-Type") });
	});
})

And the last step is to encode it to base64 and add to img src attribute:

let img = ...; // get image element from DOM
let result = await getImage(url);
img.setAttribute("src", `data:${result.type};base64,${Buffer.from(result.data, "binary").toString("base64")}`)

After that image from SiteA will be shown on SiteB even if users don't have access to SiteA directly.

Wednesday, February 23, 2022

How to enable DevTools for Microsoft Teams desktop client

If you develop apps for MS Teams (here we will use SPFx app running inside Teams) at some point you will most probably face with the need of debugging it in Teams desktop client. Of course in some scenarios you may use web client https://teams.microsoft.com/ and use regular browser developer tools (F12) for debugging however it is not always possible because some bugs may be reproduced only in desktop client. In this case you will need to find a way to debug them in desktop client.

The most simple approach is to use good old window.alert() across the code. Without browser console accessible in desktop client it will help to understand what happens in the code execution flow. But if you had experience with that (i.e. if you are old enough when everybody used it for debugging :) ) you probably know that this is quite boring and time consuming approach.

More powerful way to debug apps in Teams desktop client is to use DevTools for Microsoft Teams. For enabling it we first need to switch Teams client to Developer preview mode. It can be done from 3 dots in the top right corner > About menu:

After that you will get nice looking "P" (preview) icon added to your logo in top right corner of the window :) Also you will be able to open DevTools window by right click on MS Teams icon in System tray:

It will open DevTools table similar to those used in browsers (with console tab, network tab, etc):

which will greatly simplify debugging of the apps in Teams desktop client.

Wednesday, February 16, 2022

Disadvantages of Azure table storage as NoSQL data storage

We used Azure table storage as NoSQL data storage for quite a time and got some experience which may be useful to share with community. First of all I want to say that I like Azure table storage: it is very simple to use (there are different SDKs), flexible with it's dynamic data schema for tables (although this must be a common thing in NoSQL world), fast (if you use it correctly and avoid full table scan scenarios) and last but not least - it is very cheap.

However at the same time it has number of cons which may be significant depending on amount of data you need to process and performance which you need to achieve. I collected them in the following table:

Cons Comments
No LIKE operator Can't perform search by sub string. Need to introduce complex data duplication patterns with different PartitionKey/RowKey for avoiding full table scan and performance issues
Only 2 system indexed columns (PartitionKey/RowKey) without possibility to add own indexes Can't add custom indexed fields for improving queries performance
No full pagination support on API level There is possibility to limit returned resultset size, but no possibility to skip number of rows
No native backup options There are 3rd party (including open source) solutions which can be used but no built-in support from MS

Like I said I personally like Azure table storage. But due to these cons you may also consider other options for data storage e.g. Azure CosmosDB which is more expensive but also more powerful at the same time.

Wednesday, February 9, 2022

Problem with SPO app bar and Teams custom app with static tabs

If you use staticTabs in your custom MS Teams app (see Manifest schema for Microsoft Teams):

{
  "$schema": "https://developer.microsoft.com/en-us/json-schemas/teams/v1.5/MicrosoftTeams.schema.json",
  "manifestVersion": "1.5",
  ...
  "staticTabs": [
    {
      "entityId": "Foo",
      "name": "Bar",
      "contentUrl": "...",
      "websiteUrl": "https://example.com",
      "scopes": [
        "personal"
      ]
    }
  ],
  ...
}

you may face with the following issue: when user clicks on app icon it correctly opens web page defined in staticsTabs. But if user clicks on that second time after web page has been loaded then SPO app bar callout will be shown with My sites/My news.

In order to fix it the following workaround can be used: identify that app on opened web page is running inside Teams (this article contains details how to do that: How to identify whether SPFx web part is running in web browser or in Teams client) and hide SPO app bar callouts via css in this case. This is how it can be done via TypeScript (for this example we assume that app is SPFx web part running on SPO page):

if (isAppRunningInsideTeams()) {
  const style = document.createElement("style");
  style.textContent = "#sp-appBar-callout { display:none !important; } ";
  const head = document.getElementsByTagName("head")[0];
  head.appendChild(style);
}

It will hide callouts only inside Teams where this problem happens and at the same time it will still work in SPO.