Thursday, June 22, 2023

Implement TAP/multithread friendly logging scopes for Microsoft.Extensions.Logging.ILogger

Some time ago I wrote a post how to implement custom logger which writes logs to Azure storage blob container: Custom logger for .Net Core for writing logs to Azure BLOB storage. This logger implements ILogger interface from Microsoft.Extensions.Logging namespace. It works quite well but doesn't support logging scopes:

public IDisposable BeginScope<TState>(TState state) => default!;

Logging scopes are quite useful - they allow to specify additional information e.g. from each method log record was added, etc:

public void Foo()
{
    using (logger.BeginScope("Outer scope"))
    {
        ...
        using (logger.BeginScope("Inner scope"))
        {
        }
    }
}

Important requirement for logging scopes is that it should work properly in Task-based asynchronous pattern (TAP) and multithread code (which is widely used nowadays). For that we will use AsyncLocal<T> class from .NET. For implementing scopes themselves we will use linked list (child-parent relation).

For implementing it we will create LogScopeProvider class which implements Microsoft.Extensions.Logging.IExternalScopeProvider interface (for creating this custom LogScopeProvider I used code from Microsoft.Extensions.Logging.Console namespace as base example):

public class LogScopeProvider : IExternalScopeProvider
{
    private readonly AsyncLocal<LogScope> currentScope = new AsyncLocal<LogScope>();

    public object Current => this.currentScope.Value?.State;

    public LogScopeProvider() {}

    public void ForEachScope<TState>(Action<object, TState> callback, TState state)
    {
        void Report(LogScope current)
        {
            if (current == null)
            {
                return;
            }
            Report(current.Parent);
            callback(current.State, state);
        }

        Report(this.currentScope.Value);
    }

    public IDisposable Push(object state)
    {
        LogScope parent = this.currentScope.Value;
        var newScope = new LogScope(this, state, parent);
        this.currentScope.Value = newScope;

        return newScope;
    }

    private class LogScope : IDisposable
    {
        private readonly LogScopeProvider provider;
        private bool isDisposed;

        internal LogScope(LogScopeProvider provider, object state, LogScope parent)
        {
            this.provider = provider;
            State = state;
            Parent = parent;
        }

        public LogScope Parent { get; }

        public object State { get; }

        public override string ToString()
        {
            return State?.ToString();
        }

        public void Dispose()
        {
            if (!this.isDisposed)
            {
                this.provider.currentScope.Value = Parent;
                this.isDisposed = true;
            }
        }
    }
}

Note that LogScopeProvider stores scopes as AsyncLocal<LogScope> which allows to use it in TAP code. So e.g. if we have await inside using(scope) it will be handled correctly:

public async Task Foo()
{
    using (logger.BeginScope("Outer scope"))
    {
        var result = await Bar();
        ...
    }
}

Now returning to our BlobStorage: all we have to is to pass LogScopeProviderto its constructor, add current scope to the log record and return new scope when it is requested (check bold code):

public class BlobLogger : ILogger
{
    private const string CONTAINER_NAME = "custom-logs";
    private string connStr;
    private string categoryName;
    private LogScopeProvider scopeProvider;
 
    public BlobLogger(string categoryName, string connStr, LogScopeProvider scopeProvider)
    {
        this.connStr = connStr;
        this.categoryName = categoryName;
        this.scopeProvider = scopeProvider;
    }
 
    public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception? exception,
        Func<TState, Exception?, string> formatter)
    {
        if (!IsEnabled(logLevel))
        {
            return;
        }
 
        string scope = this.scopeProvider.Current as string;
        using (var ms = new MemoryStream(Encoding.UTF8.GetBytes($"[{this.categoryName}: {logLevel,-12}] {scope} {formatter(state, exception)}{Environment.NewLine}")))
        {
            var container = this.ensureContainer();
            var now = DateTime.UtcNow;
            var blob = container.GetAppendBlobClient($"{now:yyyyMMdd}/log.txt");
            blob.CreateIfNotExists();
            blob.AppendBlock(ms);
        }
    }
 
    private BlobContainerClient ensureContainer()
    {
        var container = new BlobContainerClient(this.connStr, CONTAINER_NAME);
        container.CreateIfNotExists();
        return container;
    }
 
    public bool IsEnabled(LogLevel logLevel) => true;
 
    public IDisposable BeginScope<TState>(TState state) => this.scopeProvider.Push(state);
}

That's it: now our logger also supports logging scopes.

Saturday, June 17, 2023

Profile MySQL db with Neor Profile SQL

If you need to profile MySQL db you may use builtin MySQL shell profiler (CLI). Also there is free GUI alternative - Neor Profile SQL. There are few things which should be done before to use it.

When you launch Profile SQL it asks to establish connection using default parameters for localhost:

If you click Test button you may get "Test is failed" error even with correct credentials of the root user. In order to avoid it you need to enable native MySQL password for the root user:

ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY '...'

after that connection should be successful.

But that is not all. If after that you will run application which connects to MySQL you won't see sessions and queries in profiler - because profiler works as a proxy with own port (4040 by default):

and in order to collect data your application should connect to profiler (not to MySQL directly). I.e. we need to change port in connection string from MySQL port (3306 by default) to profiler port (4040):

{
  "ConnectionStrings": {
    "Default": "Server=localhost;Port=4040;Database=Test;User ID=test;Password=..."
  }
}

If after that connection will fail with System.IO.IOException (from MySql.Data.Common.Ssl namespace methods) add ";SSL Mode=None" to connection string:

{
  "ConnectionStrings": {
    "Default": "Server=localhost;Port=4040;Database=Test;User ID=test;Password=...;SSL Mode=None"
  }
}

After that application should connect to profiler successfully and you should see profiled data.

StackOverflowException when use hierarchical object mapping with AutoMapper and Fluent NHibernate lazy loading

Suppose that we have the following class Category with self-reference (regular parent-child hierarchy):

public class Category
{
    public virtual T Id { get; set; }
    public virtual string Name { get; set; }
    public virtual Category ParentCategory { get; set; }
    public virtual IList<Category> ChildCategories { get; set; }
}

(properties are virtual which is required by NHibernate). Regular Fluent NHibernate mapping for this class looks like this:

public class CategoryMap : ClassMap<Category>
{
    public CategoryMap()
    {
        Table("[Category]");
        Id(x => x.Id);
        
        Map(x => x.Name);

        References(x => x.ParentCategory)
            .Column("ParentCategoryId");
        
        HasMany(x => x.ChildCategories)
            .KeyColumn("ParentCategoryId")
            .Inverse()
            .AsBag();
    }
}

Then we want to show category in the view of ASP.Net Core MVC app and use the following view model for that:

public class CategoryModel
{
    public int? Id { get; set; }

    public string Name { get; set; }

    public string ParentCategoryName { get; set; }
}

For mapping Category to CategoryModel the following AutoMapper mapping is used:

public class ViewModelProfile : Profile
{
    public ViewModelProfile()
    {
        CreateMap<Category, CategoryModel>()
            .ForMember(x => x.ParentCategoryName, o => o.MapFrom(c => c.ParentCategory == null ? "" : c.ParentCategory.Name))
    }
}

i.e. when ParentCategory is null then ParentCategoryName property of view model is empty string, otherwise ParentCategoryName will return name of parent category. Until now everything looks good. Problems will start when we will try to create category of nesting level which is greater than 1, i.e. this will work:

child category -> parent category

but with deeper hierarchy:

child category1 -> child category2 -> parent category

we will get StackOverflowException when will try to pass view model (created by AutoMapper) to the view. Googling this problem suggests to use MaxDepth(2) call when create AotuMapper map profile to limit number of processed hierarchy levels but it didn't help because of some reason.

Solution which worked was to specify to not use lazy loading for ParentCategory property in Fluent NHibernate mapping (NHibernate uses lazy loading by default):

public class CategoryMap : ClassMap<Category>
{
    public CategoryMap()
    {
        Table("[Category]");
        Id(x => x.Id);
        
        Map(x => x.Name);

        References(x => x.ParentCategory)
            .Column("ParentCategoryId")
            .Not.LazyLoad();
        
        HasMany(x => x.ChildCategories)
            .KeyColumn("ParentCategoryId")
            .Inverse()
            .AsBag();
    }
}

In theory it may cause performance problems with large amount of data and deep hierarchy levels but with reasonable amount of data should work quite well.

Monday, May 29, 2023

Run ASP.Net Core Web API on Kestrel dev web server with https on Windows

In this post I will describe how to run ASP.Net Core Web API on Kestrel development web server under https. First of all we need to create self-signed SSL certificate. We may generate it with PowerShell (see Use self-signed SSL certificate for web API hosted in Azure App service) or openssl tool:

openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout test.key -out test.crt -config test.conf -passin pass:123
openssl pkcs12 -export -out test.pfx -inkey test.key -in test.crt -passout pass:123

For running above commands we will need config file test.conf with information about domain name. It may look like that:

[req]
default_bits = 2048
default_keyfile = test.key
distinguished_name = req_distinguished_name
req_extensions = req_ext
x509_extensions = v3_ca

[req_distinguished_name]
countryName =
countryName_default =
stateOrProvinceName =
stateOrProvinceName_default =
localityName =
localityName_default =
organizationName = Test
organizationName_default = Test
organizationalUnitName = organizationalunit
organizationalUnitName_default = Development
commonName = api.example.com
commonName_default = api.example.com
commonName_max = 64

[req_ext]
subjectAltName = @alt_names

[v3_ca]
subjectAltName = @alt_names

[alt_names]
DNS.1 = api.example.com

Once private key (pfx) is created we may install it to the local certificates store: double click pfx, and follow certificate installation wizard with default settings:


We will also need to provide password for private key in this wizard (in above example "123").

Once certificate is installed to the certificates store we need to set the following parameters in appsettings.json file of our ASP.Net Core Web API project:

{
  "Kestrel": {
    "Endpoints": {
      "Http": {
        "Url": "http://api.example.com:5057"
      },
      "HttpsInlineCertStore": {
        "Url": "https://api.example.com:5058",
        "Certificate": {
          "Subject": "api.example.com",
          "Store": "My",
          "Location": "CurrentUser",
          "AllowInvalid": true
        }
      }
    }
  }
}

(since in our example self-signed certificate is used we need to set AllowInvalid: true parameter). If everything is done correctly Web API will run on local Kestrel dev server under https.

Thursday, May 18, 2023

Generate action urls from C# lambda expressions in ASP.Net MVC Core: good news for those who missed it there

In ASP.Net MVC (on top on .Net Framework) there was useful mechanism for generating actions urls from C# lambda expressions. Let's say we have controller UserController for managing users which has List action with 3 params:

  • page number (in case if users list is large and we need to use pagination)
  • sort by (first name, last name, etc)
  • sort direction (asc or desc)
public enum SortDirection
{
    Asc,
    Desc
}

public class UserController : Controller
{
    public ActionResult List(int pageNumber, string sortBy, SortDirection sortDirection)
    {
        ...
    }
}

In this case we could generate url for this action using generic ActionLink<T> method like this:

Html.ActionLink<UserController>(c => c.List(0, "FirstName", SortDirection.Asc), "All users")

which will generate the following url:

/user/list?pageNumber=0&sortBy=FirstName&sortDirection=SortDirection.Asc

That is convenient method since we have compile-time check for the code. Compare it with classic way:

Html.ActionLink("All users", "List", "Users", new { pageNumber = 0, sortBy = "FirstName", sortDirection = SortDirection.Asc })

In the last case if e.g. action name, parameters names or number of params will be changed we won't get any errors or warning during compilation. Instead we may get unexpected behavior or runtime error (e.g. if new parameter was added it will have default value if we won't explicitly add it to link generation code).

The problem is that strongly-typed method is not available in ASP.Net Core. Don't know what was the reason for not adding it there (except mentioned advantage of having compile-time check it may also be a problem during migration of old ASP.Net MVC app to ASP.Net Core) but fortunately it is possible to get it back there.

Lets check steps which are needed for generating action url from expression: we need to get controller name (/user), action name (/list), list action parameters names (or get route values from expression how it is called in ASP.Net MVC) and (most tricky one) get value of each parameter passed to expression. And then concatenate all parts to one string. First 3 steps are relatively easy: they can be done by basic reflection. But last step (get values of parameters passed to lambda expression) needs extra attention. In the past I already faced with that need in Camlex.NET (open source library for Sharepoint developers which I maintain in free time): Runtime evaluation of lambda expressions. We can use the same technique here as well. Also (as I found out during experiments) code for generating routing values from expression (list parameters names) can be reused with small changes from internal method of ASP.Net MVC Microsoft.Web.Mvc.Internal.ExpressionHelper.GetRouteValuesFromExpression in ASP.NET Core - it will help to save our time.

Now if we will combine all of this we may create helper class for generating urls from expressions in ASP.Net Core:

public static class HtmlHelperExtensions
{
    public static string ActionLink<TController>(this IHtmlHelper html, Expression<Action<TController>> actionExpression)
    {
        return ActionLink(actionExpression, GetRouteValuesFromExpression(actionExpression));
    }

    public static string ActionLink<TController>(this IHtmlHelper html, Expression<Action<TController>> actionExpression, RouteValueDictionary routeValues)
    {
        return ActionLink(actionExpression, routeValues);
    }

    public static string ActionLink<TController>(Expression<Action<TController>> actionExpression, RouteValueDictionary routeValues)
    {
        string controllerName = typeof(TController).GetControllerName();
        string actionName = actionExpression.GetActionName();
        var sb = new StringBuilder($"/{controllerName}/{actionName}");
        if (routeValues != null)
        {
            bool isFirst = true;
            foreach (var routeValue in routeValues)
            {
                sb.Append(isFirst ? "?" : "&");
                sb.Append($"{routeValue.Key}={routeValue.Value}");
                isFirst = false;
            }
        }
        return sb.ToString();
    }

    private static string GetControllerName(this Type controllerType)
    {
        return controllerType.Name.Replace("Controller", string.Empty);
    }

    private static string GetActionName(this LambdaExpression actionExpression)
    {
        return ((MethodCallExpression)actionExpression.Body).Method.Name;
    }

    // copy of Microsoft.Web.Mvc.Internal.ExpressionHelper.GetRouteValuesFromExpression
    private static RouteValueDictionary GetRouteValuesFromExpression<TController>(
        Expression<Action<TController>> action)
    {
        if (action == null)
            throw new ArgumentNullException(nameof(action));
        if (!(action.Body is MethodCallExpression body))
            throw new ArgumentException("MustBeMethodCall");
        string name = typeof(TController).Name;
        string str = name.EndsWith("Controller", StringComparison.OrdinalIgnoreCase)
            ? name.Substring(0, name.Length - "Controller".Length)
            : throw new ArgumentException("TargetMustEndInController");
        if (str.Length == 0)
            throw new ArgumentException("CannotRouteToController");
        string targetActionName = GetTargetActionName(body.Method);
        var rvd = new RouteValueDictionary();
        AddParameterValuesFromExpressionToDictionary(rvd, body);
        return rvd;
    }

    private static void AddParameterValuesFromExpressionToDictionary(
        RouteValueDictionary rvd,
        MethodCallExpression call)
    {
        ParameterInfo[] parameters = call.Method.GetParameters();
        if (parameters.Length <= 0)
            return;
        for (int index = 0; index < parameters.Length; ++index)
        {
            Expression expression = call.Arguments[index];
            object obj = !(expression is ConstantExpression constantExpression)
                ? Expression.Lambda<Func<object, object>>((Expression)Expression.Convert(expression, typeof(object)), Expression.Parameter(typeof(object), "_unused")).Compile().Invoke((object)null)
                : constantExpression.Value;
            rvd.Add(parameters[index].Name, obj);
        }
    }

    private static string GetTargetActionName(MethodInfo methodInfo)
    {
        string name = methodInfo.Name;
        ActionNameAttribute actionNameAttribute = !methodInfo.IsDefined(typeof(NonActionAttribute), true)
            ? methodInfo.GetCustomAttributes(typeof(ActionNameAttribute), true).OfType<ActionNameAttribute>().FirstOrDefault<ActionNameAttribute>()
            : throw new InvalidOperationException("CannotCallNonAction");
        if (actionNameAttribute != null)
            return actionNameAttribute.Name;
        return name;
    }
}

Using this helper class we may again generate actions links from C# expressions in ASP.Net Core.


Wednesday, May 3, 2023

Custom logger for .Net Core for writing logs to Azure BLOB storage

If you create .Net Core app you may use standard console logger (from Microsoft.Extensions.Logging.Console nuget package). For development it works quite well but if app goes to production you probably want to store logs in some persistent storage in order to be able to check them when needed. In this post I will show how to create custom logger which will write logs to Azure BLOB storage.

At first need to mention that there is BlobLoggerProvider from MS (from Microsoft.Extensions.Logging.AzureAppServices nuget package) which creates BatchingLogger which in turn stores logs to BLOB storage as well. However it has dependency on Azure App Services infrastructure so on practice you may use it only if your code is running inside Azure App Service. If this limitation is important we may go with custom logger implementation.

First of all we need to implement logger itself which inherits Microsoft.Extensions.Logging.ILogger interface and writes logs to the BLOB:

public class BlobLogger : ILogger
{
	private const string CONTAINER_NAME = "custom-logs";
	private string connStr;
	private string categoryName;

	public BlobLogger(string categoryName, string connStr)
	{
		this.connStr = connStr;
		this.categoryName = categoryName;
	}

	public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception? exception,
		Func<TState, Exception?, string> formatter)
	{
		if (!IsEnabled(logLevel))
		{
			return;
		}

		using (var ms = new MemoryStream(Encoding.UTF8.GetBytes($"[{this.categoryName}: {logLevel,-12}] {formatter(state, exception)}{Environment.NewLine}")))
		{
			var container = this.ensureContainer();
			var now = DateTime.UtcNow;
			var blob = container.GetAppendBlobClient($"{now:yyyyMMdd}/log.txt");
			blob.CreateIfNotExists();
			blob.AppendBlock(ms);
		}
	}

	private BlobContainerClient ensureContainer()
	{
		var container = new BlobContainerClient(this.connStr, CONTAINER_NAME);
		container.CreateIfNotExists();
		return container;
	}

	public bool IsEnabled(LogLevel logLevel) => true;

	public IDisposable BeginScope<TState>(TState state) => default!;
}

BlobLogger creates new container "custom-logs" (if it doesn't exist yet) in specified BLOB storage. Inside this container it also creates folders using current date as folder name yyyyMMdd (own folder for each day) and writes messages to the log.txt file inside this folder. Note that for working with Azure BLOB storage we used BlobContainerClient class from Azure.Storage.Blobs nuget package since. It will allow us to use instance of our logger as singleton (see below) because instance methods of this client class are guaranteed to be thread safe:

Thread safety
We guarantee that all client instance methods are thread-safe and independent of each other (guideline). This ensures that the recommendation of reusing client instances is always safe, even across threads.

In order to create BlobLogger we need to pass connection string to Azure storage and logging category name. It will be done in logger provider which will be responsible for creating BlobLogger:

public class BlobLoggerProvider : ILoggerProvider
{
	private string connStr;

	public BlobLoggerProvider(string connStr)
	{
		this.connStr = connStr;
	}

	public ILogger CreateLogger(string categoryName)
	{
		return new BlobLogger(categoryName, this.connStr);
	}

	public void Dispose()
	{
	}
}

Now everything is ready for using our logger in .Net Core app. If we want to use our BLOB logger together with console logger (which is quite convenient since logging is done both into console and into BLOB storage) we may use LoggingFactory and pass both standard ConsoleLoggerProvider and our BlobLoggerProvider:

var configuration = builder.GetContext().Configuration;
var azureStorageConnectionString = configuration.GetValue<string>("AzureWebJobsStorage");
var logger = new LoggerFactory(new ILoggerProvider[]
{
	new ConsoleLoggerProvider(new OptionsMonitor<ConsoleLoggerOptions>(new ConsoleLoggerOptions())),
	new BlobLoggerProvider(azureStorageConnectionString)
}).CreateLogger("CustomLogger");
builder.Services.AddSingleton(logger);

For creating instance of ConsoleLoggerProvider I used OptionsMonitor<T> helper class from How to create a ConsoleLoggerProvider without the full hosting framework (don't know why MS didn't provide this option OTB and made it so complex there):

public class OptionsMonitor<T> : IOptionsMonitor<T>
{
	private readonly T options;

	public OptionsMonitor(T options)
	{
		this.options = options;
	}

	public T CurrentValue => options;

	public T Get(string name) => options;

	public IDisposable OnChange(Action<T, string> listener) => new NullDisposable();

	private class NullDisposable : IDisposable
	{
		public void Dispose() { }
	}
}

After that you will have logs both in console and in Azure BLOB storage.

Update 2023-06-22: wrote another post how to add support of logging scopes which work in task-based asynchronous pattern (TAP) and multithread code: Custom logger for .Net Core for writing logs to Azure BLOB storage.

Monday, April 17, 2023

Add basic authentication to ASP.Net Core Web API

Basic authentication is probably simplest authentication type for Web API when HTTP requests are authenticated using username and passwords provided in HTTP request headers. In this post I will describe how to add basic authentication to ASP.Net Core Web API.

At first we need to add reference to idunno.Authentication.Basic nuget package. It contains infrastructure for basic authentication ready to be used in ASP.Net Core/.NET Core projects. Also we will need simple validation service which will check provided username/password and based on that will authenticate/reject requests:

public interface IBasicAuthValidationService
{
    bool AreCredentialsValid(string username, string password);
}

public class BasicAuthValidationService : IBasicAuthValidationService
{
    private string username;
    private string password;

    public BasicAuthValidationService(string username, string password)
    {
        this.username = username;
        this.password = password;
    }

    public bool AreCredentialsValid(string username, string password)
    {
        return string.Compare(this.username, username, true) == 0 && this.password == password;
    }
}

In our example BasicAuthValidationService simply compares credentials coming from HTTP request with predefined allowed username/password.

Then in Web API's Program.cs we need to configure basic authentication itself. Since username and password are sent in HTTP request headers it is important to force using HTTPS for securing communication with our API. We will do that by adding RequireHttpsAttribute filter and UseHttpsRedirection middleware:

builder.Services.AddScoped<IBasicAuthValidationService>(c =>
{
    // get allowed username and password to authenticate requests
    string username = ...;
    string password = ...;
    return new BasicAuthValidationService(uername, password);
});

// add basic auth
builder.Services.AddAuthentication(BasicAuthenticationDefaults.AuthenticationScheme)
    .AddBasic(options =>
    {
        options.Realm = "Test.Api";
        options.Events = new BasicAuthenticationEvents
        {
            OnValidateCredentials = context =>
            {
                var validationService = context.HttpContext.RequestServices.GetService<IBasicAuthValidationService>();
                if (validationService.AreCredentialsValid(context.Username, context.Password))
                {
                    var claims = new[]
                    {
                        new Claim(ClaimTypes.NameIdentifier, context.Username, ClaimValueTypes.String, context.Options.ClaimsIssuer),
                        new Claim(ClaimTypes.Name, context.Username, ClaimValueTypes.String, context.Options.ClaimsIssuer)
                    };

                    context.Principal = new ClaimsPrincipal(new ClaimsIdentity(claims, context.Scheme.Name));
                    context.Success();
                }

                return Task.CompletedTask;
            }
        };
    });

builder.Services.Configure<MvcOptions>(options =>
{
    options.Filters.Add(new RequireHttpsAttribute());
});

var app = builder.Build();
... app.UseHttpsRedirection();

Our Web API is now ready to be used with basic authentication. If you use Swagger for generating client for Web API (see Generate strongly-typed C# client for ASP.Net Core Web API with OpenAPI (swagger) support running on localhost) then you may configure your Web API client for basic authentication like this:

services.AddHttpClient<HttpClient>();
services.AddSingleton<IApiClient>(c =>
{
    string baseUrl = ...;
    string username = ...;
    string password = ...;
    var httpClientFactory = c.GetRequiredService<IHttpClientFactory>();
    var httpClient = httpClientFactory.CreateClient();
    httpClient.DefaultRequestHeaders.Authorization = new BasicAuthenticationHeaderValue(username, password);
    return new ApiClient(baseUrl, httpClient);
});

Now both server and client sides support basic authentication.