Showing posts with label WebJobs. Show all posts
Showing posts with label WebJobs. Show all posts

Friday, June 17, 2022

Get log of triggered Azure web job via REST API programmatically in PowerShell

Continue series of posts about Azure web jobs with this post how to fetch log of triggered Azure web job programmatically via REST API. Previous articles from the series can be found here:

In order to get log of triggered Azure web job programmatically we need to use the following endpoint:

https://{webAppName}.scm.azurewebsites.net/api/triggeredwebjobs/{webJobName}/history

It returns array of runs - information about last run is stored in first element of array. One of the properties there is output_url which contains url of the log of this run. Having this information we may write the following PowerShell function which will fetch log of Azure web job run:

function Get-Web-Job-Most-Recent-Log {
    param (
        $webAppName,
        $webJobName
    )
	$token = Get-AzAccessToken
	$headers = @{ "Authorization" = "Bearer $($token.Token)" }
	$userAgent = "powershell/1.0"
	$history = Invoke-RestMethod -Uri "https://$($webAppName).scm.azurewebsites.net/api/triggeredwebjobs/$($webJobName)/history" -Headers $headers -UserAgent $userAgent -Method GET
	if ($history -eq $null -or $history.runs -eq $null -or $history.runs.Length -eq 0) {
		return $null
	}
	$log = Invoke-RestMethod -Uri $history.runs[0].output_url -Headers $headers -UserAgent $userAgent -Method GET
	return $log
}

With this method we may fetch log of Azure web job programmatically.

Tuesday, June 14, 2022

Trigger web job via web jobs REST API

In my previous post I showed how to get status of triggered Azure web job via web jobs REST API in PowerShell (see Get status of timer triggered Azure web job via REST API in PowerShell). In this post I will show how to trigger web job programmatically also via REST API.

For triggering web job via REST API we need to send HTTP POST request to the following endpoint:

https://{webAppName}.scm.azurewebsites.net/api/triggeredwebjobs/{webJobName}/run

PowerShell method which triggers web job looks like that:

function Run-Web-Job {
    param (
        $webAppName,
	$webJobName
    )
	$token = Get-AzAccessToken
	$headers = @{ "Authorization" = "Bearer $($token.Token)" }
	$userAgent = "powershell/1.0"
	Invoke-RestMethod -Uri "https://$($webAppName).scm.azurewebsites.net/api/triggeredwebjobs/$($webJobName)/run" -Headers $headers -UserAgent $userAgent -Method POST
}

Here we first get access token via Get-AzAccessToken cmdlet and sent POST request with received bearer token. As result it will trigger specified web job to run now.

Friday, June 10, 2022

Get status of timer triggered Azure web job via REST API in PowerShell

In order to get current status of timer triggered Azure web job we may use the following web jobs REST API endpoint:

https://{webAppName}.scm.azurewebsites.net/api/triggeredwebjobs/{webJobName}/history

It will return object with "runs" property which is array containing information about recent runs:

id         : 202206101445293407
name       : 202206101445293407
status     : Running
start_time : 2022-06-10T14:45:29.3564164Z
end_time   : 0001-01-01T00:00:00
duration   : 00:00:04.0337019
output_url : https://{webAppName}.scm.azurewe.scm.azurewebsites.net/vfs/data/jobs/triggered/{webJobName}/202206101445293407/output_log.txt
error_url  :
url        : https://{webAppName}.scm.azurewe.scm.azurewebsites.net/api/triggeredwebjobs/{webJobName}/history/202206101445293407
job_name   : test
trigger    : External -

First element of this array corresponds to the most recent run. If web job is currently running status property will be set to Running. Here is function which we can use for fetching web job status in PowerShell:

function Get-Web-Job-Status {
    param (
    	$webAppName,
	$webJobName
    )
	$token = Get-AzAccessToken
	$headers = @{ "Authorization" = "Bearer $($token.Token)" }
	$userAgent = "powershell/1.0"
	$history = Invoke-RestMethod -Uri "https://$($webAppName).scm.azurewebsites.net/api/triggeredwebjobs/$($webJobName)/history" -Headers $headers -UserAgent $userAgent -Method GET
	if ($history -eq $null -or $history.runs -eq $null -or $history.runs.Length -eq 0) {
		return $null
	}
	return $history.runs[0].status
}

It makes HTTP GET request to mentioned endpoint and gets list of runs for specified web job.

Tuesday, May 3, 2022

Get functions invocations list for continuous Azure web job via REST API

This post is continuation of series of articles about continuous Azure web jobs. You may find previous articles about continuous web jobs here:

In this post I will show how to get list of functions' invocations for continuous web job via REST API.

As you probably know (and as it was shown in one of above articles) when you work with continuous web job we need to implement function which will be triggered e.g. using Azure queue trigger. Log which is shown in Azure portal looks bit different for continuous web jobs comparing with timer triggered web jobs: every time when this handler function got triggered new invocation is added to the log. It contains details of this particular function's invocation:


In order to get this list of functions' invocations programmatically we may use WebJobs REST API and more specifically the following endpoint:

https://{app-service-name}.scm.azurewebsites.net/azurejobs/api/jobs/continuous/{webJob}/functions?limit={limit}

Response will look like that:

{
    "entries": [{
            "executingJobRunId": null,
            "id": "0399f7d5-a7aa-43c9-8853-70886174fda3",
            "functionId": null,
            "functionName": null,
            "functionFullName": null,
            "functionDisplayTitle": "ProcessQueueMessage (5540, )",
            "status": "CompletedSuccess",
            "whenUtc": "2022-05-03T12:12:05.3054119Z",
            "duration": 101650.2167,
            "exceptionMessage": null,
            "exceptionType": null,
            "hostInstanceId": "00000000-0000-0000-0000-000000000000",
            "instanceQueueName": null
        }, {
            "executingJobRunId": null,
            "id": "d581d6fa-8ba0-4c4d-ad39-c8505d21b851",
            "functionId": null,
            "functionName": null,
            "functionFullName": null,
            "functionDisplayTitle": "ProcessQueueMessage (5539, )",
            "status": "CompletedSuccess",
            "whenUtc": "2022-05-03T12:12:04.4842786Z",
            "duration": 100893.4565,
            "exceptionMessage": null,
            "exceptionType": null,
            "hostInstanceId": "00000000-0000-0000-0000-000000000000",
            "instanceQueueName": null
        }
    ],
    "continuationToken": null,
    "isOldHost": false
}

It contains array of objects which represent invocation details. Id property contains actual invocationId which may be used e.g. for getting invocation log by the following url:

https://{app-service-name}.scm.azurewebsites.net/azurejobs/#/functions/invocations/{invocationId}

Property functionDisplayTitle contains name of invocation together with passed parameter. It may hep if you need to map input parameters with invocation log. Also pay attention on continuationToken - if there are many invocations (more than passed in limit query string parameter) it will contain continuation token using which you may get whole list of invocations by pages.

Wednesday, December 1, 2021

How to run continuous Azure web job as singleton

Continuous Azure web jobs may be used as subscribers to some external events (e.g. new item has been added to Azure storage queue). In opposite to scheduled based web jobs which are running by scheduler (you need to specify CRON expression for them) continuous web jobs are always running and react on events they are subscribed to.

When you scale your Azure app service on multiple instances (web jobs may run on different VMs in background) by default web jobs are also scaled i.e. they will run on all instances. However it is possible to change this behavior and run continuous web job as singleton only on 1 instance.

When create continuous web job in Azure portal there is Scale field which is by default set to Multi instance:


As tooltip says:

Multi-instance will scale your WebJob across all instances of your App Service plan, single instance will only keep a single copy of your WebJob running regardless of App Service plan instance count.

So during creation of web job we may set Scale = Single instance and Azure will create it as singleton.

If you don't want to rely on this setting which can be changed from UI you may add settings.job file with the following content:

{ "is_singleton": true }

to the root folder of your web job (the same folder which contains executable file of web job). In this case Azure will create web job as singleton even if Scale = Multi instance select is selected in UI. I.e. settings.job has priority over UI setting.

If you will check logs of continuous web job which was created using above method you should see something like that:

Status changed to Starting
WebJob singleton settings is True
WebJob singleton lock is acquired

 Which will prove that job runs as singleton.


Wednesday, November 3, 2021

How to configure number of retries for processing failed Azure queue message before it will be moved to poison queue

In my previous post I showed how to configure Azure web job host to set number of queue messages which may be processed simultaneously (in parallel): How to set limit on simultaneously processed queue messages with continuous Azure web job. In this post I will show how to configure web job host for processing failed messages.

As it was mentioned in previous post when message is added to Azure queue runtime will call method with 1st parameter decorated with [QueueTrigger] attribute. If there was unhandled exception this message will be marked as failed. Azure runtime will try to process this message again - by default there are maximum 5 retries for failed messages.

If we don't need to make 5 attempts (e.g. if our logic requires only 1 attempt) we may change this behavior using the following code:

public virtual void RunContinuous()
{
    var storageConn = ConfigurationManager.ConnectionStrings["AzureWebJobsDashboard"].ConnectionString;
    var config = new JobHostConfiguration();
    
    config.Queues.MaxDequeueCount = 1;
    
    var host = new JobHost(config);
    host.RunAndBlock();
}
 
public void ProcessQueueMessage([QueueTrigger("orders")]string message, TextWriter log)
{
    log.WriteLine("New message has arrived from queue");
    ...
}

I.e. we set config.Queues.MaxDequeueCount to 1. In this case failed message will be moved to poison queue immediately and web job won't try to process it again.

Tuesday, November 2, 2021

How to set limit on simultaneously processed queue messages with continuous Azure web job

With continuous Azure web job we may create web job host which will listen to specified Azure storage queue. When new message will be added to queue Azure will trigger handler method found in assembly of continuous web job (public method which has parameter with [QueueTrigger] attribute):

public virtual void RunContinuous()
{
    var storageConn = ConfigurationManager.ConnectionStrings["AzureWebJobsDashboard"].ConnectionString;
    var config = new JobHostConfiguration();
    config.StorageConnectionString = storageConn;
    var host = new JobHost(config);
    host.RunAndBlock();
}

public void ProcessQueueMessage([QueueTrigger("orders")]string message, TextWriter log)
{
    log.WriteLine("New message has arrived from queue");
	...
}

If several messages were added simultaneously Azure will trigger several instances of handler method in parallel. Number of messages which can be processed in parallel can be configured via JobHostConfiguration.Queues.BatchSize property. By default it is set to 16 i.e. by default 16 messages can be processed simultaneously in one batch:

 

If we want e.g. configure continuous web job so it will process only 1 message at time we can set BatchSize to 1:

public virtual void RunContinuous()
{
    var storageConn = ConfigurationManager.ConnectionStrings["AzureWebJobsDashboard"].ConnectionString;
    var config = new JobHostConfiguration();
    config.StorageConnectionString = storageConn;
	
    // set batch size to 1
    config.Queues.BatchSize = 1;
	
    var host = new JobHost(config);
    host.RunAndBlock();
}

public void ProcessQueueMessage([QueueTrigger("orders")]string message, TextWriter log)
{
    log.WriteLine("New message has arrived from queue");
	...
}

Using the same approach we can increase BatchSize from default 16 to bigger value. However as you may notice from above screenshot MaxBatchSize is set to 32 to you may increase BatchSize only up to 32.

Friday, October 8, 2021

Fix Azure web job startup error "Storage account connection string 'AzureWebJobsStorage' does not exist"

If you develop continuous Azure web job which monitors e.g. Azure storage queue messages then your code may look like this (in our example we will use latest v3 version of Microsoft.Azure.WebJobs):

var builder = new HostBuilder();
static void Main(string[] args)
{
    var builder = new HostBuilder();
    builder.UseEnvironment("development");
    builder.ConfigureWebJobs(b =>
    {
        b.AddAzureStorageCoreServices();
        b.AddAzureStorage();
    });
    builder.ConfigureLogging((context, b) =>
    {
        b.AddConsole();
    });
    var host = builder.Build();
    using (host)
    {
        host.Run();
    }
}

public static void ProcessQueueMessage([QueueTrigger("myqueue")] string message, TextWriter log)
{
    ...
}

Here in Main method (which is entry point for continuous web job) we create host which subscribes on specified queue. When new message will be added to the queue it should trigger our ProcessQueueMessage handler. However on startup you may get the following error:

Storage account connection string 'AzureWebJobsStorage' does not exist

This error is thrown when host can't find connection string AzureWebJobsStorage of Azure storage where specified queue is located. In order to avoid this error we need to create appsettings.json file in our project which looks like this:

{
  "ConnectionStrings": {
    "AzureWebJobsStorage": "..."
  }
}

Also in file's properties we need to set Copy to output directory = Copy always so it will be always copied in output directory with exe file. After that error should disappear and web job should be successfully started.

Wednesday, September 1, 2021

List and delete Azure web jobs using KUDU Web Jobs REST API

Some time ago I wrote how to remove Azure web jobs using AzureRM PowerShell module: How to remove Azure web job via Azure RM PowerShell. However since that time AzureRM was discontinued. MS recommended to switch to new Az module instead. The problem that if you will follow AzureRM migration guide then the same code won't work anymore. The main problem is that new Get-AzResource cmdlet doesn't return resources with type microsoft.web/sites/triggeredwebjobs (although it still returns  other Azure resources like App services, Azure functions, Storage accounts, KeyVaults, SSL certificates). It means that you also can't use Remove-AzResource cmdlet because there won't be resource name which you may pass there. For now I submitted issue to Azure Powershell github about that: Get-AzResource doesn't return web jobs. But let's see how we may workaround that until MS will fix it.

There is another open issue in Azure PowerShell github ARM :: WebApp Web Job Cmdlets which reports about lack of cmdlets for managing web jobs. It was created in 2015 and is still opened. One suggestion there is to use az cli tool. However it is not always possible (e.g. if you run your PowerShell in runbook or can't add this dependency to your script).

Fortunately there is KUDU Web Job REST API which may be utilized for this purpos. In order to list available web jobs we may use the following code:

$token = Get-AzAccessToken
$headers = @{ "Authorization" = "Bearer $($token.Token)" }
$userAgent = "powershell/1.0"
Invoke-RestMethod -Uri "https://{appServiceName}.scm.azurewebsites.net/api/triggeredwebjobs" -Headers $headers -UserAgent $userAgent -Method GET

It will return list of all existing (in our example triggered) web jobs in specific App service. In order to delete web job we need to send HTTP DELETE request to web job's endpoint:

$token = Get-AzAccessToken
Invoke-RestMethod -Uri "https://{appServiceName}.scm.azurewebsites.net/api/triggeredwebjobs/{webjobName}" -Headers $headers -UserAgent $userAgent -Method DELETE

There are other commands as well (like run job, list running history and others). You may check them in REST API documentation using link posted above.

Saturday, January 9, 2021

One way to fix Failed to run web job error on Azure App Service

If you have just updated web job (see the following related articles: Upload WebJob to Azure App service with predefined schedule and How to remove Azure web job via Azure RM PowerShell) you may face with the following issue: when you will try to run updated web job you will get the error “Failed to run web job”. The problem is that error notification won’t contain any description and it will be hard to find the actual reason of this problem. In this article I will describe one of the reasons.

When web job is started it creates special lock file triggeredJob.lock in /data/jobs/triggered/{webJobName} folder of SCM site. You may check it if will connect to the site via FTP:

In the same folder there will be sub folders with timestamps which correspond to webjob runs. In these sub folders you will find output logs of each web job run – the same logs which are shown in Azure portal > App service > Web jobs > Logs. The purpose of this triggeredJob.lock file is to prevent launch of second instance of web job when previous instance didn’t finish yet. And the problem is that if during update web job was running this lock file may not be successfully deleted. As result when you will try to run updated version of web job you will get “Failed to run web job” error.

Solution will be to delete this file manually. However it is also not that straightforward. If you will try to remove it from FTP client you will get error

“The process cannot access the file because it is being used by another process”:

In order to delete it we need to stop both App Service and SCM site. Note that it is mandatory to stop both sites – if you will only stop App Service from Azure portal triggeredJob.lock will be still used by the process. Stopping of SCM site is more tricky than App Service. The process is described here: Full stopping a Web App. You need to go to Resource Explorer (azure.com) and select your App service. After that in JSON view on the right side click Edit and do 2 things:

  1. Change “state” from “Running” to “Stopped” – this is the same as if you would stop App service form Azure portal
  2. Find “scmSiteAlsoStopped” property and set it from “false” to “true” – it will stop SCM file

After that click PUT button on the top. It will stop both sites and you will be able to delete triggeredJob.lock now. Then go to Azure portal and start App service – it will start both sites. After all these steps you should be able to run web job again.

Tuesday, December 1, 2020

App.config vs App service configuration settings for storing app settings for Azure web jobs

As you probably know when you upload zip package with exe application to Azure App service > Web jobs (to run it as Azure web job by scheduler) – you may override app settings in App service > Configuration > Application settings section. In this case settings defined in Azure will have priority over settings defined in app.config which is deployed as part of zip package:

However it is important to note that even if you will use app settings from Azure – it is still necessary to include app.config to zip package. Otherwise if you will exclude app.config from zip package – then ConfigurationManager.AppSettings will return empty collection. So good practice is to add app.config with empty values to zip package and then manage app settings in Azure. Don’t forget to remove values from app.config – otherwise sensitive information (client secrets, passwords, keys, etc) may be accidentally leaked with zip package of web job.

Thursday, July 16, 2020

One problem with caching old web job binaries after Azure web job update with WAWSDeploy

WAWSDeploy is convenient console utility which allows to deploy a folder or a zip file to an Azure Website using WebDeploy (see https://github.com/davidebbo/WAWSDeploy). Using this tool it is possible to automate tasks in your Azure deployment. E.g. you may upload Azure web job binaries (they should be packaged to zip file for that). However you should be aware of one potential problem.

Triggered web job binaries are uploaded to "app_data/jobs/triggered/{JobName}" path (full path will be /site/wwwroot/app_data/jobs/triggered/{JobName}). You may connect to your Azure App Service via FTP (connection settings may be copied from Azure portal > App service > Deployment center > FTP) and check that your web jobs folders are there

Note that you need to use FTP protocol (not FTPS) if you will connect e.g. via WinSCP tool.

The problem is the following: you may create zip package with web job binaries just by selecting all files for web jobs and adding them to archive – in this case there won’t be root folder in archive and all files will be located inside archive root. Alternatively you may select their parent folder (e.g. bin/Release) and add whole folder to archive – in this case archive will contain Release root folder.

So if 1st time you created archive using 1st method (by archiving files), then created another archive using 2nd method (by archiving folder) and uploaded this 2nd zip archive via WAWSDeploy – it will just create subfolder Release inside app_data/jobs/triggered/{JobName}. I.e. there will be 2 versions of web job: one in the app_data/jobs/triggered/{JobName} and another in app_data/jobs/triggered/{JobName}/Release. Azure runtime will use 1st found exe file which will be exe file from the root (not from Release sub folder) and will use that for running web job.

One way to avoid this problem is to always delete existing web job before to update it (you may check how to delete web job via PowerShell here: How to remove Azure web job via Azure RM PowerShell). In this case old binaries will be deleted so selected archiving method won’t cause problems during update.

Wednesday, July 15, 2020

How to remove Azure web job via Azure RM PowerShell

Azure web jobs allow to run your code by scheduler. Schedule is configured via CRON expression. This is convinient tool to perform repeated actions. Sometimes we need to remove them via PowerShell in automation process. In this article I will show how to remove Azure web job using Azure RM PowerShell cmdlets.

If you will try to google this topic you will most probably find Remove-AzureWebsiteJob cmdlet. The problem however is that this cmdlet is used for classic resources and you will most probably have problems using it (e.g. because of error Select-AzureSubscription -Default). So we need some other way to remove Azure web jobs – preferably by using Azure RM cmdlets. Fortunately it is possible – we may remove web job by using another cmdlet Remove-AzureRmResource. Here is syntax for removing triggered web job:

$resourceName = $appServiceName + "/" + $webJobName
Remove-AzureRmResource -ResourceGroupName $resourceGroupName -ResourceName $resourceName -ResourceType microsoft.web/sites/triggeredwebjobs

The important thing to note here is how ResourceName param is specified for web job (app service name/web job name) and ResourceType which is set to microsoft.web/sites/triggeredwebjobs. If you need to delete continuous web job you need to use microsoft.web/sites/continuouswebjobs resourceType.

Using this command you will be able to delete Azure web jobs via Azure RM PowerShell.

Monday, July 6, 2020

Run Azure function as Web job without code change

Sometime it may be needed to test existing Azure function as Web job (e.g. if you decide to move some long-running process from AF to web job). In order to do that you need to create App service first. After that create new console app project in Visual Studio, reference your Azure functions project as it would be regular class library and from Program.cs call necessary Azure function (it is possible since AFs are created as static methods of the class).

The problem however is that if you read some app settings in Azure function code – you most probably do it like that:

string foo = Environment.GetEnvironmentVariable("Foo", EnvironmentVariableTarget.Process);

But if you will just copy AF app settings to appSettings section of app.config of the new console project then above code will return empty string for any app setting. So in order to call our Azure function from console application and run it as web job we need to perform additional step: iterate through all app settings and fill environment variables for the process:

var appSettings = ConfigurationManager.AppSettings;
foreach (var key in appSettings.AllKeys)
{
 Environment.SetEnvironmentVariable(key, appSettings[key],
  EnvironmentVariableTarget.Process);
}

After that app settings will be successfully read and Azure function may run as web job without changing its code.

Monday, June 1, 2020

Upload WebJob to Azure App service with predefined schedule

As you probably know in Azure Portal it is possible to create Azure web jobs with Type = Triggered and specify CRON expression which will define schedule of the job. In this post I will describe how to create Azure web job with predefined CRON expression.

Before to start we need to create Azure app service in the tenant and prepare zip file with executable file for web job itself. Usually it can be created by archiving bin/Release (or bin/Debug) folder of the console project with web job logic. However there is one important detail: in order to set schedule of web job we need to add special file to the project called settings.job and define CRON expression inside this file. E.g. if we want our job to run every 15 minutes we need to define it like this:

{“schedule”: “* */15 * * * *”}

After that also go to File properties in Visual Studio and set "Copy to target directory value to “Copy Always”. After that compile project and create zip file from bin/Release (or bin/Debug) folder.

Next step is to create web job from zip file. It can be done by the following convenient command line util available on github: WAWSDeploy. With this tool web job can be created created e.g. from PowerShell by running the following command:

WAWSDeploy.exe MyPackage.zip publishSettings.txt /t "app_data\jobs\triggered\MyJob"

In this command publishSettings.txt is the file with publishing settings for your App service. It can be created by Get-AzureRmWebAppPublishingProfile cmdlet. This command will create new web job called MyJob based on provided zip file with predefined schedule.