Azure Batch – (Highly Scalable) Batch Processing with Microsoft Azure (and a Successor to GeRes2)

Batch processing is something nearly everyone I have been working with is doing in some or the other way on Microsoft Azure. Last year some colleagues and I did work with several global partners that had this requirement. Therefore since nothing at the scale of Azure Batch was available, yet, we created GeRes2 as an open source project. GeRes2 covers batch processing in a simple, pragmatic yet scalable way (at full scale of Web/Worker roles as opposed to the limited scale of WebJobs SDK in WebSites). But now, the times when you need GeRes2 are over. Instead you definitely should consider Azure Batch, directly.

What is Azure Batch?

Azure Batch is a completely managed service, you could see it as “batch processing as a service”.

It can be used for simple batch processing up to High-Performance-Compute (HPC) types of workloads. As a managed service, Azure Batch is fully operated by Microsoft and made accessible through an HTTP REST API. While for simpler requirements you might want to use WebJobs SDK, if you really need to scale beyond the options provided by WebSites and do not want to manage infrastructure (e.g. such as it is the case with Windows Server HPC Pack), Azure Batch is your best friend.

A sample on Azure Batch to help you Getting Started!

As a first introduction and to help you get started, I thought to publish a sample inspired by a session from one of the Azure Batch product managers, Mark Scurrell, at TechEd EMEA 2014: an OCR image recognition based on the Tesseract OCR Engine. The whole sample is available on my github.com repository:

https://github.com/mszcool/AzureBatchTesseractSample

The following sketch outlines the flow of the sample which I’ve built to help you get started. Note that I also include a PowerShell Script that sets up the environment in your Azure Subscription (please setup Azure PowerShell correctly, before). It creates all required Azure service accounts (storage, batch), updates app.config configurations, builds the sample and finally uploads sample data as it is needed for testing the scenario right away.

As you can see, the overall solution consists of a client that creates compute pools, submits jobs to that compute pool that use Tesseract for an OCR recognition on-top of PNG-based images stored in Azure Blob Storage and then makes the results available in Azure Blob storage, as well. Although there are different alternative ways with Azure Batch, for this first sample I also wrote a little console application that downloads the source-images from Azure Blob to the task virtual machines (TVM), processes them using tesseract.exe and then uploads the results back to BLOB-storage.

Understanding the fundamental Azure Batch Concepts

Before you can get started, you should understand the fundamental concepts of Azure Batch. Let’s get started with the following terms:

  • Azure Batch Account
    An account is a management unit used to group batch services and batch apps together in a single unit with security access keys.
  • Azure Batch REST API
    For each account, Batch is made available through an HTTP REST API. For .NET developers the team ships an SDK, already. Other languages will follow (or are available after I’ve published this article, already).
  • Azure Batch Apps
    Batch can be used in two flavors, through the low-level REST-API which is more complex but provides you with a bigger set of options and control or through a more managed experience with a management portal and a light-weight API called Batch Apps.
  • Compute Pools
    These are groups of compute nodes used for executing work. Pools can contain many compute nodes and can be configured with auto scaling rules so they add/remove resources based on the load on a pool.
  • Task Virtual Machines (TVM)
    A TVM is a single compute node which is part of a compute pool. Essentially behind the scenes TVMs are worker roles since Azure Batch by itself is implemented with Web/Worker roles. For you as developer, just think of them as scalable, stateless virtual machines.
  • Work Items
    Work Items are used to describe classes of units of works (aka jobs) and configure scheduling properties of jobs (e.g. execute once, regular time-controlled execution etc.). It also allows specifying the compute pools on which jobs of a work item are executed.
  • Jobs
    A job is a unit of work. It describes a set of concrete execution items called tasks. Each task gets executed on a TVM part of a pool that is tied to the work item the job belongs to.
  • Tasks
    A task is a single execution step of a job. Essentially a task is an executable that you need to provide and specify as part of the scheduling process that will be executed on TVMs.

Creating an Azure Batch Account with Azure PowerShell

The first thing you need to do is creating an Azure Batch Account. Note that at the time of writing this article, Batch was still in Preview. Therefore make sure you activate it for your subscription, first. The PowerShell script I provide for setting up the sample does that for you. Note that Azure Batch can only be managed as part of the new Azure Resource Manager inside of a Resource Group. Therefore in an Azure PowerShell CmdLet you first need to switch the Azure PowerShell mode to AzureResourceManager:

   1: Switch-AzureMode -Name AzureResourceManager

   2: New-AzureResourceGroup -Name $batchSampleResourceGroupName -Location $regionName

   3: New-AzureBatchAccount -AccountName $azureBatchAccountName `

   4:                       -ResourceGroupName $batchSampleResourceGroupName `

   5:                       -Location $regionName

Below a screen-shot of the script I wrote to setup things in action. That screen shot shows you, how-to call the script to setup the environment required for the demo:

Azure Batch NuGet Package

Now that we have an account, you can start developing. In Visual Studio you need to use a NuGet package called Azure Batch. Please make sure to use the core Azure Batch package and not the Batch Apps package in case you want to use the full-blown, low-level Batch APIs.

Creating Compute Pools using the API

The first thing you need to do is creating some compute pools with task virtual machines. The .NET SDK wraps the REST API for such purposes in a very convenient way. After having set-up BatchCredentials and a BatchClient you can open various managers that encapsulate certain management operations, e.g. a Compute Pool Manager as shown below.

   1: using (var pm = batchClient.OpenPoolManager())

   2: {

   3:     var pools = pm.ListPools().ToList();

   4:     var poolExists = (pools.Select(p => p.Name).Contains(PoolName));

   5:     if (!poolExists)

   6:     {

   7:         var newPool = pm.CreatePool

   8:             (PoolName, "3", "small", 5

   9:             );

  10:         newPool.StartTask = new StartTask

  11:         {

  12:             ResourceFiles = binaryResourceFiles,

  13:             CommandLine = "cmd /c CopyFiles.cmd",

  14:             WaitForSuccess = true

  15:         };

  16:         newPool.CommitAsync().Wait();

  17:     }

  18: }

One interesting aspect of the compute pool creation above is the definition of a startup task. This startup task in case of the sample defines a list of files that should be downloaded from Azure BLOB storage to the TVMs as part of the bootstrapping process. That list of files is prepared earlier in the code as shown below:

   1: var binaryResourceFiles = new List<IResourceFile>();

   2: Console.WriteLine("Get list of 'resource files' required for execution from BLOB storage...");

   3: foreach (var resFile in blobTesseractContainer.ListBlobs(useFlatBlobListing: true))

   4: {

   5:     var sharedAccessSig = CreateSharedAccessSignature(blobTesseractContainer, resFile);

   6:     var fullUriString = resFile.Uri.ToString();

   7:     var relativeUriString = fullUriString.Replace(blobTesseractContainer.Uri + "/", "");

   8:  

   9:     Console.WriteLine("- {0} ", relativeUriString);

  10:  

  11:     binaryResourceFiles.Add(

  12:         new ResourceFile

  13:             (

  14:             fullUriString + sharedAccessSig,

  15:             relativeUriString.Replace("/", @"\")

  16:             )

  17:         );

  18: }

Note that even if the blob container is publicly accessible I had to use shared access signatures to allow Azure Batch downloading those files. These files will be placed in a directory dedicated to the startup task (meaning actual tasks executed later don’t have access to this directory). Therefore one of the files downloaded is a batch-script which is then executed as part of the startup procedure. This batch script copies the file from the startup task working directory to the shared-directory to which all tasks do have access to as shown below:

   1: @echo off

   2: echo "List Files for diagnostics..."

   3: echo %WATASK_TVM_ROOT_DIR%

   4: dir .\ /s

   5: echo.

   6: echo Moving BatchTesseractWrapper files to shared task directory

   7: robocopy /MIR .\ %WATASK_TVM_ROOT_DIR%\shared

   8: if "%errorlevel%" LEQ "4" (

   9:    SET errorlevel=0

  10: )

One little hint here: Azure Batch considers a task (incl. the startup task) to be successful when it returns an exit code of 0. Since I am using robocopy.exe in my script I need to consider, that robocopy has several non-0 success exit codes. Therefore I map those to the exit code 0.

After my startup task finally completed, I do have all the tesseract-binaries as well as the tesseract wrapper executable I’ve written (which downloads files from BLOB, processes them with tesseract.exe and then uploads the result back to BLOB) in the shared task working directory.

Note that all of these working directories for tasks on TVMs are placed under a sub-directory of the task root directory exposed through the WATASK_TVM_ROOT_DIR environment variable. These directories are:

  • %WATASK_TVM_ROOT_DIR%\shared
    is a shared directory to which all tasks executed on the TVM do have read and execute permissions.
  • %WATASK_TVM_ROOT_DIR%\startup
    is a directory dedicated to the startup tasks specified as part of the compute pool creation. No other tasks do have any access to this directory.
  • %WATASK_TVM_ROOT_DIR%\tasks\<workitemname>\<jobname>\<taskname>
    is the directory dedicated for task execution whereas every work-item, job and task gets his own directory in this working directory.

A really cool tool for confirming that everything in our startup tasks did work is the Azure Batch explorer. It is also cool for exploring the true, detailed directory structure which I’ve outlined above. I might write about this tool in a subsequent blog post and will continue focusing on the code in this one.

Scheduling Jobs for Execution

Once the compute pool runs and all the TVMs are prepared with the tesseract binaries through the startup task, we can start scheduling jobs. This happens with the WorkItem manager and by creating a WorkItem with a Job and adding tasks to that job. In my sample I create one task for each file I want to OCR-recognize which I’ve previously uploaded to BLOB storage.

Note:the PowerShell setup script I do provide uploads some sample data I’ve included in the git-repository so that you can get started right away.

   1: using (var wiMgr = batchClient.OpenWorkItemManager())

   2: {

   3:     var workItemName = string.Format("ocr-{0}", DateTime.UtcNow.Ticks);

   4:     var ocrWorkItem = wiMgr.CreateWorkItem(workItemName);

   5:     ocrWorkItem.JobExecutionEnvironment =

   6:         new JobExecutionEnvironment

   7:         {

   8:             PoolName = PoolName

   9:         };

  10:     ocrWorkItem.CommitAsync().Wait();

  11:  

  12:     var taskNr = 0;

  13:     const string defaultJobName = "job-0000000001";

  14:     var job = wiMgr.GetJob(workItemName, defaultJobName);

  15:  

  16:     foreach (var ocrFile in filesToProcess)

  17:     {

  18:         var taskName = string.Format("task_no_{0}", taskNr++);

  19:         var taskCmd =

  20:             string.Format(

  21:                 "cmd /c %WATASK_TVM_ROOT_DIR%\\shared\\BatchTesseractWrapper.exe \"{0}\" \"{1}\"",

  22:                 ocrFile.BlobSource,

  23:                 Path.GetFileNameWithoutExtension(ocrFile.FilePath));

  24:  

  25:         ICloudTask cloudTask = new CloudTask(taskName, taskCmd);

  26:  

  27:         job.AddTask(cloudTask);

  28:     }

  29:     job.Commit();

The submission here happens with the WorkItemManager. Every WorkItem gets a default job (job-0000000001) which can be used immediately for adding tasks to be executed. This is exactly what is done in the code snippet above.

Note: since the job-creation might not be completed, yet, the call to wiMgr.GetJob() should be wrapped with some retry-logic.

For each file I’ve stored in BLOB storage I do create a task. That task ultimately just executes a command shell with the executable that downloads the source image to the TVM, calls tesseract.exe to OCR recognize the image and uploads the resulting text-file back to Azure Blob storage. Since this should be straight-forward for Azure-experienced developers, I leave it to you to look at the code on my github-repository.

Waiting for the tasks to complete and get results

This final step is optional, but in case your program needs to wait for the tasks to complete before doing something else, the code below is helpful.

   1: var toolBox = batchClient.OpenToolbox();

   2: var stateMonitor = toolBox.CreateTaskStateMonitor();

   3: var runningTasks = wiMgr.ListTasks(workItemName, defaultJobName);

   4: stateMonitor.WaitAll(runningTasks, TaskState.Completed, TimeSpan.FromMinutes(10));

   5:  

   6: var tasksFinalResult = wiMgr.ListTasks(workItemName, defaultJobName);

   7: foreach (var t in tasksFinalResult)

   8: {

   9:     Console.WriteLine("- Task {0}: {1}, exit code {2}", t.Name, t.State,

  10:         t.ExecutionInformation.ExitCode);

  11: }

The Azure Batch SDK for .NET comes with a set of handy utility classes to allow you doing exactly that. Through the TaskStateMonitor utility class the code above waits till all tasks have completed their work, loads the most recent data from the Azure Batch REST API and displays the results.

Important here is that every task, whether succeeded or failed, will be in the completed-state except you terminated it before it completed through the REST API. It’s up to you to check if the task was successful by looking at the exit code of the executable for your task.

The actual results of the tasks should be visible now in your Azure BLOB storage account in a container called ocr-results as shown below.

Final Words

This article and the sample I published on GitHub should help you to understand the basic principles and how-to get started with Azure Batch. As one of the creators of GeRes2, which is also doing scalable execution of jobs on Worker Roles, I really experienced how much effort it is to build something like this on your own.

As a managed service, Azure Batch really does everything for you. You can focus on creating your tasks as well as scheduling your jobs/tasks instead of dealing with all the plumbing infrastructure (e.g. distributing work across compute nodes, compute node management, logging, auto scaling, task binary deployments etc.).

The only thing that you might miss as compared to GeRes2 is notifications via Service Bus and SignalR. But to be honest, when using GeRes2, those notifications did only work up to a maximum of 100 compute nodes since we did not scale out the message queues on Service Bus which we used for the notifications. Of course we could have done that, but based on the needs of those partners who used notifications with GeRes2 it was just not necessary to support > 100 instances while those which used GeRes2 with more than 100 instances did really not use notifications. And there’s still the option of polling the tasks’ status through the APIs made available by Azure Batch. But eventually we’ll work on something to show you, how-to get this remaining piece to Azure Batch, as well.

Azure Batch is really the premium service when it comes to highly scalable batch processing on Azure. It is easy to use and it saves you from a whole lot of plumbing work that needs to be done if you do batch processing manually. In a future blog-post I plan to write about my personal opinion and view on comparing Azure Batch to other batch processing options available on Azure such as WebJobs SDK or running HPC Pack in Windows Server VMs in Azure.

In the meantime, feel free to share some feedback or ask questions via Twitter (http://twitter.com/mszcool).

GeRes2 – An OSS Framework for Reliable Execution of Background Jobs on Azure

A few weeks ago we published an open source framework that covers a scenario required by many partners I’ve been working with on Azure over the course of the past 4 years: asynchronous execution of jobs on compute instances in the background – or simple batch processing. Finally now I found the time (on the plane) to blog a bit about it and point you to that framework.

Being one of the product owners of this OSS framework I am very proud of what we achieved. Note that there might be something way more sophisticated and complete coming as a platform Service in Microsoft Azure at some point in time in the future. Until that time, our OSS Framework fills this requirement of “batch-processing” in a more simplified way.

Generic Resource Scheduler v2.0 (GeRes2) for Microsoft Azure

In a nutshell, GeRes2 is a framework for scalable, reliable and asynchronous execution of a large amount of jobs on many Worker Machines in Windows Azure. It is built with Azure Cloud Services, provides a Web API for submitting jobs and querying data about jobs and executes Jobs in parallel across multiple worker roles.

As such, GeRes2 implements a ready-to-use framework which you can use in your own solutions. It is all available as open source and you can use the code or the deployment packages to get GeRes2 running in your own subscriptions.

http://geres2.codeplex.com

GeRes2 essentially implements an all-so well-known pattern from the Azure-world as shown in the sketch below:

Classic scenarios partners are implementing through a pattern like this are:

– Generating documents in the background (e.g. massive amount of documents)
– Image or Video post processing (although for Video you should consider WAMS)
– Complex, distributed calculations
– Asynchronous provisioning jobs (e.g. when provisioning new tenants for your system)
– anything that needs to run multiple jobs in parallel reliably and asynchronously…

Of course this outlines just the generic pattern and it looks fairly simple. But when digging into details there are a whole lot of challenges that need to be solved when implementing such an architecture such as:

– Execution of long-running and short-running jobs across multiple instances
– Status tracking & status-querying of jobs
– Notifications (status & progress) to consumers and clients
– Prioritization of jobs
– Reliability features (e.g. retry logic, dead-letter handling etc.)
– AutoScaling that is aware of which instances are not executing long-running jobs

Exactly those are the features GeRes2 implements for you – so you can take care of the implementation and scheduling of your jobs. For the while GeRes2 fills a gap that might be filled through a native Azure platform service later (as of today no details are available).

Side Note: We built GeRes2 because we do have partners that needed the functionality outlined below right now. Note that some for you obvious features might be missing since partners we’ve worked with did not immediately require those features, today.

Side Note: if you have HPC-background than you might think about a separation of jobs and tasks (jobs as compositions of more fine-granular tasks). In GeRes2 we don’t make that separation, so for GeRes2 Jobs are equal to Tasks and you can use that tger

Backend Work: Implementing a Job for GeRes2

Creating jobs is fairly easy (whereas we focused on .NET developers, first): You need to create a library that implements an Interface which contains the logic of your job. The interface is part of the server-SDK from GeRes2 that defines the IJobImplementation-interface.

public class FinancialYearEnd : IJobImplementation
{
    // make the property volatile as it may be set by more than one thread
    private volatile bool _cancellationToken = false;


    // default constructor (required by MEF)
    public FinancialYearEnd()
    {
    }

    public JobProcessResult DoWork(Job job,
        string jobPackagePath, string jobWorkingPath, 
        Action<string> progressCallback)
    {
        // Your Job-logic goes here
    }

    public string JobType
    {
        get { return "FinancialYearEnd"; }
    }


    public void CancelProcessCallback()
    {
        _cancellationToken = true;
    }
}

Every job gets executed on a single Worker Role instance reserved by GeRes2 for job-execution. The number of jobs executed in parallel is defined by the number of workers you reserve for the job execution. On each worker. every job gets his own, isolated directory where the job files (executables, anything required by a job) are temporarily stored for execution as well as a temporary folder available for local processing.

Job files are deployed automatically by GeRes2 from Azure Blob storage to the worker VMs on job execution. So all you need to do is implement your job, package it in a ZIP-archive and push it into a container called “jobprocessorfiles” in Blob Storage (in a sub-folder matching your TenantName). For more details look at the documentation Wiki from the GeRes2 OSS project workspace.

Front-End Work: Submission of Jobs for Execution and Notifications

When GeRes2 is deployed in your subscription it provides a WebAPI secured by Azure Active Directory. More details are also available on the documentation wiki from GeRes2. We also provide a client/consumer-SDK for applications and services that want to use GeRes2 for submitting jobs.

With that SDK the submission of a Job is as simple as shown in the following lines of code:

var geresServiceClient = new GeresAzureAdServiceClient
                            (
                                baseUrl,
                                azureAdTenant,
                                azureAdWebApiId
                            );

geresServiceClient.Authenticate(clientId, clientSecret).Wait();

var newJob = new Job
{
    JobType = "FinancialYearEnd",
    JobName = "Job Display Name",
    TenantName = "YourTenant",
    JobProcessorPackageName = "yourjobpackage.zip",
    Parameters = "any parameters you need to pass in"
};

string jobId = geresServiceClient.Management.SubmitJob(newJob, batchId).Result;

This leads to the execution of a job in the GeRes2 system on a worker. The status of a job can be tracked in two ways:

– Through “near-real-time” notifications via SignalR and/or Service Bus.
– By polling the monitoring API of GeRes2.

Setting up notifications with SignalR can happen through the client-SDK, directly, as shown in the following code-snippet:

//
// Setup notification handlers
//
geresServiceClient.Notifications.JobStarted += (sender, args) =>
{
    // ...
};
geresServiceClient.Notifications.JobProgressed += (sender, args) =>
{
    // ...
};
geresServiceClient.Notifications.JobCompleted += (sender, args) =>
{
    // ...
};

//
// Connect to the SignalR Hub
//
geresServiceClient.Notifications.Connect().Wait();

//
// Schedule jobs and subscribe to job notifications
//
string jobId = geresServiceClient.Management.SubmitJob(newJob, batchId).Result;
geresServiceClient.Notifications.SubscribeToJob(jobId);

As mentioned, there’s also the option to query the status of jobs manually. The status of jobs is kept in an Azure-table. Of course you can query that table, directly since you are in full control of your GeRes2 deployment. But we’ve created a WebAPI and a corresponding client-API that should make it a bit easier for you as shown below:

var batchesTask = geresServiceClient.Monitoring.GetBatches();

geresServiceClient.Monitoring.PageSize = pageSize;
var jobsList = geresServiceClient.Monitoring.GetJobsFromBatch(batchIdToGetJobs);

var jobDetails = geresServiceClient.Monitoring.GetJobDetails(j.JobId);

Final remarks:

We’ve created GeRes2 based on concrete partner projects we’ve been involved into across the world. They all started implementing the pattern outlined above manually and it always turned out it is a whole lot of work that needs to get done.

GeRes2 implements a simple set of batch-processing features we’ve seen a lot with partners across the world. Of course it is not a full-blown HPC or batch-processing/job-scheduling platform. It covers simple yet often required scenarios. Nevertheless, what I can confirm is that we’ve partners who are using GeRes2 with > 100 job execution instances and tested way above 100 instances (to be precise: with 800 instances; but note that the built-in AutoScaler works up to 100 instances, only, but is not required to be used, either).

If you’re interested, then go to our project workspace, download and play the source code, dig into our docs and share feedback if you have.

http://geres2.codeplex.com
is where you can look for further details.

Have much fun and thanks for your interest!