Single-Sign-On with SAP HANA, Azure Active Directory and Office 365

Signle-Sign-On with Azure Active Directory for HANA

At last SAP Sapphire (May 2017) we announced several improvements and also new offerings for SAP on Azure as you can read here. The most prominent ones are more HANA Certifications as well as SAP Cloud Platform on Azure (as you can read from my last blog post specifically focused on SAP CP.

One of the less discussed and visible announcements, despite being mentioned, is the broad support of Enterprise-Grade Single-Sign-On across many SAP technologies with Azure Active Directory. This post is solely about one of these offerings – HANA integration with Azure AD.

Pre-Requisites for HANA / AAD Single-Sign-On

An integration of HANA with Azure AD (AAD) as primary Identity Provider works for HANA Instances you can run anywhere you want (on-premises, any public IaaS, Azure VMs or SAP Large Instances in Azure). The only requirement is, that the end-user accessing apps (Web Administration, XSA, Fiori) running inside of the HANA instance has access to the Internet to be able to sign-in against Azure AD.

For this post, I start with an SAP HANA Instance that runs inside of an Azure Virtual Machine. You can deploy such HANA instances either manually or through the SAP Cloud Appliance Library.

In addition to just running HANA, I’ve also installed XRDP for the Linux VM in Azure and SAP HANA Studio inside of the Virtual Machine to be able to perform necessary configurations across both, the XSA Administration Web Interface as well as HANA Studio as needd.

Finally, you need to have access to an Azure Active Directory tenant for which you are the Global Administrator or have the appropriate permissions to add configurations for Enterprise Applications to that Azure AD Tenant!

The following figure gives an overview of the HANA VM environment I used for this blog-post. The important part is the Azure Network Security Group which opens up the ports for HTTP and HTTPS for HANA which are following the pattern 80xx and 43xx for regular HTTP and HTTPS, respectively.

HANA VM in Azure Overview

Azure Active Directory Marketplace instead of manual configuration

SAP HANA is configured through the Azure Active Directory Marketplace rather than the regular App Registration model followed for custom developed Apps in Azure AD. There are several reasons for this, here are the most important ones outlined:

  • SAML-P is required.Most SAP assets follow SAML-P for Web-based Single-Sign-On. While it is possible in Azure AD, when setting it up manually with advanced options, Azure AD Premium Edition is required. For offerings from the Azure AD Marketplace (Gallery), standard edition is sufficient. While that’s not the primary reason, it’s a neat one!
  • Entity Identifier Formats for SAP Assets.When registering an application in Azure AD through the regular App Registration model, Application IDs (Entity IDs in federation metadata documents) are required to be URNs with a protocol prefix (xyz://…). SAP applications use Entity IDs with arbirtray strings not following any specific format. Hence a regular app registration does not work. Again, this challenge can be solved through the Enterprise App Integration in AAD Premium. But when taking the pre-configured Offering from the Marketplace, you don’t need to take care of such things!
  • Name ID formats in issued SAML Tokens.Users are typically identified using Name ID assertions (claims). In requests, Azure AD accepts nameid-format:persistent, nameid-format:emailAddress, nameid-format:unspecified and nameid-format:transient. All of these are documented here in detail. Now, the challenge here is:
    • HANA sends requests with nameid-format:unspecified.
    • This leads to Azure AD selecting the format for uniquely identifying a user.
    • But HANA expects the Name ID claim to contain the plain user name (johndoe instead of domain\johndoe or johndoe@domain.com).
    • This leads to a mismatch and HANA not detecting the user as a valid user even if the user exists inside of the HANA system!

    The Azure AD Marketplace item is configured and on-boarded in a way, that enables this technical challenge to be resolved.

  • Pre-configured claims.While that’s not a need for HANA in specific, for most of the other SAP-related offerings, the marketplace-based integrartion pre-configures the SSO-configuration with claims/assertions typically required by the respective SAP technology.

Step #1 – Register HANA in Azure Active Directory

Assuming you have HANA running in a VM as I explained earlier in this post, the first step to configure Azure AD as an Identity Provider for HANA is to add HANA as an Enterprise Application to your Azure AD Tenant. You need to select the offer as shown in the screen shot below:

Selecting the HANA AAD Gallery Offering

Within the first step, you just need to specify a Display Name for the app as shown in the Azure AD management portal. The details are configured later down the road as next steps. Indeed you can get more detailed instructions from within the Azure AD management portal, directly. Just open up the Signle Sign-On-section, select SAML-based Sign-On in the very top Dropdown-Box, then scroll to the bottom and click the button for detailed demo instructions.

Detailed Demo instructions for SAML-P

If you’re filling out the SAML-P Sign-In settings according to these instructions, you’re definitely on a good path. So, let’s just walk through the settings so you get an example of what you need to enter there:

  • Identifier: should be the Entity ID which HANA uses in it’s Federation Metadata. Needs to be unique across all enterprise apps you have configured. I’ll show you later down in this post, where you can find it. Essentially you need to navigate to HANA’s Federation Metadata in the XSA Administration Web Interface.
  • Reply URL: use the XSA SAML login endpoint of your HANA system for this setting. For my Azure VM, it had a public IP address bound to the Azure DNS name marioszpsaphanaaaddemo.westeurope.cloudapp.azure.com, therefore I had to configure https://marioszpsaphanaaaddemo.westeurope.cloudapp.azure.com:4300/sap/hana/xs/saml/login.xscfunc for it.
  • User Identifier: this is one of the most important settings you must not forget. The default, user.userprincipalname will NOT work with HANA. You need to select a function called ExtractMailPrefix() in the Dropdown and select the user.userprincipalname for the Mail-parameter of this function.

Detailed Settings Visualized

Super-Important: Don’t ignore the information-message shown right below the certificate list and the link for getting the Federation Metadata. You need to check the box Make new certificate active so that the signatures will be correctly applied as part of the sign-in process. Otherwise, HANA won’t be able to verify the signature.

Step #2 – Download the Federation Metadata from Azure AD

After you have configured all settings, you need to save the SAML configuration before moving on. Once saved, you need to download the Federation Metadata for configuring SSO with Azure AD within the HANA administration interfaces. The previous screen-shot highlights the download-button in the lower, right corner.

Downloading the federation metadata document is the easiest way to get the required certificate and the name / entity identifier configured in your target HANA system.

Step #3 – Login to your HANA XSA Web Console and Configure a SAML IdP

We have done all required configurations on the Azure AD side for now. As a next step, we need to enable SAML-P Authentication within HANA and configure Azure AD as a valid identity provider for your HANA System. For this purpose, open up the XSA web console of your HANA System by browsing to the respective HTTPS-endpoint. For my Azure VM, that was:

https://marioszpsaphanaaaddemo.westeurope.cloudapp.azure.com:4300/sap/hana/xs/admin

Of course, HANA will still redirect you to a Forms-based login-page because we have not configured SAML-P, yet. So, sign-in with your current XSA Administrator Account in the system to start the configuration.

Tip: take note of the Forms-Authentication URL. If you break something in your SAML-P configuration later down the road, you can always use it to sign back in via Forms Authentication to fix the configuration! The respective URL to take note of, is: https://marioszpsaphanaaaddemo.westeurope.cloudapp.azure.com:4300/sap/hana/xs/formLogin/login.html?x-sap-origin-location=%2Fsap%2Fhana%2Fxs%2Fadmin%2F.

Now the previously downloaded federation metadata document from Step #2 above becomes relevant. In the XSA Web Interface, you need to navigate to SAML Identity Providers and from there, click the “+”-button on the bottom of the screen. In the form opening now, just paste the previously downloaded federation metadata document into the large text box on the top of the screen. This will do most of the remaining job for you! But, you need to fix a few fields.

  • The name in the General Data must not contain any special characters, also no spaces.
  • The SSO URL is not filled by default since we don’t have it in the AAD metadata, yet. So you need to manually fill it as per the guidance from within the Azure AD portal shown above in this post.

HANA SAML IdP Data Filled

Since we are in the HANA XSA tool, it’s the right point in time to show you, where I retrieved the information required earlier in the Azure AD portal when registering HANA as an App there – the Identifier as shown in the last screen shot from the Azure AD console, above.

Indeed, these details are retrieved from the SAML Service Provider configuration section as highlighted in the screen shot below. A quick side-note: this is one of the rare cases where I constantly needed to switch to Microsoft Edge as a browser instead of Google Chrome. For some reasons, in Chrome I was unable to open the metadata tab, while in Edge I typically can open the metadata-tab which shows the entire Federation Metadata document for this HANA instance. From there, you can also grab the identifier required for Azure AD since this is the Entity ID inside of the Federation Metadata document.

HANA SAML Federation Metadata

Ok, we have configured Azure AD as valid IdP for this HANA system. But we did not really enable SAML-based authentication for anything. This happens now at the level of applications managed by the XS-environment inside of HANA (that’s how I understand this with my limited HANA knowledge:)). You can enable SAML-P on a per-package basis inside of XSA, means it’s fully up to you to decide for which components you plan to enable SAML-P and for which you stay with other authentication methods. Below a screen shot that enables SAML-P for SAP-provided package! But stick with a warning: if you enable SAML-P for those, this might also have impact on other systems interacting with those packages. They should probably also support SAML-P as a means of authentication, especially if you disable other options, entirely!

HANA SAML Federation Metadata

By enabling the sap-package for SAML-P, we get SSO based on Azure AD for a range of built-in functions including the XSA web interface, but also Fiori-interfaces hosted inside of the HANA instance for which you configured the setting.

Step #4 – Troubleshooting

So far so good, seems we could try it out, right? So, let’s logout, open an In-Private-Browsing session with your browser of choice and navigate to your HANA XSA Administration application, again. You will see, that this time by default you will get redirected to Azure AD for signing into the HANA System. Let’s see what happens when trying to login with a valid user from the Azure AD tenant.

HANA SAML Federation Metadata

Seems the login was not so successful. Big question is why. This is now where we need access to the HANA system with HANA Studio and access to the system’s trace log. For my configuraiton, I installed XRDP on the Linux machine and have the HANA Studio running directly on that machine. So, best way to start is connecting to the machine, starting HANA Studio and navigating to the system configuration settings.

HANA Diagnosis for Sign-In Failing

The error-message is kind of confusing and miss-leading, though. We spent some time when onboarding HANA into the AAD Marketplace to figure out what was going wrong. So much ahead – Fiddler-Traces and issues with Certificates where not the problem! The resolution for this is to be found in an entirely different section. Nevertheless, I wanted to show this here, because it really is extremly valuable to understand, how-to troubleshoot stuff when it’s not going well.

The main reason for this failure is a miss-match in timeout configurations. The signatures are created based on some time stamps. One of those timestamps is used for ensuring, that authentication messages are valid only a given amount of time. That time is set to a very low limit in HANA by default resulting into this, quite miss-leading error message.

Anyways, to fix it, you need to stay in the HANA System Level Properties within HANA Studio and make some tweaks and adjustments. Within the system properties of the Configuration Tab, just filter settings by SAML and adjust the assertion_timeout setting. It’s impossible to do an entire, user-driven sign-in process within 10 sec. Think about it, the user navigates to a HANA App, gets re-directed to Azure AD, needs to enter her/his username/password, then eventually there’s Multi-Factor-Auth involved and finally upon success, the user gets redirected back to the respective HANA application. Impossible within 10sec. So, in my case, I adjusted it to two min.

HANA Diagnosis for Sign-In Failing

Btw. this behavior and required configuration is also documented on official SAP Support Notes as an outcome of the collaboration between us and the SAP HANA team as part of enabling SSO with Azure AD (thanks for the great collaboration, again:)):

SAP Support Note 2476310 – SAML SSO to HANA System Using Microsoft AAD – Your Browser Shows the Error “Assertion did not contain a valid Message ID”

Ok, time for a next attempt. Now, if you still have the same error message about not being able to validate the signature, you probably forgot something earlier in the game. Make sure that when configuring HANA in Azure AD you make the certificate active by hitting the Make new certificate active checkbox I’ve mentioned earlier… below the same screen shot with the important informational message, again!

Don't forget Make new certificate active

Step #4 – Configuring a HANA Database User

If you’ve followed all the steps so far, the Sign-In with a User from Azure AD will still not succeed. Again, the trace logs from HANA are giving more insights on what’s going on and why the sign-in is failing this time.

Trace about User does not exist in HANA

HANA is complaining that it does not know about the user. This is a fair complaint since Azure AD (or any other SAML Identity Provider) takes care for authentication, only. Authorization needs to happen in the actual target system (the service provider, also often called relying party application). For being able to Authorize, the user needs to be known to the service provdier. That means, at least some sort of user entity needs to be configured.

  • With HANA, that means you essentially create a database user and enable Single Sign-On for this database user.
  • HANA then uses the NameID-assertion from the resulting SAML token to assign the user authenticated by the IdP, Azure AD in this case, to map that successfully authenticated user to a HANA database user. This is where the format of the NameID in the issued token is so important and why we had to configure the ExtractMailPrefix()-strategy in the Azure AD portal as part of Step #1.

So, to make all of this happen and finally get to a successful login, we need to create a user in HANA, enable SSO and make sure that user has the appropriate permissions in HANA to e.g. access Fiori Apps or the XSA Administration Web Interface. This happens in HANA Studio, again.

Detailed Settings Visualized

Super-Important: The left-most part of the figure above visualizes the mapping from the SAML-Token’s perspective. So it defines the IdP as per the previous configurations and the user as it will end up being set in the NameID-assertion of the resulting SAML-token. With Azure AD users, these will mostly be lower-case! Case matters here!!! Make sure you enter the value lower-case here, otherwise you’ll get a weird message about dynamic user creation failing!!!

The next step is to make sure that the user has the appropriate permissions. For me, as a non-HANA-expert, I just gave the user all permissions to make sure I can show success as part of this demo. Of course, that’s not a best practice. You should give those permissions appropriate for your use cases, only.

Detailed Settings Visualized

Step #5 – A Successful Login

Finally, we made it! If you have completed all the steps above, you can start using HANA with full Single-Sign-On across applications also integrated with your Azrue AD tenant. For example, the screen shot below shows my globaladmin-user account signing into the HANA Test VM I used, navigating to the HANA XSA Administration web console and then navigating from there to Office 365 Outlook… It all works like a charm without me being required to enter credentials, again!

Detailed Settings Visualized

That is kind-of cool, isn’t it! It would then even work with navigating back and forth between those environments. Now, this scenario would work for any application that runs inside of the XS-environment.

But for now, at least for enterprise administrators, it means they can secure very important parts of their HANA systems with a prooven Identity platform using Azure AD. They can even configure Multi-Factor Authentication in Azure AD and thus even further protect HANA environments along other applications using the same Azure AD tenant as an Identity Provider.

Final Words

Finally, this is the simplest possible way of integrating Single-Sign-On with SAP applications using Azure AD, only. SAP Netweaver would be similarly simple as it is documented here. There’s even a more detailed tutorial available for Fiori Launch Pad on Netweaver based on these efforts we’ve implemented on SAP blogs here.

The tip of the iceberg is then the most advanced SSO we’ve implemented with SAP Cloud Platform Identity Authentication Services. This will give you centralized SSO-management through both company’s Identity-as-a-Service offerings (Azure AD, SAP Cloud Platform Identity Services). As part of that offering, SAP even includes automated identity provisioning which would remove the need for manually creating users as we did above.

I think, over the past year, we achieved a lot with the partnership between SAP and Microsoft. But if you ask for my personal opinion, I think the most significant achievements are HANA on Azure (of course, right:)), SAP Cloud Platform on Azure and … the Single-Sign-On Offerings across all sorts of SAP technologies and services!

I hope you found this super-interesting. It is most probably my last blog post as member of the SAP Global Alliance Team from the technical side since I am moving forward to the customer facing part of Azure Engineering (Azure Customer Advisory Team) as an engineer. Still, I am part of the family and will engage as needed out of my new role with SAP, that’s for sure!

Cloud Foundry – SAP Cloud Platform on Azure

Cloud Foundry – SAP Cloud Platform on Azure (Beta)

The main project that kept me busy for the past 7 months was the work with SAP Labs Israel to get a beta version of SAP Cloud Platform running on Azure. During SAP Sapphire, SCP on Azure got now announced together with other public cloud vendors as part of SAP’s multi-cloud strategy (see here and here. It is currently available as public beta for anyone to try on Azure. Here I dig into some first basics…

Note: as outlined in my previous blog post about HANA Express, please note that this is my personal blog which reflects my personal thoughts instead of Microsoft’s official opinion. For official announcements and statements, please refer to the Microsoft Azure Blog.

Cloud Foundry on Azure and SCP on Azure

Before digging into the details of SAP Cloud Platform (short: SCP), just a quick reminder about how we support Cloud Foundry as a platform on Azure in general.

With SCP, we now support the 2nd commercially available Cloud Foundry flavor on Azure next to Pivotal Cloud Foundry. But, what makes SCP on Azure really different to the offers we had so far is, that it is a fully managed PaaS… managed by SAP. So using SCP on Azure is exactly the same as on any other public cloud platform which gives you full portability in a multi-cloud world. We do have a few differentiators in my opinion, though:

Finally, I strongly believe that from the work we’ve done with SAP, Cloud Foundry on Azure benefits across the board, in general. We’ve made numerous improvements to the Bosh CPI implementation for Azure as part of the efforts and we fixed several bugs. Examples of that are e.g. the Keep-Alive of unhealthy virtual machines for debugging purposes or support for managed disks (which was on the plans, already, but we needed it earlier:)).

Until General Availabiltiy of SAP Cloud Platform on Azure, you’ll see many more improvements in our CPI which I cannot elaborate in detail at the moment.

SAP Cloud Platform on Azure – Getting Started

To get SCP on Azure, you work through the regular SCP administation cockpit. So your starting point is the same as for any other work you do with SCP. You register or log-in as usual, nothing Azure specific with that. For activating Azure, which is at the time of writing this post available as Beta+Trial, you click on the “Start Cloud Foundry Trial”-button within the cockpit!

https://account.hanatrial.ondemand.com

SCP Cockpit Login and highlight activation button

When clicking the button, the SCP cockpit will ask you for selecting a region in which you want to activate the trail. Now, within SCP, activated Azure regions do just appear as an SCP-region within the cockpit. That means, you would pick an SCP-region for the activation that runs on-top of Azure as shown below:

Pick the SCP region

Until now, I’ve done several trial activations. It usually takes a minute till the tenant is activated and available for you. For Cloud Foundry knowledable folks, what happens is the creation of an organization and a space for you within the SCP Cloud Foundry environment.

Exploring what’s available

After that process is completed, you can navigate to your organization and space within the SCP cockpit. Through the cockpit, you have convenient ways to browse and montior deployed applications or also browse the marketplace to review, which backing services SAP has enabled for the Beta+Trial which we’ve enabled on Azure. Cloud Foundry veterans will find it quite easy to navigate through the portal.

SCP Marketplace

Now, since this is a beta as well as a trial, the backing services are all running as single instances in containers behind the scenes. I think that’s more than fair because you’re not paying for the service as long as it is in beta. And… this is also the general approach followed by SAP for using trial accounts on generally available regions, as well. For GA and production (non-trial) environments, these backing services will run in Virtual Machine clusters, of course.

The exception even for today in that list is… guess what… SAP HANA of course. The HANA backing service offered as part of the Azure SCP beta is a multi-tenant, shared HANA environment that runs in Virtual Machines behind the scenes.

Note: When using any of the backing services from the SCP market place, you indeed use SAP-deployed backing services. That gives you a higher level of portability since SAP and not Azure controls the versions and configurations used for the technologies that are the foundation for those backing services. Now, if you still would prefer to use native Azure services, you can use the Azure Meta Service Broker to hook them up with applications you run in SCP the Cloud Foundry way (you can use them, directly, as well, of course). But in that case, you need to be aware that this has impact on how portable your applications are across multiple cloud vendors if that’s relevant for you.

Using the Cloud Foundry CLI

Now, the cool thing about SCP is, that it is Cloud Foundry. It’s a Cloud Foundry PaaS enriched by many SAP services (ok, it will take a while till we have them all active on Azure as that requires further engineering work). It ultimately means, you manage and work with it as with other Cloud Foundry environments including the CLI and the APIs. For SAP-specific services such as HANA, SAP provides plug-ins for the CLI to make it easier to work with those services such as the HANA MTA plug-in which I think SAP announced during this year’s Sapphire, as well.

For CLI/API interaction purposes, SCP uses different API endpoints for each region. When using the Azure Beta region which we’ve deployed right now, you’d use the following API endpoint:

cf api api.cf.us20.hana.ondemand.com
cf login

From that moment forward, everything looks and feels like Cloud Foundry. You can push apps, you can explore your marketplace etc. The learning curve is quite easy for Cloud Foundry experienced developers, I’d say:

SCP CF CLI in Action

Of course, I was curious what development runtimes SAP has enabled and as a .NET developer I wanted to know about .NET. Looking at the screen shot above, I found it great that .NET Core was enabled. But be aware: SAP typically offers some sort of enterprise support for languages and runtimes in their environment, but from the build packs listed, only a few enjoy that status. .NET Core does not. So, check back with SAP on that… .NET Core only gets community-level support through the Cloud Foundry Foundation.

Using the Azure Meta Service Broker with SAP Cloud Platform

When you’re running on Azure, it makes sense to use Azure-native services if you want to. At the end of the day, that’s when you can unleash the full power of running on a specific cloud platform, right:)? And… you still could implement your “portability” thorugh IoC/DI at the application level to have a more effective/efficient integration into native cloud services but still stay portable to a certain extent.

Fortunately, SAP allows you to enable the Azure Meta Service Broker at the space-level when using SCP. I thought I wanna try that out again now this is in beta (we prototyped that in early phases). Essentially, you really only need to follow the standard-procedure to enable the Azure Meta Service Broker as documented here.

The first step, though, is to create an Azure SQL Database instance and a Service Principal in your Azure Active Directory, as shown below. The Service Principal is needed because the service broker dynamically creates/deletes resources as they are requested through the Cloud Foundry API (or the CLI).

$ az ad app create --display-name=yourazureadappdisplayname --homepage=http://yourhomepage.com --identifier-uris="https://yourazureadserviceprincipalname" --key-type=Password --reply-urls="https://notreallyimportantwhatyouputhere"
AppId                                 DisplayName                  Homepage                 ObjectId                              ObjectType
------------------------------------  ---------------------------  -----------------------  ------------------------------------  ------------
yourazureadserviceprincipalappid      yourazureadappdisplayname    http:/yourhomepage.com   thecreatedappobjectid                 Application
$
$ az ad sp create --id https://yourazureadserviceprincipalname
AppId                                 DisplayName                  ObjectId                              ObjectType
------------------------------------  ---------------------------  ------------------------------------  ----------------
yourazureadappdisplayname             yourazureadappdisplayname    thecreatedappobjectid                 ServicePrincipal
$
$ az ad app update --id=yourazureadserviceprincipalappid --password="yourazureadserviceprincipalpassword"
$

Don’t forget to give the created service principal contributor-rights to the resource group you want to use for the resources of the Meta Service Broker.

After you’ve done the basic creation of the assets mentioned above, the next step is to clone the Azure Meta Service Broker repositiory and adjust the Cloud Foundry Application Manifest for it. The Meta Service Broker essentially is a CF-application that represents the Service Broker for several Azure Services as documented. You can run it wherever you want, but it’s built for being pushed as an application into CF with configuration data about your Azure Subscription to use for the resources created by the Meta Service Broker. Here’s a sample configuration, so that’s how a filled manifest.yml for the Azure Meta Service Broker looks like.

Note that the parameters for SUBSCRIPTION_ID, TENANT_ID, CLIENT_ID and CLIENT_SECRET need to match those you’ve used or retrieved through the Azure CLI commands above as they represent your subscription as well as the servvice principal you’ve created above! Ok, the commands above don’t give you the subscription ID. You can retrieve that by issuing az account list --output=json or by looking it up in the Azure Portal.

---
applications:
- name: meta-azure-service-broker
  buildpack: https://github.com/cloudfoundry/nodejs-buildpack
  instances: 1
  env:
    ENVIRONMENT: AzureCloud
    SUBSCRIPTION_ID: yoursubscriptionid
    TENANT_ID: yourazureadtenantidfromtheserviceprincipal
    CLIENT_ID: yourazureadserviceprincipalappid
    CLIENT_SECRET: yourazureadserviceprincipalpassword
    SECURITY_USER_NAME: usertoauthenticateagainstmetaservicebroker
    SECURITY_USER_PASSWORD: passwordtoauthenticateagainstmetaservicebroker
    AZURE_BROKER_DATABASE_PROVIDER: sqlserver
    AZURE_BROKER_DATABASE_SERVER: yourazuresqldbserver.database.windows.net
    AZURE_BROKER_DATABASE_USER: yourazuresqldbuser@yourazuresqldbserver
    AZURE_BROKER_DATABASE_PASSWORD: yourazuresqldbpassword
    AZURE_BROKER_DATABASE_NAME: yourazuresqldbdatabasename
    AZURE_BROKER_DATABASE_ENCRYPTION_KEY: 32charactersofyourchoicegohereok
    AZURE_SQLDB_ALLOW_TO_CREATE_SQL_SERVER: false
    AZURE_SQLDB_ENABLE_TRANSPARENT_DATA_ENCRYPTION: true

Assuming you have logged-in with cf login as mentioned above, you can just push the meta service broker application into your SCP tenant using cf push. Important tip: since the settings specified above in the manifest are set as environment variables, be careful with special characters used by bash when defining the passwords. Either avoid or escapte them!! A successful deployment of the meta service broker app should end up with the following output on your console:

Service Broker App Pushed

When you’ve pushed the application, you only created the meta service broker but you did not tell Cloud Foundry that this is indeed a Service Broker and not any other type of general application. So, as a next step after the meta service broker has been deployed, successfully, is to register it as a service broker. The steps for that are:

  • Review the deployed application and take note of it’s entry point URL.
  • Create the service broker using that URL. You also need to authenticate against the meta service broker. For that purpose, use the previously specified SECURITY_USER_NAME and SECURITY_USER_PASSWORD parameters from the manifest file above.
$ cf apps
Getting apps in org yourorgname_trial / space dev as your@username.com...
OK

name                        requested state   instances   memory   disk   urls
meta-azure-service-broker   started           1/1         1G       1G     urlfromyourapppush.cfapps.us20.hana.ondemand.com
$
$
$ cf create-service-broker nameofyourbroker usertoauthenticateagainstmetaservicebroker passwordtoauthenticateagainstmetaservicebroker  https://urlfromyourapppush.cfapps.us20.hana.ondemand.com --space-scoped

Important Note: since SCP is a multi-tenant Cloud Foundry environment, you need to know that you are only the administrator of the spaces you’re creating. Permissions to organizations are limited and you don’t have any permissions beyond organizations. That means when activating the service broker, you need to use the --space-scoped switch to activate the broker on your space. When you do that, you are using a so called private broker with private plans which become active, automatically. That means, although the documentation of the Azure Meta Service Broker states it, you do not need to call cf enable-service-access since all services are enabled for your space in that case, by default. A simple cf marketplace confirms that:

cf marketplace results

As a next step, you can start using the services exposed by the Azure Meta Service Broker. At the time of writing this post, those services included: Azure Storage, Azure SQL DB, Azure DocumentDB, Azure Service Bus, Azure Redis Cache and Azure Key Vault services. Means you can start using those services right away. If you have asks/requests for additional services Microsoft should enable, I think you’re best by posting requests/issues into the GitHub Repository for the Azure Meta Service Broker or file a pull request.

The way how you make use of the services follows standard cloud foundry patterns with service brokers. You create a service and then you bind the service to one or many applications using the Cloud Foundry service broker APIs or CLI-commands. Let’s create an Azure Storage account that way so you get a sense of it. The approach is always the same for all services offered through the Azure Meta Service Broker.

First you create a parameters-file specifying some of the provisioning paramters for creating the respective Azure service. An example of such a parameters JSON-file for creating an Azure storage account via the service broker looks as follows:

{
  "resource_group_name": "marioszpScpMetaBroker",
  "storage_account_name": "marioszpscpteststrg",
  "location": "westus",
  "account_type": "Standard_LRS"
}

With this JSON-file in-place, you can leverage the service broker CLI commands or Cloud Foundry APIs to create a new service and make Cloud Foundry aware of it:

$ cf create-service azure-storage standard marioszpscpstoragetest -c .\params.json

$ cf service marioszpscpstoragetest

Service instance: marioszpscpstoragetest
Service: azure-storage
Bound apps:
Tags:
Plan: standard
Description: Azure Storage Service
Documentation url:
Dashboard:

Last Operation
Status: create succeeded
Message: Created the storage account, state: Succeeded
Started: 2017-05-19T11:02:21Z
Updated: 2017-05-19T11:02:40Z
$

Reults of creating a service with the broker

Now, once this is done you can check on the provisioning status and availability of that service using the cf service marioszpscpstoragetest command which gives you all the details about the created service. Finally, by using the cf bind-service command, you can bind the service to any application you’ve deployed. Configuration parameters will then be exposed through environment variables as per the documentation of the respective service in the Azure Meta Service Broker GitHub. For an Azure storage account, an environment variable with a JSON-document will be added to your applications Environment variables that contains e.g. the storage account name and storage account keys you can use to authenticate and execute calls against the created storage account.

What’s really cool now is, that all of what I’ve done above with the Cloud Foundry CLI is visible in the SAP Cloud Platform Cockpit, as well. For example, since I’ve activated the Azure Meta Service Broker, you’ll see all the services in the marketplace in the portal as well. Even the Azure Storage Account which is surfaced as a service as shown below. That means the Azure Meta Service Broker really brings SCP and Azure close together and allows you to look at Azure resources through the SCP as well as the Azure management perspectives.

SCP Cockpit with Azure Services

Final Words

This is just the beginning of SAP Cloud Platform on Azure. Note that we’ve announced Beta. During our path forward for becoming generally available, we’ll get to see several improvements in Azure’s Bosh CPI and optimized backing services (currently all except HANA run in single instance containers in the Beta/Trial).

There’s one question a lot of people ask when looking at something like this, though: how does that relate to native Azure PaaS services such as Azure App Services or Service Fabric? The answer is simple: Azure is an open Cloud Platform. Microsoft’s strategy is clearly to enable as many relevant platforms as possible in addition to Microsoft’s own technology. This gives customers and partners choice and allows them to leverage the skills and knowledge they have available in their portfolio, already. Cloud Foundry and it’s flaviours do fit very well into this strategy. If you prefer Cloud Foundry based PaaS platforms, then Azure is a place where you can go… and if you want you can even integrate with all the native goodness of Azure including especially Azure Active Directory for Single Sign-On or all of the other services in addition to those offered with the Azure Meta Service Broker.

Developed an SAP HANA Express Azure Quick Start Template

Context

This is an exciting week for me… although I am usually not that much into attending Business-oriented conferences, over the past two years I did so by attending SAP Sapphire.

Up until know, that mainly is caused by the fact that I’ve contributed some aspects to what was announced with regards to the partnership between Microsoft and SAP at the conference. Last year, my part was mainly about Office 365 with the work I’ve supported for Concur, Ariba, Fieldglass and SuccessFactors as well as the Sports Basement Demo which was shown in the Key Note to highlight the HANA One Certification for Azure in our DS14 VM-Series at that time.

While I cannot write about the major project I supported for this year, yet, here’s a little nugget to get started with – a Quick Start Template for SAP HANA Express!

Important Note: While I am working for Microsoft, this blog summarizes my personal opinions and my personal understanding of topics. This means that all I am writing here is not related to Microsoft’s official opinion, at all! If you want to get that view, look at the Azure Blog or Jason Zander’s Blog for official announcements!

Important Note: The Pull-Request into the official Microsoft Azure Quick Start Templates GitHub repository is not completed, yet. Therefore, the link still redirects to the working branch in my GitHub repository. I’ll update the blog-post as soon as the pull request is completed (there’s currently a general issue with the Travis CI pipeline used for validation on the Azure Quick Start Templates GitHub repository that caused an unexpected delay for the pull request to go through in time for Sapphire).

SAP HANA Express

As you might know, already, SAP HANA is SAP’s in-memory Database Engine which is supposed to back all of the major future releases of SAP’s core business suites centered around S/4HANA. But, HANA can be used as a stand-alone database system for developing custom solutions, as well. The Sports Basement Demo on Azure DS14 Instances from last year’s Sapphire Key Note is an example of that. It was a plain HANA Database fronted by a Java Web Application.

Now, within last year’s Sapphire and now, SAP released a version of SAP HANA that is free for development and testing purposes up to 32GBs of RAM called HANA Express.

For more details, you should navigate to the SAP HANA Express Homepage to get the full picture and the official view:

https://www.sap.com/developer/topics/sap-hana-express.html

Azure Quick-Start-Templates

Now, shortly before Sapphire some folks from Microsoft and SAP approached me for creating an Azure Marketplace Image for HANA Express. That’s something we’re working on, but it is not something that can be done in just a few days. Too short… but since HANA Express clearly addresses developers, I thought a good solution that can be implemented within a few days is an Azure Quick-Start-Template.

For those of you who are new to Azure: Azure Quick-Start-Templates are open source based Azure Resource Manager Templates and Deployment Scripts which can be used to quickly spin-up Solutions on Azure. Many of those are used as a learning resource, but some of them can definitely be used for dev/test scenarios or even as a starting point for production scenarios.

All of these are available under the following two links:

Now, the main point with these quick start templates is, that they’re automating most of the setup/provisioning procedure by using Scripts and Templates so you can get started, quickly.

SAP HANA Express Quick Start Template

I decided to build such a template for HANA Express and make it available as part of the Quick-Start-Templates. This Quick-Start works with SAP HANA Express 2.0 SPS1 and it also should work with other version, but I’ve only tested it with this one.

http://aka.ms/sap-hana-express-quickstart

There’s just one caveat: since SAP requires you to accept the EULA for SAP HANA Express, you first need to go to SAP’s HANA Express Home Page, register and download the SAP HANA Express Setup Images manually before being able to start using the Quick Start Template I’ve created. But from there on you can start using the template. The basic workings of the template are:

  1. First you register with SAP and download the HANA Express Setup Packge.
  2. Then you use the quick start template to upload the setup package into your private Azure Storage Account.
  3. From that moment forward you can use the Azure Resource Manager Template included in the template to deploy as many HANA Express Instances into your own subscription as you want.

Starting with Step 2, everything is automated with Scripts and Templates. That means, only the first step – downloading the setup packges and accepting the EULA at the SAP HANA Express Setup Homepage is something you need to do manually. Sure, a Marketplace Image would be more convenient, but we’ll work with SAP on that…

All the details are explained in the sections below in this blog post including how you can validate, that the installation really went well at the end of the entire process.

Important Note: Please don’t forget, that quick start templates are not something that is backed by Microsoft Support of any means. They are here to help you getting started on your own, they are full open source and maintained at a best effort basis!

Requirements

Before moving on, all you need on your local machine are the following assets/tools:

Register Download HANA Express Setup from SAP

Yes, that’s the first step. It needs to be done for two reasons: first, you need to accept SAP’s EULA and second SAP will inform you about important updates and service releases for HANA when registering at the registration page.

That said, the first thing you do is navigating to the SAP HANA Express Home Page to register with SAP and Accept the EULA:

HANA Express

HANA Express Download Manager Option

Next, you’ll need to use the SAP Download Manager to download the SAP HANA Express Setup packages. The Download Manager exists as native version for Linux or Windows or as a Cross-Platform Version for Java (distributed as JAR-package). I’ve used the JAR-package version, but it should not matter at all.

What matters is selecting the right type of setup packages. The quick-start-template I’ve built is tested for the Server-Version, only, without XA Advanced services. So you should select the following for download when using the SAP HANA Express Download Manager:

HANA Express Server Only Download Option

The downloaded setup package will appear as TAR-Archive in your local Downloads-folder (or wherever you downloaded it to). It should be called something along the lines of hxe.tgz.

Upload the HANA Express to your Azure Storage Account

For automating the setup procedure of SAP HANA Express inside of an Azure Virtual Machine, the HANA Express Setup packages need to be available for download through an automation script! To avoid uploading the setup package (~ 1.6GBs of data) with each new VM, the best approach is uploading that into an Azure storage account and using a shared access signature to feed the setup files into the provisioning script for Virtual Machines.

Now, performing those steps can be done manually by using a tool such as the Azure Storage Explorer. But I decided to automate the procedure using the Azure CLI 2.0.

Assuming that you have your SAP HANA Express Setup packages downloaded to the folder /mnt/c/temp/hxe.tgz, you can execute the following command:

# Login into your Azure Subscription using the Azure CLI 2.0
az login

# Create a resource group for the storage account
az group create --name "sampleresourcegroupname" --location "westeurope"

# Upload the HANA Express Setup Files to your Azure Storage Account
./prepare-hxe-setup-files.sh sampleresourcegroupname samplestorageaccountname samplecontainer westeurope /home/mydirectory/hxe.tgz

Now, as I mentioned before, the automated setup of HANA Express happens in a script that runs in the post-provisioning phase of the Azure VM. That means, this script needs to have access to those setup files for an automated download without user interaction. To enable that scenario, the prepare-hxe-setup-files.sh-Script of my Quick Start uploads the setup packages for HANA Express to an Azure Storage Account and generates a Shared Access Signature URL which allows to simply download the packages using that Signature as a means of authentication by using wget or any similar shell-tool.

The following Screen-Shot shows the output of the prepare-hxe-setup-files-sh-script. You should especially take note of the Shared Access Signature URL the script outputs at the end!

Output of prepare-hxe-setup-files.sh

Deploy an SAP HANA Express using the templates

With the Setup packages for SAP HANA Express uploaded to an Azure Storage Account and the Storage Shared Access Signature generated as mentioned above, you can deploy as many SAP HANA Express Virtual Machines as you need to.

Important Note: the script prepare-hxe-setup-files.sh above generates Shared Access Signatures that are valid for a year. That means, after a year, you need to run the script, again, to generate a new Shared Access Signature. Note that the script is smart in detecting, if the files have been uploaded, already, and if so it generates the signature for the existing blob instead of uploading, again!

When using the quick-start-template, you can either use the “Deploy-To-Azure”-Button presented on the landing page of the Quick Start or you fill out the parameters in the azuredeploy.parameters.json-file as shown below and deploy the template via PowerShell or the Azure CLI:

Parameters filled in the azuredeploy.parameters.json

After you’ve filled out the parameters – the screen shot above shows the minimum ones you need to fill out – you can move ahead and deploy the template using code similar to the following:

# Create a resource group for your HANA Express VM Resources
az group create --name "samplehanaexpressgroup" --location "westeurope"

# Deploy the template with the filled parameters file
az group deployment create --resource-group="samplehanaexpressgroup" \
                           --template-file="azuredeploy.json" \
                           --parameters="@azuredeploy.sample.parameters.json" \
                           --name="samplehanaexpress"

The output of that script should look similar to the following screen shot:
Parameters filled in the azuredeploy.parameters.json

Validating the Installation

So far so good, if the output looks similar to the screen shot above, then you should be all set! But you can of course validate your installation – in two ways: using regular HANA tools to see if your instance is responsive or look at the installation logs from the provisioning process.

For that, you need to understand some background. I am using Azure Custom Script Extensions for Linux to automatically execute the HANA Installation with all required pre-requisites during the post-VM provisioning phase of the Azure Virtual Machine. That is expressed in the Azure Resource Manager Template with the following code:

... REST OF THE TEMPLATE ...

    {
        "type": "extensions",
        "name": "hxeinstallextension",
        "apiVersion": "2016-04-30-preview",
        "location": "[resourceGroup().location]",
        "dependsOn": [
            "[concat('Microsoft.Compute/virtualMachines/', parameters('vmNamePrefix'))]"
        ],
        "properties": {
            "publisher": "Microsoft.Azure.Extensions",
            "type": "CustomScript",
            "typeHandlerVersion": "2.0",
            "autoUpgradeMinorVersion": true,
            "settings": {
                "fileUris": [
                    "[parameters('hxeInstallScriptUrl')]"
                ]
            },
            "protectedSettings": {
                "commandToExecute": "[concat('sudo ./', parameters('hxeInstallScriptName'), ' \"', parameters('hxeSetupFileUrl'), '\" \"', parameters('hxeMasterPwd'), '\" && exit 0')]"
            }
        }
    }

... REST OF THE TEMPLATE ...

This part of the template shows, that after the Virtual Machine Resource has been provisionied, it uses the Azure Virtual Machine Agent to run the script specified in the template. This script is downloaded from the Quick-Start-Templates GitHub repository, directly. So no further steps needed to enable this.

If you now want to validate, whether the installation script from SAP HANA Express ran, successfully, you first should review the deployment logs within the Azure Portal similar to what’s shown in the following screen shot:

Azure Portal Deployment Log

If you still need to see more, then you just need to SSH into the virtual machine created (refer to the DNS name specified in the azuredeploy.parameters.json as per the screen shots above) and output the content of the stdout and stderr files within the /var/lib/waagent/custom-script/download/0 directory similar to what’s shown in the following screen shot:

Virtual Machine Deployment Log

When you look at the output, you’ll quickly realize, how much this really accelerates you. The setup script automatically performs the following steps for you:

  • Install the needed Oracle JDK on your machine
  • Install required library packages using zypper
  • Download and Extract the HANA Express Setup Packages to the VM
  • Install HANA Express on the VM

Final Words

The post above went quite into the details of every step of how the quick-start works. But the essence can literally be done within about 30min. depending on how fast your Internet connection is:

  • Register for an Azure Subscription if you don’t have one, yet
  • Setup the Azure CLI 2.0 on your machine if not done so, yet
  • Register with SAP
  • Download the SAP HANA Express Setup Packages
  • Execute the script prepare-hxe-setup-files.sh and note the generated Shared Access Signature
  • Click the “Deploy-to-Azure Button” or update the parameters file and execute az group deployment create

All of the needed pre-requisites and of course SAP HANA Express itself gets set up for you and within about 30min. you have an instance of it running in Microsoft Azure. I hope you find this more valuable and helps you to accelerate the setup of dev/test environments with SAP HANA Express since you don’t need to walk through all of the needed setup steps, manually.

Microsoft Cognitive Services – Shopping Offer Comparisons with Bing Image Search v5

This week I got the chance for my first attempts with Cognitive services and image search – based on a request from my Global ISV partner:) While the services are actually easy to use for a developer, documentation is behind and Internet Search is highly miss-leading (since it mostly points to a previous Bing-search API that does not exist, anymore). That’s why I decided to blog about it and point people into the right direction.

The Case – Price Comparisons

The use case is simple and has been made available with our latest Bing-App for the iOS: perform price comparisons of products across multiple shops. But instead of an App, this blog-post is about doing this from within any application with right usage of the new Bing Search APIs in their version 5.0.

Cognitive Services

At the annual //build 2016 conference, Microsoft announced its new Cognitive services. These are services exposed through simple-to-use APIs for doing all sorts of intelligent stuff based on extensive data and machine learning algorithms. Face recognition, Speech Recognition and the likes are some of the more advanced APIs.

What many people don’t know is, that Bing APIs are also part of the Cognitive Services, now. That’s because a lot of the intelligence behind Cognitive Services is powered by Bing services incl. their intelligence and machine learning components. So don’t let yourself miss-lead by Internet-search results pointing to any kind of previous services.

That means don’t search the Internet, just navigate to Cognitive services right away and dig into the documentation.

Walking through the Use Case

Ok, let’s walk through the Use Case in a schematic way by looking at the APIs and their responses. That will give you a good view on how it actually works.

  1. You need a Microsoft Account, so if you don’t have one, sign up for one first.
  2. If you have a Microsoft Account, the first thing you need is signing up for the Cognitive Services Preview. For that purpose just navigate to the Cognitive Services Subscriptions Page.
  3. Once you have signed up for the subscriptions, you get application keys for each of the different types of APIs as shown in my screen-shot below. You need to “Show” and “Copy” the key for Bing Search to implement the case I’ve described above:
    Subscription Keys
  4. Once you have copied the subscription key for Bing Search, you can use Bing Image Search to look for images products you want to get price comparisons for.
    • I know, that sounds a bit confusing. But let’s assume you want to look for offerings of a sports watch such as Garmin Forerunner 225 (which I am currently interested in:)).
    • To get to shopping offers through Bing or Bing APIs, you’d do an image search for “Garmin Forerunner 225” and then the “new Bing” and “new Bing APIs” will give you that additional details of shopping offers.
    • Let me show you how that works with the API, but note that the same thing works with Bing Search in the browser for end-users.
  5. For testing the APIs I use the available Test User Interfaces that you can use for learning the APIs without writing code right away.
  6. Now, let’s dig into a few requests. If you want to get offers for e.g. a “Garmin Forerunner” watch, first you need to find images for that watch.
    GET https://bingapis.azure-api.net/api/v5/images/search?q=garmin forerunner 225&count=10&offset=0&mkt=en-us&safeSearch=Moderate HTTP/1.1
    Host: bingapis.azure-api.net
    Ocp-Apim-Subscription-Key: <<your API Key taken from the previous screen shot above>>
    
  7. Next you need to examine the results of the request above. The interesting pieces of the JSON response are the imageInsightsToken and the insightsSourcesSummary elements as highlighted below:
    Insights Details Highlighted
  8. These two attributes are used for the following purposes:
    • imageInsightsToken is used in the next subsequent request to get further details about the image kept and managed by Bing.
    • insightsSourcesSummary is something you can use to asses if it’s worth querying further insights for the image. E.g. in my case I wanted to get as many shop-offers for the Garmin as possible. So if I’d write this in a program I
      would search the top-most search results (first page of JSON elemetns I got from the previous API request) and pick those with the highest shoppingSourcesCount value as a simple strategy.
  9. So, let’s use the imageInsightsToken to get further details from the image. the following code shows the next request we’re executing with the token passed in as an additional parameter. Also note the use of the modulesRequested parameter
    which I use to specify which kinds of additional details I’d love to get from Bing for that image. That said, there are different modules providing additional information beyond the shopping sources.

    GET https://bingapis.azure-api.net/api/v5/images/search?
      q=garmin forerunner 225&count=10&offset=0
      &mkt=en-us
      &safeSearch=Moderate
      &modulesRequested=shoppingSources
      &insightstoken=ccid_eDfbozgF*mid_7F932B775F08CAD51E0BC3609B3ABA1B7AB73856*simid_608041772346312873 
      HTTP/1.1
    Host: bingapis.azure-api.net
    Ocp-Apim-Subscription-Key: <<your API Key taken from the previous screen shot above>>
    
  10. Finally we get the results we want to have from this request. We see a list of sources which are offering the product for a given price and the basic stock-information which Bing extracted from the web pages of that source for my Garmin Forerunner 225 search.
    Shopping Sources Results

I really find this kind-of cool. I found it really interesting to work with my ISV partner through that and given that documentation is not always most up2date on those new services I thought it’s worth blogging about it:).

Further resources for reading: Here are a few helpful links with further details and information. Fortunately from the time I started writing this post until I got it published, the team has updated additional documentation on MSDN!

Let me know your thoughts, best via http://twitter.com/mszcool!

Azure VMs – SQL Server AlwaysOn Setup across multiple Data Centers fully automated (Classic Service Management)

Last December I started working with two of my peers, Max Knor and Igor Pagliai, with a partner in Madrid on implementing a Cross-Data Center SQL Server AlwaysOn availability group setup for a financial services solution which is supposed to be provided to 1000s of banks across the world running in Azure. Igor posted about our setup experience which we partially automated with Azure PowerShell and Windows PowerShell – see here.

At the moment the partner’s software still requires SQL Server in VMs as opposed to Azure SQL Databases because of some legacy functions they use from full SQL Server – therefore this decision.

One of the bold goals was to fully enable the partner and their customers to embrace DevOps and continuous delivery across multiple environments. For this purpose we wanted to FULLY AUTOMATE the setup of their application together with an entire cross-data-center SQL Server AlwaysOn environment as outlined in the following picture:

In December we did a one-week hackfest to start these efforts. We successfully did setup the environment, but partially automated, only. Over the past weeks we went through the final effort to fully automate the process. I’ve published the result on my github repository here:

Deployment Scripts Sample Published on my GitHub Repository

Note: Not Azure Resource Groups, yet

Since Azure Resource Manager v2 which would allow us to dramatically improve the performance and reduce the complexity of the basic Azure VM environment setup is still in Preview, we were forced to use traditional Azure Service Management.

But about 50%-60% of the efforts we have done are re-usable given the way we built up the scripts. E.g. all the database setup and custom service account setup which is primarily built on-top of Azure Custom Script VM Extensions can be re-used after the basic VM setup is completed. We are planning to create a next version of the scripts that does the fundamental setup using Azure Resource Groups because we clearly see the advantages.

Basic Architecture of the Scripts

Essentially the scripts are structured into the following main parts which you would need to touch if you want to leverage them or understand them for learning purposes as shown below:

  • Prep-ProvisionMachine.ps1 (prepare deployment machine)
    A basic script you should execute on a machine before starting first automated deployments. It installs certificates for encrypting passwords used as parameters to Custom Script VM Extensions as well as copying the basic PowerShell modules into the local PowerShell module directories so they can be found.
  • Main-ProvisionConfig.psd1 (primary configuration)
    A nice little trick by Max which is nice to provide at least some sort of declarative configuration was to build a separate script file that creates an object-tree with all the configuration data typically used for building up the cluster. It contains cluster configuration settings, node configuration settings and default subscription selection data.
  • Main-ProvisionCrossRegionAlwaysOn.ps1 (main script for automation)
    This is the main deployment script. It performs all the actions to setup the entire cross-region cluster including the following setups:
    • Setup your subscription if requested
    • Setup storage accounts if they do not exist, yet
    • Upload scripts required for setup inside of the VMs to storage
    • Setup cloud services if requested
    • Create Virtual Networks in both regions (Primary/Secondary)
    • Connect the Virtual Networks by creating VPN Gateways
    • Set the primary AD Forest VM and the Forest inside of the VM
    • Setup secondary AD DC VMs including installing AD
    • Provision SQL Server VMs
    • Setup the Internal Load Balancer for the AlwaysOn Listener
    • Configure all SQL VMs to have AlwaysOn enabled
    • Configure the Primary AlwaysOn node with the initial database setup
    • Join secondary AlwaysOn nodes and restore databases for sync
    • Configure a file-share based witness in the cluster
  • VmSetupScripts Folder
    This is essentially a folder with a series of PowerShell scripts that do perform single installation/configuration steps inside of the Virtual Machines. They are downloaded with a Custom Script VM Extension into the Virtual Machines and executed through VM Extensions, as well.

Executing the Script and Looking at the Results

Before executing the main command make sure to execute .\Prep-ProvisionMachine.ps1 to setup certificates or import the default certificate which I provide as part of the sample. If you plan to seriously use those scripts, please create your own certificate. Prep-ProvisionMachine.ps1 provides you with that capability assuming you have makecert.exe somewhere on your machines installed (please check Util-CertsPasswords for the paths in which I look for makecert.exe).

# To install a new certificate
.\Prep-ProvisionMachine.ps1

# To install a new certificate (overwriting existing ones with same Subject Names)
.\Prep-ProvisionMachine.ps1 -overwriteExistingCerts

# Or to install the sample certificate I deliver as part of the sample:
.\Prep-ProvisionMachine.ps1 -importDefaultCertificate

Then everything should be fine to execute the main script. If you don’t specify the certificate-related parameters as shown below I assume you use my sample default certificate I include in the repository to encrypt secrets pushed into VM Custom Script Extensions.

# Enter the Domain Admin Credentials
$domainCreds = Get-Credential

# Perform the main provisioning

.\Main-ProvisionCrossRegionAlwaysOn.ps1 -SetupNetwork -SetupADDCForest -SetupSecondaryADDCs -SetupSQLVMs -SetupSQLAG -UploadSetupScripts -ServiceName "mszsqlagustest" -StorageAccountNamePrimaryRegion "mszsqlagusprim" -StorageAccountNameSecondaryRegion "mszsqlagussec" -RegionPrimary "East US" -RegionSecondary "East US 2" -DomainAdminCreds $domainCreds -DomainName "msztest.local" -DomainNameShort "msztest" -Verbose

After executing a main script command such as the following, you will get 5 VMs in the primary region and 2 VMs in the secondary region acting as a manual failover. 

The following image shows several aspects in action such as the failover cluster resources which are part of the AlwaysOn availability group as well as SQL Server Management Studio accessing the AlwaysOn Availability Group Listener as well as SQL Nodes, directly. Click on the image to enlarge it and see all details.

Please note that the failover in the secondary region needs to happen MANUALLY by executing either a planned manual failover or a forced manual failover as documented on MSDN. Failover in the primary region (from the first to the second SQL Server) is configured to happen automatically.

In addition on Azure it means to take the IP cluster resource for the secondary region online which by default is offline in the cluster setup as you can see on the previous image.

Customizing the Parts you Should Customize

As you can see in the image above, the script creates sample databases which it sets up for the AlwaysOn Availability Group to be synchronized across two nodes in the main. This happens based on *.sql scripts you can add to your configuration. To customize the SQL Scripts and Databases affected, you need to perform the following steps:

  • Create *.sql scripts with T-SQL code that creates the databases you want to create as part of your AlwaysOn Availability Group.
  • Copy the *.sql Files into the VmSetupScripts directory BEFORE starting the execution of the main script. That leads to have them included into the package that gets pushed to the SQL Server VMs
  • Open up the main configuration file and customize the database list based on the databases created with your SQL scripts as well as the list of SQL Scripts that should be pushed into osql.exe/sqlcmd.exe as part of the setup process for creating the databases.
  • Also don’t forget to customize the subscription name if you plan to not override it through the script-parameters (as it happens with the example above).

The following image shows those configuration settings highlighted (in our newly released Visual Studio Code editor which also has basic support for PowerShell):


Fundamental Challenges

The main script can primarily be seen as a PowerShell workflow (we didn’t have the time to really implement it as a Workflow, but that would be a logical next step after applying Azure Resource Groups).

It creates one set of Azure VMs after another and joins them to the virtual networks it has created before. It then executes scripts on the Virtual Machines locally which are doing the setup by using Azure VM Custom Script Extensions. Although custom script extensions are cool, you have two main challenges with them for which the overall package I published provides re-usable solutions:

  • Passing “Secrets” as Parameters to VM Custom Script Extensions such as passwords or storage account keys in a more secure way as opposed to clear-text.
  • Running Scripts under a Domain User Account as part of Custom Script Extensions that require full process level access to the target VMs and Domains (which means PowerShell Remoting does not work in most cases even with CredSSP enabled … such as for Cluster setups).

For these two purposes the overall script package ships with some additional PowerShell Modules I have written, e.g. based on a blog-post from my colleague Haishi Bai here.

Running Azure VM Custom Script Extensions under a different User

Util-PowerShellRunAs.psm1 includes a function called Invoke-PoSHRunAs which allows you to run a target script with its parameters under a different user account as part of a custom script VM Extension. A basic invocation of that script looks as follows:

$scriptName = [System.IO.Path]::Combine($scriptsBaseDirectory, "Sql-Basic01-SqlBasic.ps1") 
Write-Host "Calling into $scriptName"
try {
    $arguments = "-domainNameShort $domainNameShort " + `
                 "-domainNameLong $domainNameLong " +  `
                 "-domainAdminUser $usrDom " +  `
                 "-dataDriveLetter $dataDriveLetter " +  `
                 "-dataDirectoryName $dataDirectoryName " +  `
                 "-logDirectoryName $logDirectoryName " +  `
                 "-backupDirectoryName $backupDirectoryName " 
    Invoke-PoSHRunAs -FileName $scriptName -Arguments $arguments -Credential $credsLocal -Verbose:($IsVerbosePresent) -LogPath ".\LogFiles" -NeedsToRunAsProcess
} catch {
    Write-Error $_.Exception.Message
    Write-Error $_.Exception.ItemName
    Write-Error ("Failed executing script " + $scriptName + "! Stopping Execution!")
    Exit
}

This function allows you to either run through PowerShell remoting or in a separate process. Many setup steps of the environment we setup do actually not work through PowerShell remoting because they rely on impersonation/delegation or do PowerShell Remoting on their own which imposes several limitations.

Therefore the second option this script provides is executing as a full-blown process. Since Custom Script Extensions to run as local system, it is nevertheless not as simple as just doing a Start-Process with credentials being passed in (or a System.Diagnostics.Process.Start() with different credentials). Local System does not have those permissions, unfortunately. So the work-around is to use the Windows Task Scheduler. For such cases the function performs the following actions:

  • Schedule a task in the Windows Task Scheduler with the credentials needed to run the process as.
  • Manually start the task using PowerShell cmdLets
    • (Start-ScheduledTask -TaskName $taskName)
  • Wait for the task to be finished from running
  • Look at the exit code
  • Throw an Exception if the exit code is non-zero, otherwise assume success
  • Delete the task again from the task scheduler

This “work-around” helped us to completely execute the entire setup steps successfully. We were also discussing with the engineers building the SQL AlwaysOn single-data-center Azure Resource Group template that is available for single-data-center deployments in the new Azure Portal, today. They are indeed doing the same thing, details are just a bit different.

Encrypting Secrets Passed to Custom Script VM Extensions

Sometimes we were just required to pass secret information to custom script extensions such as storage account keys. Since Azure VM Custom Script Extensions are logged very verbose, it would be a piece of cake to get to that secret information by doing a Get-AzureVM and looking at the ResourceExtensionStatusList member which contains the status and detailed call information for all VM Extensions.

Therefore we wanted to encrypt secrets as they are passed to Azure VM Extensions. The basic (yet not perfect) approach works based on some guidance from a blog post from Haishi Bai as mentioned earlier.

I’ve essentially written another PowerShell module (Util-CertsPasswords) which can perform the following actions:

  • Create a self-signed certificate as per guidance on MSDN for Azure.
  • Encrypt Passwords using such a certificate and return a base64-encoded, encrypted version.
  • Decrypt Passwords using such a certificate and return the clear-text password.

In our overall workflow all secrets including passwords and storage account keys which are passed to VM Custom Script Extensions as parameters are passed as encrypted values using this module.

Using Azure CmdLets we make sure that the certificates are published with the VM as part of our main provisioning script as per Michael Washams guidance from the Azure Product group.

Every script that gets executed as part of a custom VM Script Extension receives an encrypted password and uses the module I’ve written to decrypt it and use it for the remaining script such as follows:

#
# Import the module that allows running PowerShell scripts easily as different user
#
Import-Module .\Util-PowerShellRunAs.psm1 -Force
Import-Module .\Util-CertsPasswords.psm1 -Force

#
# Decrypt encrypted passwords using the passed certificate
#
Write-Verbose "Decrypting Password with Password Utility Module..."
$localAdminPwd = Get-DecryptedPassword -certName $certNamePwdEnc -encryptedBase64Password $localAdminPwdEnc 
$domainAdminPwd = Get-DecryptedPassword -certName $certNamePwdEnc -encryptedBase64Password $domainAdminPwdEnc 
Write-Verbose "Successfully decrypted VM Extension passed password"

The main provisioning script encrypts the passwords and secrets using that very same module before being passed into VM Custom Script Extensions as follows:

$vmExtParamStorageAccountKeyEnc = `
Get-EncryptedPassword -certName $certNameForPwdEncryption `             -passwordToEncrypt ($StorageAccountPrimaryRegionKey.Primary)

That way we at least make sure that no un-encrypted secret is visible in the Azure VM Custom Script Extension logs that can easily be retrieved with the Azure Service Management API PowerShell CmdLets.

Final Words and More…

As I said, there are lots of other re-usable parts in the package I’ve just published on my Github Repository which even can be used to apply further setup and configuration steps on VM environments which have entirely been provisioned with Azure Resource Groups and Azure Resource Manager. A few examples:

  • Execute additional Custom Script VM Extensions on running VMs.
  • Wait for Custom Script VM Extensions to complete on running VMs.
  • A ready-to-use PowerShell function that makes it easier to Remote PowerShell into provisioned VMs.

We also make use of an AzureNetworking PowerShell module published on the Technet Gallery. But note that we also made some bug-fixes in that module (such as dealing with “totally empty VNET configuration XML files”).

Generally the experience of building these ~2500 lines of PowerShell code was super-hard but a great learning experience. I am really keen to publish the follow-up post on this that demonstrates how much easier Azure Resource Group templates to make such a complex setup.

Also I do hope that we will have such a multi-data-center template in the default gallery soon since it is highly valuable for all partners and customers that do need to provide high-availability across multiple data centers using SQL Server Virtual Machines. In the meantime we will try to provide a sample based on this work above as soon as we can have time/resources for implementation.

Finally – thanks to Max Knor and Igor Pagliai – without their help we would not have achieved these goals at this level of completeness!

Azure Websites – Updating my WordPress Blog and PHP runtime in a save way with Websites Backup

As some of my readers know my blog runs on Microsoft Azure Websites. Since I launched this new version back in summer 2014 I literally had no maintenance tasks to do. But about half a year ago I started getting notifications that PHP 5.3 will be deprecated on Azure Websites. Since my blog up until know ran WordPress 3.x with PHP 5.3 I finally had to do something to ensure a smooth transition.

I know the notification below is in German, but essentially it tells me that the Azure team is deprecating PHP 5.3 and they would move me to 5.4 automatically if I don’t do anything until January 31st.

Now, I definitely did not want to have any bad surprises. So I wanted to do the upgrade by myself, test it and also upgrade to the latest version of WordPress as part of that operation. Of course I wanted to have all of this in a safe way, with an option to roll-back if something does not work.

Azure Websites Backup

What I wanted is doing a backup using the out-of-the-box service provided as part of Azure Websites – Backup. Since you can only make use of that function in STANDARD-scale mode, the first thing I did, was switching to STANDARD (temporarily for the time of this activity).

With that I was able to create a backup of my website into my Azure storage account. The cool part of it is that backup includes everything needed: the site and its file contents as well as the linked ClearDB database.

Since I wanted to switch back to Shared-mode after this activity, again, to save costs, I did not create automated and scheduled backups. But for more serious workloads I’d definitely suggest to leverage this fairly cool feature, as well.

Of course I wanted to try if the backup really worked. So as a next step I thought I try to restore the site in a newly created, empty website. Again I created the site in standard mode so that I was able to leverage the functionality. Then I went to the Backup-tab and tried the restore option. With the restore option you can either restore backups made for this web site or explicitly select a backup file from your storage account. Since I wanted to restore into a new website I wanted to restore from the file, directly.

The restore did only work partially – it did run in an error because it also tried to restore the custom domain associated with the website which was still associated with my production website. But at least it was able to read and parse the backup file. That was enough for me to trust that in a worst case a restore would work:)

Next I did decide to upgrade the WordPress version before I upgrade the PHP runtime. I did not want to upgrade the runtime and then have eventually issues with my older WordPress version running on the newer PHP runtime. Upgrading WordPress could not be simpler. Just sign in into the control panel and it will offer you to upgrade, anyway. That’s then exactly what I did.

And what’s really cool is that just after 2 or 3 min. the browser opened back up with the following screen – bumm, my blog was upgraded to WordPress 4.1. Isn’t that awesome. That’s exactly how I’d expect such things to work:

Now the remaining upgrade-step was to move away from PHP 5.3 as it will be deprecated – which was the original motivation for me to start this whole effort.

Finally since I pay for this with my private check I wanted to run on the more cost-effective instance-tier, again. So after everything worked out well I switched back to “Shared”-mode (as I have the custom domain blog.mszcool.com associated with the website).

That’s it. With that and with a short period of time running in a more expensive instance mode the blog is backed-up and now runs on the latest PHP runtime as well as WordPress version. It’s impressive how well Azure Websites, PHP and WordPress plays together. Cool stuff…

Hope that how-to was interesting for you, as well…

Azure Batch – (Highly Scalable) Batch Processing with Microsoft Azure (and a Successor to GeRes2)

Batch processing is something nearly everyone I have been working with is doing in some or the other way on Microsoft Azure. Last year some colleagues and I did work with several global partners that had this requirement. Therefore since nothing at the scale of Azure Batch was available, yet, we created GeRes2 as an open source project. GeRes2 covers batch processing in a simple, pragmatic yet scalable way (at full scale of Web/Worker roles as opposed to the limited scale of WebJobs SDK in WebSites). But now, the times when you need GeRes2 are over. Instead you definitely should consider Azure Batch, directly.

What is Azure Batch?

Azure Batch is a completely managed service, you could see it as “batch processing as a service”.

It can be used for simple batch processing up to High-Performance-Compute (HPC) types of workloads. As a managed service, Azure Batch is fully operated by Microsoft and made accessible through an HTTP REST API. While for simpler requirements you might want to use WebJobs SDK, if you really need to scale beyond the options provided by WebSites and do not want to manage infrastructure (e.g. such as it is the case with Windows Server HPC Pack), Azure Batch is your best friend.

A sample on Azure Batch to help you Getting Started!

As a first introduction and to help you get started, I thought to publish a sample inspired by a session from one of the Azure Batch product managers, Mark Scurrell, at TechEd EMEA 2014: an OCR image recognition based on the Tesseract OCR Engine. The whole sample is available on my github.com repository:

https://github.com/mszcool/AzureBatchTesseractSample

The following sketch outlines the flow of the sample which I’ve built to help you get started. Note that I also include a PowerShell Script that sets up the environment in your Azure Subscription (please setup Azure PowerShell correctly, before). It creates all required Azure service accounts (storage, batch), updates app.config configurations, builds the sample and finally uploads sample data as it is needed for testing the scenario right away.

As you can see, the overall solution consists of a client that creates compute pools, submits jobs to that compute pool that use Tesseract for an OCR recognition on-top of PNG-based images stored in Azure Blob Storage and then makes the results available in Azure Blob storage, as well. Although there are different alternative ways with Azure Batch, for this first sample I also wrote a little console application that downloads the source-images from Azure Blob to the task virtual machines (TVM), processes them using tesseract.exe and then uploads the results back to BLOB-storage.

Understanding the fundamental Azure Batch Concepts

Before you can get started, you should understand the fundamental concepts of Azure Batch. Let’s get started with the following terms:

  • Azure Batch Account
    An account is a management unit used to group batch services and batch apps together in a single unit with security access keys.
  • Azure Batch REST API
    For each account, Batch is made available through an HTTP REST API. For .NET developers the team ships an SDK, already. Other languages will follow (or are available after I’ve published this article, already).
  • Azure Batch Apps
    Batch can be used in two flavors, through the low-level REST-API which is more complex but provides you with a bigger set of options and control or through a more managed experience with a management portal and a light-weight API called Batch Apps.
  • Compute Pools
    These are groups of compute nodes used for executing work. Pools can contain many compute nodes and can be configured with auto scaling rules so they add/remove resources based on the load on a pool.
  • Task Virtual Machines (TVM)
    A TVM is a single compute node which is part of a compute pool. Essentially behind the scenes TVMs are worker roles since Azure Batch by itself is implemented with Web/Worker roles. For you as developer, just think of them as scalable, stateless virtual machines.
  • Work Items
    Work Items are used to describe classes of units of works (aka jobs) and configure scheduling properties of jobs (e.g. execute once, regular time-controlled execution etc.). It also allows specifying the compute pools on which jobs of a work item are executed.
  • Jobs
    A job is a unit of work. It describes a set of concrete execution items called tasks. Each task gets executed on a TVM part of a pool that is tied to the work item the job belongs to.
  • Tasks
    A task is a single execution step of a job. Essentially a task is an executable that you need to provide and specify as part of the scheduling process that will be executed on TVMs.

Creating an Azure Batch Account with Azure PowerShell

The first thing you need to do is creating an Azure Batch Account. Note that at the time of writing this article, Batch was still in Preview. Therefore make sure you activate it for your subscription, first. The PowerShell script I provide for setting up the sample does that for you. Note that Azure Batch can only be managed as part of the new Azure Resource Manager inside of a Resource Group. Therefore in an Azure PowerShell CmdLet you first need to switch the Azure PowerShell mode to AzureResourceManager:

   1: Switch-AzureMode -Name AzureResourceManager

   2: New-AzureResourceGroup -Name $batchSampleResourceGroupName -Location $regionName

   3: New-AzureBatchAccount -AccountName $azureBatchAccountName `

   4:                       -ResourceGroupName $batchSampleResourceGroupName `

   5:                       -Location $regionName

Below a screen-shot of the script I wrote to setup things in action. That screen shot shows you, how-to call the script to setup the environment required for the demo:

Azure Batch NuGet Package

Now that we have an account, you can start developing. In Visual Studio you need to use a NuGet package called Azure Batch. Please make sure to use the core Azure Batch package and not the Batch Apps package in case you want to use the full-blown, low-level Batch APIs.

Creating Compute Pools using the API

The first thing you need to do is creating some compute pools with task virtual machines. The .NET SDK wraps the REST API for such purposes in a very convenient way. After having set-up BatchCredentials and a BatchClient you can open various managers that encapsulate certain management operations, e.g. a Compute Pool Manager as shown below.

   1: using (var pm = batchClient.OpenPoolManager())

   2: {

   3:     var pools = pm.ListPools().ToList();

   4:     var poolExists = (pools.Select(p => p.Name).Contains(PoolName));

   5:     if (!poolExists)

   6:     {

   7:         var newPool = pm.CreatePool

   8:             (PoolName, "3", "small", 5

   9:             );

  10:         newPool.StartTask = new StartTask

  11:         {

  12:             ResourceFiles = binaryResourceFiles,

  13:             CommandLine = "cmd /c CopyFiles.cmd",

  14:             WaitForSuccess = true

  15:         };

  16:         newPool.CommitAsync().Wait();

  17:     }

  18: }

One interesting aspect of the compute pool creation above is the definition of a startup task. This startup task in case of the sample defines a list of files that should be downloaded from Azure BLOB storage to the TVMs as part of the bootstrapping process. That list of files is prepared earlier in the code as shown below:

   1: var binaryResourceFiles = new List<IResourceFile>();

   2: Console.WriteLine("Get list of 'resource files' required for execution from BLOB storage...");

   3: foreach (var resFile in blobTesseractContainer.ListBlobs(useFlatBlobListing: true))

   4: {

   5:     var sharedAccessSig = CreateSharedAccessSignature(blobTesseractContainer, resFile);

   6:     var fullUriString = resFile.Uri.ToString();

   7:     var relativeUriString = fullUriString.Replace(blobTesseractContainer.Uri + "/", "");

   8:  

   9:     Console.WriteLine("- {0} ", relativeUriString);

  10:  

  11:     binaryResourceFiles.Add(

  12:         new ResourceFile

  13:             (

  14:             fullUriString + sharedAccessSig,

  15:             relativeUriString.Replace("/", @"\")

  16:             )

  17:         );

  18: }

Note that even if the blob container is publicly accessible I had to use shared access signatures to allow Azure Batch downloading those files. These files will be placed in a directory dedicated to the startup task (meaning actual tasks executed later don’t have access to this directory). Therefore one of the files downloaded is a batch-script which is then executed as part of the startup procedure. This batch script copies the file from the startup task working directory to the shared-directory to which all tasks do have access to as shown below:

   1: @echo off

   2: echo "List Files for diagnostics..."

   3: echo %WATASK_TVM_ROOT_DIR%

   4: dir .\ /s

   5: echo.

   6: echo Moving BatchTesseractWrapper files to shared task directory

   7: robocopy /MIR .\ %WATASK_TVM_ROOT_DIR%\shared

   8: if "%errorlevel%" LEQ "4" (

   9:    SET errorlevel=0

  10: )

One little hint here: Azure Batch considers a task (incl. the startup task) to be successful when it returns an exit code of 0. Since I am using robocopy.exe in my script I need to consider, that robocopy has several non-0 success exit codes. Therefore I map those to the exit code 0.

After my startup task finally completed, I do have all the tesseract-binaries as well as the tesseract wrapper executable I’ve written (which downloads files from BLOB, processes them with tesseract.exe and then uploads the result back to BLOB) in the shared task working directory.

Note that all of these working directories for tasks on TVMs are placed under a sub-directory of the task root directory exposed through the WATASK_TVM_ROOT_DIR environment variable. These directories are:

  • %WATASK_TVM_ROOT_DIR%\shared
    is a shared directory to which all tasks executed on the TVM do have read and execute permissions.
  • %WATASK_TVM_ROOT_DIR%\startup
    is a directory dedicated to the startup tasks specified as part of the compute pool creation. No other tasks do have any access to this directory.
  • %WATASK_TVM_ROOT_DIR%\tasks\<workitemname>\<jobname>\<taskname>
    is the directory dedicated for task execution whereas every work-item, job and task gets his own directory in this working directory.

A really cool tool for confirming that everything in our startup tasks did work is the Azure Batch explorer. It is also cool for exploring the true, detailed directory structure which I’ve outlined above. I might write about this tool in a subsequent blog post and will continue focusing on the code in this one.

Scheduling Jobs for Execution

Once the compute pool runs and all the TVMs are prepared with the tesseract binaries through the startup task, we can start scheduling jobs. This happens with the WorkItem manager and by creating a WorkItem with a Job and adding tasks to that job. In my sample I create one task for each file I want to OCR-recognize which I’ve previously uploaded to BLOB storage.

Note:the PowerShell setup script I do provide uploads some sample data I’ve included in the git-repository so that you can get started right away.

   1: using (var wiMgr = batchClient.OpenWorkItemManager())

   2: {

   3:     var workItemName = string.Format("ocr-{0}", DateTime.UtcNow.Ticks);

   4:     var ocrWorkItem = wiMgr.CreateWorkItem(workItemName);

   5:     ocrWorkItem.JobExecutionEnvironment =

   6:         new JobExecutionEnvironment

   7:         {

   8:             PoolName = PoolName

   9:         };

  10:     ocrWorkItem.CommitAsync().Wait();

  11:  

  12:     var taskNr = 0;

  13:     const string defaultJobName = "job-0000000001";

  14:     var job = wiMgr.GetJob(workItemName, defaultJobName);

  15:  

  16:     foreach (var ocrFile in filesToProcess)

  17:     {

  18:         var taskName = string.Format("task_no_{0}", taskNr++);

  19:         var taskCmd =

  20:             string.Format(

  21:                 "cmd /c %WATASK_TVM_ROOT_DIR%\\shared\\BatchTesseractWrapper.exe \"{0}\" \"{1}\"",

  22:                 ocrFile.BlobSource,

  23:                 Path.GetFileNameWithoutExtension(ocrFile.FilePath));

  24:  

  25:         ICloudTask cloudTask = new CloudTask(taskName, taskCmd);

  26:  

  27:         job.AddTask(cloudTask);

  28:     }

  29:     job.Commit();

The submission here happens with the WorkItemManager. Every WorkItem gets a default job (job-0000000001) which can be used immediately for adding tasks to be executed. This is exactly what is done in the code snippet above.

Note: since the job-creation might not be completed, yet, the call to wiMgr.GetJob() should be wrapped with some retry-logic.

For each file I’ve stored in BLOB storage I do create a task. That task ultimately just executes a command shell with the executable that downloads the source image to the TVM, calls tesseract.exe to OCR recognize the image and uploads the resulting text-file back to Azure Blob storage. Since this should be straight-forward for Azure-experienced developers, I leave it to you to look at the code on my github-repository.

Waiting for the tasks to complete and get results

This final step is optional, but in case your program needs to wait for the tasks to complete before doing something else, the code below is helpful.

   1: var toolBox = batchClient.OpenToolbox();

   2: var stateMonitor = toolBox.CreateTaskStateMonitor();

   3: var runningTasks = wiMgr.ListTasks(workItemName, defaultJobName);

   4: stateMonitor.WaitAll(runningTasks, TaskState.Completed, TimeSpan.FromMinutes(10));

   5:  

   6: var tasksFinalResult = wiMgr.ListTasks(workItemName, defaultJobName);

   7: foreach (var t in tasksFinalResult)

   8: {

   9:     Console.WriteLine("- Task {0}: {1}, exit code {2}", t.Name, t.State,

  10:         t.ExecutionInformation.ExitCode);

  11: }

The Azure Batch SDK for .NET comes with a set of handy utility classes to allow you doing exactly that. Through the TaskStateMonitor utility class the code above waits till all tasks have completed their work, loads the most recent data from the Azure Batch REST API and displays the results.

Important here is that every task, whether succeeded or failed, will be in the completed-state except you terminated it before it completed through the REST API. It’s up to you to check if the task was successful by looking at the exit code of the executable for your task.

The actual results of the tasks should be visible now in your Azure BLOB storage account in a container called ocr-results as shown below.

Final Words

This article and the sample I published on GitHub should help you to understand the basic principles and how-to get started with Azure Batch. As one of the creators of GeRes2, which is also doing scalable execution of jobs on Worker Roles, I really experienced how much effort it is to build something like this on your own.

As a managed service, Azure Batch really does everything for you. You can focus on creating your tasks as well as scheduling your jobs/tasks instead of dealing with all the plumbing infrastructure (e.g. distributing work across compute nodes, compute node management, logging, auto scaling, task binary deployments etc.).

The only thing that you might miss as compared to GeRes2 is notifications via Service Bus and SignalR. But to be honest, when using GeRes2, those notifications did only work up to a maximum of 100 compute nodes since we did not scale out the message queues on Service Bus which we used for the notifications. Of course we could have done that, but based on the needs of those partners who used notifications with GeRes2 it was just not necessary to support > 100 instances while those which used GeRes2 with more than 100 instances did really not use notifications. And there’s still the option of polling the tasks’ status through the APIs made available by Azure Batch. But eventually we’ll work on something to show you, how-to get this remaining piece to Azure Batch, as well.

Azure Batch is really the premium service when it comes to highly scalable batch processing on Azure. It is easy to use and it saves you from a whole lot of plumbing work that needs to be done if you do batch processing manually. In a future blog-post I plan to write about my personal opinion and view on comparing Azure Batch to other batch processing options available on Azure such as WebJobs SDK or running HPC Pack in Windows Server VMs in Azure.

In the meantime, feel free to share some feedback or ask questions via Twitter (http://twitter.com/mszcool).

AllJoyn IoT Peer-To-Peer Protocol & Framework – Making it Work with Visual Studio 2013

Our global team is running several industry subject matter working groups with different focus areas. One of them is targeted to Internet-Of-Things (IoT).

The charter of this working group is the exploration of IoT standards, protocols and frameworks across various industries with the goal of developing recommendations, reference architectures and integration points to and on the Microsoft platform as well as feeding IoT-related product groups of Microsoft with input for technologies, services and features we’re building as a company.

Together with peers from this working group including @joshholmes, @tomconte, @mszcool, @reichenseer, @jmspring, @timpark, @irjudson, @ankoduizer, @rachelyehe and daniele-colonna we explored AllJoyn as a technology.

Background on AllJoyn and our activities

I had the chance to work with this working group on exploring AllJoyn as a technology – especially because Microsoft joined the AllSeen alliance (www.allseenalliance.org). AllSeen has the sole target of making AllJoyn a defacto-standard for device-2-device communications in a mash-up oriented way across platforms and operating systems.

In a nutshell, AllJoyn is a peer-2-peer protocol framework that allows all sorts of device-2-device communication in local networks. It is built on-top of DBUS and TCP/UDP and includes advanced services such as on-boarding of devices to a network, device/service discovery, communication and notifications.

If you want to learn more about the fundamentals, Rachel has published a great summary on AllJoyn on her blog, which explains the capabilities and services provided by AllJoyn and the architectural principles, in general!

Why is this relevant?

Well, I think AllJoyn does have the potential revolutionize, how devices find and talk to each other in a standardized, cross-platform way in local networks and across wide networks (through a gateway). If done right and assuming, that a broad ecosystem of devices will adopt AllJoyn, this can lead to seamless detection and usage of nearby devices through other (smart) devices such as phones.

Think about the following scenario: you enter a hotel room as part of a business trip and your phone, regardless of which platform it is using, detects the TV, the coffee machine and the wake up radio in your room and you can “configure” and “use” those devices right away through your phone without the need of other remote controls or getting up from the bed to start brewing your coffee. Also media sharing could become much easier than it is today across devices from different vendors using different operating systems.

The potential is huge, the ecosystem nevertheless needs to be developed. And since Microsoft joined the alliance around this protocol and services framework I think we could and want to drive some of this development actively. Let’s see what the future brings based on our early efforts here right now;)

Setup of an AllJoyn Dev-Environment with Visual Studio 2013 on Windows

This blog is solely focused on what you need to do to get the AllJoyn SDK working with Visual Studio 2013. This sounds like a simple thing, but so far the AllJoyn SDK is available for VS 2012, only. Since AllJoyn is using a specific combination of build tools from the OSS-world, tuning the setup to work with VS 2013 requires a few further steps which I’ll dig into as part of this blog-post.

Once you have completed your setup you can start developing AllJoyn enabled services for a variety of devices on Windows machines with Visual Studio 2013 including Windows Services, Desktop Applications and backend-services (e.g. ASP.NET) making use of AllJoyn as a technology and protocol framework.

To setup a full development environment that works with Visual Studio 2013 (and 2012 in parallel), follow these steps below. You need to install exactly the versions of the tools provided below as opposed to the official docs from the AllJoyn home page since these versions of the dependent tools also work with Visual Studio 2013.

  1. Download and extract the AllJoyn Source Code Suite.
    1. Downloading the Windows SDK will give you libraries compiled with VS2012. You definitely will run into issues with using them on VS2013 since there were some changes relevant for AllJoyn in the VS2013 C++ compiler.
    2. In my tests I did use the version 14.06 from the SDK.
    3. For details on how-to work with the Thin SDK, look at Thomas’ blog post who writes about how-to get the Thin SDK to compile with VS 2013 and use it with Intel Galileo Boards running Windows.
    4. Note: for extracting the SDK, I suggest installing a ZIP-tool such as 7-zip which is capable of dealing with *.tar and *.gz archive formats.
  2. Download & install Python 2.7.8.
  3. Install SCONS 2.3.3 (use the Windows Installer Download) or higher (don’t use earlier versions of SCONS).
  4. Install the following tools in addition to Python and SCONS. These are optional, but I’ve installed all of them to make sure not running into other distracting issues:
    1. DoxyGen 1.8.8
    2. Basic MikTex 2.9.5105
    3. GraphViz 2.3.8
    4. Uncrustify 0.57 for Win32
  5. Make sure to have Python as well as SCONs and the other tools in your PATH environment variable.
  6. Fine-tune some of the source files for the AllJoyn SDK before compilation due to changes made from VS2012 to VS2013 in the C++ Libraries and Compiler.
  7. Compile the AllJoyn SDK using SCONS.
  8. Create your VS2013 C++ project to test your compiled AllJoyn Library.

For more details on how-to setup and verify the development environment, in general, also look at Rachel’s blog. She will create a post to explain, how-to install the tools above, make sure you have the environment variables setup correctly and verify if the tools are all available in your PATH environment variable. This post nevertheless will just explain, how-to setup for VS2012 based on the learnings with the official current releases from AllJoyn available at the time of writing this article.

Update the AllJoyn Source to work with VS2013

As mentioned above, Microsoft made some changes in the Visual Studio 2013 compiler (making it more compliant to certain C/C++ standards and defacto-standards). This resulted in a refactoring of some of the standard template and base libraries the AllJoyn SDK makes use of. Also some workarounds AllJoyn used for its cross-platform build with SCONS are not needed in VS2013, anymore. So we need to get rid of those.

Fortunately the changes you have to make are not that many (although they were a bit more challenging to be tracked down:)).

  1. For the following steps, replace <alljoynsdkroot> with the root folder in which you have installed the AllJoyn SDK. To ensure we’re talking about the same directory structure, this is what I assume the AllJoyn root directory looks like:
  2. Helpful background info: the core SDK for AllJoyn is built in C++. For other platforms including C, Java, Node etc. the AllJoyn group has built language bindings which are all based on the core C++ libraries. Therefore some headers are available multiple times in the source code control structure for the different language bindings.
  3. The first file we need to touch is one of the platform mapping headers AllJoyn is using the the core SDK. These header files are used to provide some macros that cover differences/workarounds for core functionality of C/C++ compilers of different platforms.
    1. Open the file <alljoynsdkroot>\alljoyn-suite-14.06.00-src\core\alljoyn\common\inc\qcc\windows\mapping.h
    2. Add the following pre-compiler macro at the beginning of the source file:
    3. Un-comment the following source files at the end of the source file to avoid duplication.
  4. The very same mappings file need to be updated for the C-bindings. For this purpose, conduct exactly the same changes as outlined in step 3 for the mapping.h file in the C language binding folder.
    1. The mapping file to update is called <alljoynsdkroot>\alljoyn-suite-14.06.00-src\core\alljoyn\alljoyn_c\inc\qcc\windows\mapping.h
    2. Perform the same changes as outlined in step 3.
  5. The final change that needs to happen is an update in the SCONs build script files for AllJoyn so that it supports the VS2013 option from SCONS 2.3.3 in addition to the existing VS2010 and VS2012 options.
    1. Open the file <alljoynsdkroot>\alljoyn-suite-14.06.00-src\core\alljoyn\build_core\SConscript
    2. Search for the section in the script that defines the MSVC_VERSION enumeration with allowed values. This will support values up to VC 11.0 incl. VC 11.0Exp for the express edition.
    3. Add an additional value “12.0” to this variable definition as shown below (this assumes you have VS2013 Professional or higher – I didn’t test express but assume 12.0Exp would make it all work with the express edition, as well):

Build with Visual Studio 2013

Now that we have made all changes to platform-headers and SCONS scripts, we can build the libraries for Visual Studio 2013. These can then be used in any VS2013 C/C++ project to enable you develop with the latest and greatest (released) development tools from Microsoft.

  1. Open up a Visual Studio 2013 command prompt.
  2. Change to the directory <alljoynsdkroot>\alljoyn-suite-14.06.00-src\core\alljoyn
  3. Execute the following command:
    scons BINDINGS=”C,C++” MSVC_VERSION=”12.0″ OS=”win7″ CPU=x86_64 WS=”off”
    1. Note that you might see some warnings since a few verifications got stricter in VS2013 C/C++.
    2. The command builds the AllJoyn SDK with the following options:
      1. Language bindings for C and C++. For the other bindings I’d suggest to just use the existing SDKs;)
      2. Visual Studio 2013 using the 12.0 version of the Visual C compiler.
      3. Target operating system Windows 7 (which perfectly works on Windows 8, as well – it does not have any impact on the compiler options or Windows SDK references since all used are standard libraries – the SCONS scripts are just using this for validation of other options, e.g. the CPU-options available for the platform of choice).
      4. White space fixing of source files turned off (WS=”off”). To turn this on, make sure that Uncrustify is set in your path appropriately.
    3. Here a screen-shot of how the start of the build should look like:
  4. Since the build runs a while, wait until it is finished and see if any errors occurred. If not, you find the ready-to-use libraries built with the VS2013 C/C++ compiler under the following folders:
    1. X86: <alljoynsdkroot>\alljoyn-suite-14.06.00-src\core\alljoyn\build\win7\x86_64\debug\dist\cpp
    2. X64: <alljoynsdkroot>\alljoyn-suite-14.06.00-src\core\alljoyn\build\win7\x86_64\debug\dist\c

Creating a VS2013 Console project for testing the libraries:

Finally to verify if things are working, you can create test apps and see if they can join other devices on an AllJoyn bus (refer to Rachel’s blog for details on what an AllJoyn Bus is). In this project you need to refer to the libraries just built.

  1. Create a new Visual C++ project. For our testing purposes I’ve created a Console application (Win32/64).
  2. Update your build configurations as you need them.
    1. E.g. by default the project template will create a Win32 32-bit application.
    2. If you want to have 64-bit, just add the 64-bit configuration as usual for any other C/C++ project.
    3. Note that the subsequent changes in this list need to happen for both, 32-bit and 64-bit build configurations.
  3. In the Project Properties dialog, add the following “VC++ Directories” to your “Include Directories” and “Library Directories” settings so that the compiler finds the dependent header files and compiled libraries. These should now point to the directories used earlier to build AllJoyn with VS2013/VC12.0.
    1. Note: in the screen-shot below I have used a relative path from my solution so that whenever I get the solution to a different machine it will still compile without issues whenever the AllJoyn SDK is put into the same directory on that new machine. I’d suggest doing something similar for your projects, as well, so that re-creating a dev-machine from scratch is predictable and easy-to-do.
    2. Include Directories
    3. Library Directories
  4. If you want to “avoid” IntelliSense complaining that it does not find headers required, also add the “Include Directories” added earlier to the more general “VC++ Directories” option to the option “C/C++ \ General \ Additional Include Directories”. These are exactly the same as those specified in step 3 for “Include Directories”.
  5. Next you need to define a pre-processor that is used by some of the header-files from the language bindings to detect the platform and define different types of Macros and platform types (remember the customization we made earlier to make stuff build on VS2013 – these are some of those). This pre-processor is called QCC_OS_GROUP_WINDOWS as shown below:
  6. Finally you need to tell the VC Linker which libraries to use for the linking process. This ultimately includes some dependencies from AllJoyn itself as well as the built AllJoyn libraries. See below for a list of items to include there:

With all these steps in place, you can start writing some code for AllJoyn. E.g. you can discover devices and services or register yourself as a service in an AllJoyn Bus network – all done with Visual Studio 2013.

For example the following code attaches itself to an existing bus service in a local network and queries for devices that do offer services with a specific service prefix name on this bus:

#include <qcc/platform.h>

#include <assert.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include <alljoyn_c/DBusStdDefines.h>
#include <alljoyn_c/BusAttachment.h>
#include <alljoyn_c/BusObject.h>
#include <alljoyn_c/MsgArg.h>
#include <alljoyn_c/InterfaceDescription.h>
#include <alljoyn_c/version.h>
#include <alljoyn_c/Status.h>

#include <alljoyn_c/BusListener.h>
#include <alljoyn_c/Session.h>
#include <alljoyn_c/PasswordManager.h>

#include <qcc/String.h>
#include <qcc/StringUtil.h>
#include <qcc/Debug.h>

#include<vector>
#include<signal.h>

int main(int argc, char** argv, char** envArg)
{
QStatus status = ER_OK;
char* connectArgs = "null:";
alljoyn_interfacedescription testIntf = NULL;
/* Create a bus listener */
alljoyn_buslistener_callbacks callbacks = {
&buslistener_registered,
NULL,
&found_advertised_name,
NULL,
&name_owner_changed,
NULL,
NULL,
NULL
};
/* Session Port variables */
alljoyn_sessionportlistener_callbacks spl_cbs = {
accept_session_joiner,
NULL
};
alljoyn_sessionopts opts;

printf("AllJoyn Library version: %s\n", alljoyn_getversion());
printf("AllJoyn Library build info: %s\n", alljoyn_getbuildinfo());

/* Install SIGINT handler */
signal(SIGINT, SigIntHandler);

/* Create a password */
alljoyn_passwordmanager_setcredentials("ALLJOYN_PIN_KEYX", "ABCDEFGH");

/* Create message bus and start it */
g_msgBus = alljoyn_busattachment_create(OBJECT_DAEMON_BUSNAME, QCC_TRUE);
if (ER_OK == status) {
status = alljoyn_busattachment_start(g_msgBus);
if (ER_OK != status) {
printf("alljoyn_busattachment_start failed\n");
}
else {
printf("alljoyn_busattachment started.\n");
}
}

/* Register a bus listener in order to get discovery indications */
g_busListener = alljoyn_buslistener_create(&callbacks, NULL);
if (ER_OK == status) {
alljoyn_busattachment_registerbuslistener(g_msgBus, g_busListener);
printf("alljoyn_buslistener registered.\n");
}

/* Connect to the bus */
if (ER_OK == status) {
status = alljoyn_busattachment_connect(g_msgBus, connectArgs);
if (ER_OK != status) {
printf("alljoyn_busattachment_connect
(\"%s\") failed\n", connectArgs);
}
else {
printf("alljoyn_busattachment connected to
\"%s\"\n", alljoyn_busattachment_getconnectspec(g_msgBus));
}
}

/* Create the mutex to avoid multiple parallel executions of foundAdvertisedNames */
gJoinSessionMutex = CreateMutex(NULL, FALSE, NULL);
if (gJoinSessionMutex == NULL) {
printf("Error creating mutex, stopping!");
return -1000;
}

/* Find the Led controller advertised name */
status = alljoyn_busattachment_findadvertisedname(g_msgBus, OBJECT_NAME);
if (ER_OK == status) {
printf("ok alljoyn_busattachment_findadvertisedname %s!\n", OBJECT_NAME);
}
else {
printf("failed alljoyn_busattachment_findadvertisedname
%s!\n", OBJECT_NAME);
}

/* Get the number of expected services
In our test-setup we expect 2 services (arduino and galileo) */
int nExpectedServices = 2;

/* Wait for join session to complete */
while ((g_interrupt == QCC_FALSE) && (i_joinedSessions < nExpectedServices)) {
Sleep(10);
}

/*
Devices found, do whatever needs to be done now
...
*/
}

The code above is using a call-back that is called by the AllJoyn core libraries whenever a device is found. More specific it looks for two devices we expected to be available in our “lab-environment” for testing purposes. One was an Arduino and the other one an Intel Galileo with a LED connected. Both where using the AllJoyn Thin Client Library to connect to the bus.

The only callback relevant was the following one since it is called by AllJoyn libraries when a new device was found that was connected to the same AllJoyn Bus (we’ve used the other callbacks for other tests):

void found_advertised_name(const void* context,
const char* name,
alljoyn_transportmask transport,
const char* namePrefix)
{
printf("\nfound_advertised_name(name=%s, prefix=%s)\n", name, namePrefix);
DWORD dwWaitResult = WaitForSingleObject(gJoinSessionMutex, INFINITE);
s_sessionNames[i_joinedSessions] = (char*)malloc(sizeof(char)*1024);
strcpy_s(s_sessionNames[i_joinedSessions], 1024, name);
i_joinedSessions++;
/* enable concurrent callbacks so joinsession can be called */
alljoyn_busattachment_enableconcurrentcallbacks(g_msgBus);
ReleaseMutex(gJoinSessionMutex);
printf("found advertisments %d\n", i_joinedSessions);
}

Note that the code above is just meant to give you some sort of impression. For full end-2-end scenarios wait for further blog-posts from us as well as look at the official AllJoyn documentation.

Final Thoughts…

Okay, the journey above is a lot of effort, isn’t it? Well, at this point it needs to be said that AllJoyn is still in a very early stage, and therefore some development steps are still a bit hard to get done. Especially setting up an environment that works end-2-end with the tools-chain of your choice (VS2013 in my case at the time of writing this post).

But, I am excited that I am involved in this journey. I see various things happening in the industry that are all geared towards some sort of device mash-up. Think of what Microsoft tried to start a few years ago with Live Mesh. Think of what Apple is doing with their seamless device-2-device interaction which really works great. And consider what Android is doing attempting to do with Android “L”. All of these efforts are really cool and enable great scenarios – but they’re all locked down to one vendor and their specific operating system.

When thinking beyond the ecosystems (Microsoft, Apple, Google) I mentioned above and involving all sorts of devices of our daily live such as (Smart) TVs, Coffee machines, HiFi stereo systems, cars and car information systems, anything related to home automation or even industrial facilities leveraging device mash-ups to solve more complex problems, there MUST be something that AVOIDS vendor lock-in.

AllJoyn can become “this something”. It has the potential. The technology is cool. All it needs now is some sort of dramatic simplification as well as an ecosystem of service interfaces and devices supporting those for all sorts of scenarios across different industries.

We’ll see and experience, how this will work out and where we will end up with this. Microsoft definitely has a huge interest in participating in this journey and you’ll definitely see more around the Microsoft platform and AllJoyn over the course of the upcoming months.

Also look at the Twitter and Blog accounts of my peers since we’re already planning a subsequent hackathon around this topic together with some product groups to dig even deeper into the world of peer-2-peer device mash-ups based on AllJoyn… so that said, expect more to come!!!

Janet Moonshot & FreeRADIUS on Microsoft Azure – An important step for Researchers for being able to use Microsoft’s Public Cloud Platform

Over the past months I’ve spent some time working with Janet the UK’s National Research and Education Network. As well as managing the operation and development of the Janet network, Janet runs a number of services for educationand research including operating eduroam(UK), the UK part of a large, global network established between all sorts of research and education facilities and institutions.

In addition to eduroam and other services, one of the most important projects Janet is leading is the development and standardization of an open platform for authentication, authorization and trust management based on existing standards:

This platform is called Project Moonshot.

Simply put, Moonshot in conjunction with FreeRADIUS is an identity provider and a security token service. Nevertheless it is primarily based on the protocols mentioned above – EAP, RADIUS and GSS-API/SSPI/SASL instead of WS-Federation, OAuth or SAML-P (although one of the token formats supported by Moonshot are SAML tokens).

What I personally think is really cool about Moonshot!?

The really cool and practical thing by being built on the protocols above, is in my personal opinion, that those protocols are supported by almost all relevant operating system platforms deeply integrated since these standards are also used by Kerberos. For example Windows as an OS has SSPI deeply built into the logon-process of Windows which means with an SSPI-provider for Moonshot, the Windows logon itself can be sourced from a federation through a variety of trust-relationships instead of the OS or a direct domain controller by itself. That is one thing that is not possible with the commonly more widely known standards such as WS-Fed, OAuth or SAML-P since they’re all web-focused.

Why is Moonshot so important for Microsoft and Microsoft Azure?

Independent of what I think the advantages are of Moonshot, the most important part for Microsoft and Microsoft Azure is, that Janet is working with NRENs, academia and research across Europe and internationally to establish Moonshot as THE prime authentication mechanism for research communities, building up trust relationships between them and thus allowing federated authentication and authorization for research projects across the world.

In other words: if Microsoft Azure wants to play an important role in research in the future, Moonshot needs to be somehow supported in Azure as a platform. Through our partnership and work with Janet we achieved a first step for this over the past months, together!

Moonshot IdP Base Image on VMDepot…

Working together with Janet we managed to get a base-image prepared, tested and published on Microsoft Open Technologies VMDepot that can be used by anyone who wants to get connected to research communities through Moonshot Trust Routers and IdPs for federated security.

You can find this image here on VMDepot for getting started.

Although it seems like a simple thing to do, we had to undergo a few steps to get this far. Moonshot was required to be updated to support the Linux-distributions officially supported on Microsoft Azure. Furthermore we had to test if the semantics of the protocols used, especially EAP, do work well on Microsoft Azure. At least for single VM deployments we did this and the image above on VMDepot contains all the bits with which we’ve tested.

Of course that’s just the first step and we know we need to take some future steps such as making the deployments ready for multi-instance deployments for the sake of high availability and eventually also performance. Nevertheless, this is a great first step which was required and enables us to move forward.

The image by itself is based on Ubuntu Linux 12.10 LTS, it has the Moonshot and FreeRADIUS package repositories configured correctly and has other useful packages installed that are required or nice to have for Moonshot and FreeRADIUS (such as “screen” for example).

Using the Moonshot/FreeRADIUS VMDepot Image

Next I’d like to summarize how you can make use of the Moonshot VMDepot image. Note that most probably you should be involved in academia or research projects for this to be useful to you J. Of course you can also setup your own, single IdP using the image, but the full power gets unleashed when you become part of the Janet Trust Router network which is what I am focusing on in this blog post right now.

Let’s start with a few assumptions / prerequisites:

  • Assumption #1:
    Since the primary target group is academia and research which is very Linux-focused, I am assuming people who’re trying this will most probably try the steps below from a Linux-machine. Therefore I am only using tools that also work on Linux (or Mac), although I am running them from a Windows machine.
  • Assumption #2:
    For the steps to complete you need an active Microsoft Azure subscription. To get one, navigate to http://azure.microsoft.com and click on the “Free Trial” button in the upper, right corner.
  • Assumption #3:
    You are able to get in touch with Janet to participate in their Moonshot pilot to get the credentials required to connect your Moonshot IdP/RP to their Trust Router Network and that way become part of the larger UK and global academia and research community implementing Moonshot.

Now let’s get started with the actual deployment of a Moonshot VM based on the image Janet and we have published together on VMDepot:

  • Install Node.js if on your machine if not done, yet.
    Node.js is needed since the Microsoft Azure Cross Platform Command Line Interface which we will use for setting up the Azure environment is built with Node.js.
  • Install the Azure Cross Platform Command Line Interface (xplat CLI).
    The Azure xplat CLI is a command line interface that allows you to script many management operations for services in your Azure subscription from either a Linux, Mac or also a Windows machine. For more details on setting it up, please refer to the Azure xplat CLI homepage.
  • Import your Subscription Publish Profile Settings through the xplat CLI.
    Before you an issue any operation to your Azure subscription in the cloud through the xplat CLI, you need to download and import a credentials file. This is the only operation that requires a GUI with a web browser, so if you issue the following command, you should sit on a machine with x-Windows installed or be on a Mac or Windows machine. Open a shell-window or a command prompt and execute the following command:
    • azure account download
      This will open a web browser and browse to a page where you’ll need to sign-in with the account that has access to your Azure subscription (your Microsoft account with which the subscription has been created or which is a Co-Admin of another subscription). It results in the download of a “xyz.publishsettings”-file which contains the credentials. Save that file to your local disk. Next execute the subsequent command:
    • azure account import <path & filename to xyz.publishsettings>
      This command finally makes the Azure xplat CLI aware of your credentials. After that step you can finally start with true management commands against your subscription.
    • Note: if you have multiple Azure subscriptions, you also need to select the subscription in which you want to create the VM using azure account set <subscription-id>
  • Create a VM image based on our VMDepot base image using the xplat CLI.
    Finally we can create the Virtual Machine based on the VMDepot image. For this purpose execute the following command in your previously opened shell:
    • azure vm create yourdnsprefix -o vmdepot-28998-1-16 -l “North Europe” yourusername yourpassword –ssh
    • This command creates a VM which will get a public DNS-name called “yourdnsprefix.cloudapp.net” through which you then can connect to your VM (e.g. via SSH).
    • The result of issuing the command should look similar to the following:
    • What you see here is that the script transfers the template for the virtual machine from VMDepot with the VMDepot image id 28998-1-16 to your storage account and then creates the VM from that template. Finally it does some clean-up stuff.
  • Make sure Moonshot/FreeRADIUS and SSH endpoints are open on the Azure firewall.
    The last step is to open up the required TCP-endpoints on the Azure firewall. This can happen after the VM has been created successfully. Ports required are 2083 and 12309 for Moonshot/FreeRADIUS, SSH is open by default on 22 given our previous command including the -ssh switch. Issue the following commands:
    • azure vm endpoint create-multiple DNS_PREFIX 2083:2083,12309:12309
    • The result should look similar to the following (note that I’ve added port 22 before, already, therefore you won’t see it in the screenshot):

After you’ve completed those steps and the VM has been created, successfully, you need to connect to the VM and perform the final Moonshot/FreeRADIUS-configuration steps. These are pretty much the same as those you’d need to do on an on-premise machine in your own data center, we’ve prepared anything in that image in a way that it should work smooth in Azure.

  • SSH into the newly created VM.
    Make sure you connect as root so you can perform all administrative tasks. All subsequent steps are to be executed in that SSH-session to your newly created VM!
  • Update to the latest package versions.
    Since the image is Ubuntu-based, use apt-* to update to the latest version of the packages. Issue the following commands:
  • Update the FreeRADIUS certificate files to match your organizational values.
    As part of the bootstrapping process, Moonshot and FreeRADIUS generate certificate files required for setting up trust relationships between your RP/IdP and other RP/IdPs. These are generated through openssl based on settings-files prepared in the bootstrap-image from VMDepot. You should customize those to match your organizational values, for example such as the common name to be used for your organization and IdP. Perform the following steps:
    • Switch to the directory /etc/freeradius/certs.
    • Open the file ca.cnf and update the following values to match your own values:
      • emailAddress
      • commonName

    It should look similar to the following if you’re using VI, if you’re really taking it serious, then also update the other values (e.g. passwords for private key files):

    • Open the file server.cnf and update the following values to match your own values:
      • emailAddress
      • commonName
      • It should look similar to the following if you’re using VI:
    • Finally update the same values also in the file client.cnf to match your own values:
    • Execute the command sudo /etc/freeradius/certs/bootstrap. This produces a lot of output, but at the end your screen will look similar to the following after executing this command:
  • Fine-tune Client Private Key Files:
    Next you need to “fine-tune” the private key files for the clients. This is supposed to be something that will be fixed/made easier in future versions of Moonshot and FreeRADIUS. Perform the following steps:
    • Change to the directory /etc/freeradius/certs.
    • Run the following command: cat client.crt client.key > client.txt
    • Now overwrite client.key with client.txt by executing the following command:
      mv client.both client.key
    • Open client.key in VI and delete all lines until the first —- BEGIN CERTIFICATE —- appearance as shown below:
  • Realm configuration – Part #1
    Now that all certificates are configured, you need to configure your “realm”-settings such as the name of your realm and other options Moonshot and FreeRADIUS allows you to set. For this blog-post we keep it with the simple creation of a realm for your setup:
    • Switch to the directory /etc/freeradius.
    • Open the file proxy.conf in VI and add the following section anywhere in the file:
      realm yourrealm.com
      {
      }
    • The realm you select should match the DNS-name you’re planning to use for setup. This DNS-name should then be mapped using a DNS CNAME-alias or DNS A-Record to your xyz.cloudapp.net setup in Azure.
    • You can look at the sample-realm configurations in the file so that you can decide which other options you’d prefer to set for your setup. For this post we keep things at a default-setup.
  • Realm configuration – Part #2:
    For the next realm-setting perform the following steps in the SSH-session:
    • Open the file /etc/freeradius/mods-enabled/realm in VI for editing.
    • Add the following section at any place in the file:
      realm suffix {
      change rp_realm = “yourserver.yourrealm.com”
      }
    • Make sure you use the same domain-name as before (e.g. yourrealm.com) and that the name you specify here (yourserver.yourrealm.com) is a resolvable DNS-name.
    • The results should look similar to the following:
  • Modify post-authentication step that issues the SAML assertion.
    Next you need to modify the post-authentication steps. One action that happens in those post-authentication steps is the definition of SAML-assertions that get issued as a token after a successful authentication. We prepared the image with a default-template that you can customize based on your need. But even if you don’t customize the assertions, there’s one step you need to complete and that’s bringing your realm into the context of the post-authentication steps.
    • Open the file /etc/freeradius/sites-enabled/default with VI.
    • Search for a configuration section starting with post auth { … }.
    • Modify it to issue the SAML-assertion for your realm as follows:
      post-auth {
          if (Realm == LOCAL) …
      change to
          if (Realm == “cloudapp.net”) (same as above)
      }
    • The result should look similar as the following:
  • Request and setup Trust Router Credentials (through Janet).
    As mentioned most use of a Moonshot/FreeRADIUS install is given when you connect it to a research community. For this purpose get in touch with Janet to join their pilot via https://www.ja.net/products-services/janet-futures/moonshotOnce accepted onto the pilot, Janet  will send you Trust Router credentials which allow you to get into a federation with the research network Janet operates.
    • Janet (or other Trust Router operators) will send you the trust router credentials for setting up the trust relationship as an XML file. Put that XML-file on your Moonshot VM created earlier.
    • Next execute the following commands (assuming the XML-file with the Trust Router credentials is called mytrustcreds.xml):
          su –shell /bin/bash freerad
          unset DISPLAY
          moonshot-webp -f mytrustcreds.xml
    • With those credentials your IdP/RP will be able to connect and federate with the Trust Router network Janet operates for academia and research (or the one you’ve received the credentials for).

Finally that’s it, we’ve completed all steps for configuring the Moonshot/FreeRADIUS setup. Now it’s up to test the environment or start using it for your single-sign-on and authentication purposes. A simple test for your environment together with Janet could look as follows:

  • Open up three terminal sessions to your Moonshot/FreeRADIUS VM you just created.
  • In Terminal #1 perform the following steps:
    • Open /etc/freeradius/users using VI.
    • Look for the following line: testuser Cleartext-Password := “testing”
    • Leave it for the test or modify it as per your needs. Also that’s where you could add your own users of your IdP. If you leave it as above that’s the credentials you can use for testing.
    • Now execute the following commands:
      • su –shell /bin/bash freerad (runs a shell under the FreeRADIUS user)
      • freeradius -fxx -l stdout (runs freeradius for debugging with logging to stdout)
  • In Terminal #2 perform the following steps:
    • moonshot-webp -f <path to previously received trust router credentials XML>
    • tids <your-external-ip> trustrouter@apc.moonshot.ja.net /var/tmp/keys
      • The external IP for your Azure VM is visible in the Azure Management portal (manage.windowsazure.com) for your virtual machine.
      • trustrouter@apc.moonshot.ja.net is an example for a trusted trust router. In case you federate with Janet, that’s most likely the one you’ll use.
  • In Terminal #3 perform the following step:
    • tidc tr1.moonshot.ja.net {your rp-realm} apc.moonshot.ja.net apc.moonshot.ja.net
  • Important Note: for the commands above to succeed, you need to have valid Janet Trust Router credentials and Janet needs to have your IdP/RP configured in their trust-settings as a trusted party! Otherwise later when executing the tidc-command the test will fail!
  • Finally to complete the test someone needs to use Moonshot and its identity selector on a client machine to authenticate using your IdP. The best way to do that is using the LiveDVD for Moonshot provided by Janet.

That’s it, now you have your Moonshot / FreeRADIUS IdP to get yourself connected with a huge community of researchers, scientists and students across the world… for further questions it’s best to get in touch with the people from Janet and Moonshot via moonshot-community@jiscmail.ac.uk. And go to the Moonshot home page to find more details here:

https://community.ja.net/groups/moonshot

https://www.ja.net/products-services/janet-futures/moonshot

GeRes2 – An OSS Framework for Reliable Execution of Background Jobs on Azure

A few weeks ago we published an open source framework that covers a scenario required by many partners I’ve been working with on Azure over the course of the past 4 years: asynchronous execution of jobs on compute instances in the background – or simple batch processing. Finally now I found the time (on the plane) to blog a bit about it and point you to that framework.

Being one of the product owners of this OSS framework I am very proud of what we achieved. Note that there might be something way more sophisticated and complete coming as a platform Service in Microsoft Azure at some point in time in the future. Until that time, our OSS Framework fills this requirement of “batch-processing” in a more simplified way.

Generic Resource Scheduler v2.0 (GeRes2) for Microsoft Azure

In a nutshell, GeRes2 is a framework for scalable, reliable and asynchronous execution of a large amount of jobs on many Worker Machines in Windows Azure. It is built with Azure Cloud Services, provides a Web API for submitting jobs and querying data about jobs and executes Jobs in parallel across multiple worker roles.

As such, GeRes2 implements a ready-to-use framework which you can use in your own solutions. It is all available as open source and you can use the code or the deployment packages to get GeRes2 running in your own subscriptions.

http://geres2.codeplex.com

GeRes2 essentially implements an all-so well-known pattern from the Azure-world as shown in the sketch below:

Classic scenarios partners are implementing through a pattern like this are:

– Generating documents in the background (e.g. massive amount of documents)
– Image or Video post processing (although for Video you should consider WAMS)
– Complex, distributed calculations
– Asynchronous provisioning jobs (e.g. when provisioning new tenants for your system)
– anything that needs to run multiple jobs in parallel reliably and asynchronously…

Of course this outlines just the generic pattern and it looks fairly simple. But when digging into details there are a whole lot of challenges that need to be solved when implementing such an architecture such as:

– Execution of long-running and short-running jobs across multiple instances
– Status tracking & status-querying of jobs
– Notifications (status & progress) to consumers and clients
– Prioritization of jobs
– Reliability features (e.g. retry logic, dead-letter handling etc.)
– AutoScaling that is aware of which instances are not executing long-running jobs

Exactly those are the features GeRes2 implements for you – so you can take care of the implementation and scheduling of your jobs. For the while GeRes2 fills a gap that might be filled through a native Azure platform service later (as of today no details are available).

Side Note: We built GeRes2 because we do have partners that needed the functionality outlined below right now. Note that some for you obvious features might be missing since partners we’ve worked with did not immediately require those features, today.

Side Note: if you have HPC-background than you might think about a separation of jobs and tasks (jobs as compositions of more fine-granular tasks). In GeRes2 we don’t make that separation, so for GeRes2 Jobs are equal to Tasks and you can use that tger

Backend Work: Implementing a Job for GeRes2

Creating jobs is fairly easy (whereas we focused on .NET developers, first): You need to create a library that implements an Interface which contains the logic of your job. The interface is part of the server-SDK from GeRes2 that defines the IJobImplementation-interface.

public class FinancialYearEnd : IJobImplementation
{
    // make the property volatile as it may be set by more than one thread
    private volatile bool _cancellationToken = false;


    // default constructor (required by MEF)
    public FinancialYearEnd()
    {
    }

    public JobProcessResult DoWork(Job job,
        string jobPackagePath, string jobWorkingPath, 
        Action<string> progressCallback)
    {
        // Your Job-logic goes here
    }

    public string JobType
    {
        get { return "FinancialYearEnd"; }
    }


    public void CancelProcessCallback()
    {
        _cancellationToken = true;
    }
}

Every job gets executed on a single Worker Role instance reserved by GeRes2 for job-execution. The number of jobs executed in parallel is defined by the number of workers you reserve for the job execution. On each worker. every job gets his own, isolated directory where the job files (executables, anything required by a job) are temporarily stored for execution as well as a temporary folder available for local processing.

Job files are deployed automatically by GeRes2 from Azure Blob storage to the worker VMs on job execution. So all you need to do is implement your job, package it in a ZIP-archive and push it into a container called “jobprocessorfiles” in Blob Storage (in a sub-folder matching your TenantName). For more details look at the documentation Wiki from the GeRes2 OSS project workspace.

Front-End Work: Submission of Jobs for Execution and Notifications

When GeRes2 is deployed in your subscription it provides a WebAPI secured by Azure Active Directory. More details are also available on the documentation wiki from GeRes2. We also provide a client/consumer-SDK for applications and services that want to use GeRes2 for submitting jobs.

With that SDK the submission of a Job is as simple as shown in the following lines of code:

var geresServiceClient = new GeresAzureAdServiceClient
                            (
                                baseUrl,
                                azureAdTenant,
                                azureAdWebApiId
                            );

geresServiceClient.Authenticate(clientId, clientSecret).Wait();

var newJob = new Job
{
    JobType = "FinancialYearEnd",
    JobName = "Job Display Name",
    TenantName = "YourTenant",
    JobProcessorPackageName = "yourjobpackage.zip",
    Parameters = "any parameters you need to pass in"
};

string jobId = geresServiceClient.Management.SubmitJob(newJob, batchId).Result;

This leads to the execution of a job in the GeRes2 system on a worker. The status of a job can be tracked in two ways:

– Through “near-real-time” notifications via SignalR and/or Service Bus.
– By polling the monitoring API of GeRes2.

Setting up notifications with SignalR can happen through the client-SDK, directly, as shown in the following code-snippet:

//
// Setup notification handlers
//
geresServiceClient.Notifications.JobStarted += (sender, args) =>
{
    // ...
};
geresServiceClient.Notifications.JobProgressed += (sender, args) =>
{
    // ...
};
geresServiceClient.Notifications.JobCompleted += (sender, args) =>
{
    // ...
};

//
// Connect to the SignalR Hub
//
geresServiceClient.Notifications.Connect().Wait();

//
// Schedule jobs and subscribe to job notifications
//
string jobId = geresServiceClient.Management.SubmitJob(newJob, batchId).Result;
geresServiceClient.Notifications.SubscribeToJob(jobId);

As mentioned, there’s also the option to query the status of jobs manually. The status of jobs is kept in an Azure-table. Of course you can query that table, directly since you are in full control of your GeRes2 deployment. But we’ve created a WebAPI and a corresponding client-API that should make it a bit easier for you as shown below:

var batchesTask = geresServiceClient.Monitoring.GetBatches();

geresServiceClient.Monitoring.PageSize = pageSize;
var jobsList = geresServiceClient.Monitoring.GetJobsFromBatch(batchIdToGetJobs);

var jobDetails = geresServiceClient.Monitoring.GetJobDetails(j.JobId);

Final remarks:

We’ve created GeRes2 based on concrete partner projects we’ve been involved into across the world. They all started implementing the pattern outlined above manually and it always turned out it is a whole lot of work that needs to get done.

GeRes2 implements a simple set of batch-processing features we’ve seen a lot with partners across the world. Of course it is not a full-blown HPC or batch-processing/job-scheduling platform. It covers simple yet often required scenarios. Nevertheless, what I can confirm is that we’ve partners who are using GeRes2 with > 100 job execution instances and tested way above 100 instances (to be precise: with 800 instances; but note that the built-in AutoScaler works up to 100 instances, only, but is not required to be used, either).

If you’re interested, then go to our project workspace, download and play the source code, dig into our docs and share feedback if you have.

http://geres2.codeplex.com
is where you can look for further details.

Have much fun and thanks for your interest!