Category: Uncategorized

Mapping OPGW Gateway versions

The trouble with OnPrem gateway services

Everyone that have been running the On Premise Data gateway for a while notice that the windows service that is installed on the machine needs to be updated. Microsoft releases a new version about once a month, and in order to maintain support, you need to update about every six months. Updating is a rather manual affair but there is another problem: Knowing which machines that needs to be updated.

The problem

You are running some know, or unknown, number of on premise data gateways in your organization. You need to know the names of the machines that host the Gateway services and the current version of the service.

The issue tend to be that gateways are installed just about everywhere and with little to no documentation. This is one of its main features: The flexibility. Much better than having to go thru the boring rigours of the network.

So you are tasked with finding all the gateways and make sure they are updated. This post help you find the gateways.

Solving it using different tools

There is usually several ways of solving a problem and this is no exception. I will show three different tools and the limitations of each.

Using the Azure Portal

One way of finding the machine name is to use the portal. Find the gateway, look at the overview page and the machine name is right there. However, that is it. You cannot see the service version or if that service version is new or old. You cannot even see if the service is online or not.

So, using the portal is not recommended.

Using the Power Platform Admin center

This very useful management tool actually contains all the information you need, but not in a very mapping-friendly way.

When you log in you get a nice long list of all your gateway clusters for the default region.

You can even get the status (online?) by clicking the arrows symbol to the right, and get the machine name and statuses by clicking the i-symbol next to the clustername in the list.

You can get the machine name in the “device” column and also get the gateway version.

There are some drawbacks though:

  • If you have gateways in multiple regions, you need to switch between them all the time. There is no listAll option.
  • If you have many installations there is a lot of clicking just to get the status.
  • You still do not know if the version is old enough to need an upgrade.

Using a custom script

As luck would have it, I have made a custom PS-script that solves these issues. You can find it on my GitHub page.

The team at Microsoft released a PowerShell Module that allows you to script information about your gateway clusters and machines, so that is what I did. You can download the script and simply run it. I will walk thru the script here.

1. PowerShell 7.x

You need to run this script on PowerShell 7.x, since that is required by the Microsoft module.

2. Download/Install the Gateway Module

Easy, just type Install-Module DataGateway and go thru the setup.

3. Login to be able to access the gateways

Type: Login-DataGatewayServiceAccount and login using the account that is granted access to gateway management.
You will only get information about the gateways you are allowed to view. This means that in some cases you might think that there are no gateways, but it’s just your limited access rights.

4. Update the regions array

On row 9 there is an array that contains all the regions you want to map. If you know you only have gateways in eastus, then put that in the array.

As ever, items in an array in PowerShell is separated by a comma.

The region names are the official Azure Region Names.

Here is an example of multiple regions.

$regions = @('northeurope','francecentral','australiaeast')

4. The script

Have you ever done any PowerShell scripting you will see that it is very straight forward:
– For each region: Get All clusters.
– For each cluster: Get Status and Member Gateways.
– For each member gateway: Get the gateway name and machine name.
– Add the row to an output object.
– When done: save the output object to a CSV-file.

5. The output

Beside the obvious ones, like clustername, there are some additional fields that you will find useful.

Header Description
GatewayMachines A list of all the machinenames connected to the cluster (hosting a gateway service)
Cluster Status Is it online (live) or not?
Version Status If “Update is required”: Update your gateway installation.

Conclusion

You can use the PowerPlatform Admin Center to get almost all the information you need, but the powershell module provides even more and using that in a PowerShell script, you can create a very useful overview of all the gateway installations in your organization.

My talk on Why use Azure API manager

In the middle of March 2022 I did a 101 talk about Why use API management. It went very well and is available on YouTube.

Thank you to Liam Hampton for assisting and focusing my talk. It is hard to make things “easy enough” when you have been using the product as long as I have.

I was very happy with the talk, and think it is a great way into the use of API management.

Coming Sessions

We have a deep dive workshop slated for March 27th. A real, live and in-person, event.
You can find more information on it on the Reactor Stockholm Meetup page.

Me on YouTube

Upload files to blob with minimal access

Uploading files from AzureDevops

When you deploy things to Azure you often find yourself using the Azure File Copy task. Sometimes to allow an APIm deploy to read policy files, sometimes when you use ARM master templates and sometimes just to deploy files to a storage.

Using minimal access levels

When deploying from Azure DevOps you use a Service Connection. That connection is represented as an Application Registration and Enterprise Application in your Azure Tenant.

Using the Enterprose Application you set the access rights needed by Azure Devops in order to deploy ARM-templates or upload files.

One way of solving a problem is to use the big hammertm and just make the Service Connection identity subscription Owner, but, like a big hammer, that can cause big problems. I like using the minimal approach. Do not assign higher access rights than the deployment needs.

What is needed to upload

They are really strange but these are the exact, minimal access-rights (or role assignments) needed for the storage account you want to upload a file to.

Role Scope
Contributor The Storage Account
Reader The Resource Group containing the storage account
Storage Blob Contributor The Storage Account

But wait there is more

Additionally you need to create the Container you want to upload files to. If you do not create the container, you will get a very misrepresenting error along the lines of

[error]Storage account: [accountname] not found. The selected service connection 'Service Principal' supports storage accounts of Azure Resource Manager type only.

Yeah, I did this … don’t make the same … mistake. I mean, the error does say that it cannot find the storage account. It should say

[error]Storage URI: The URI [URI here] was not found. Make sure the whole URI exists and is accessable by the Service Principal.

How to deploy a host key to Azure Functions

The reason for this post is faulty documentation, and there are even some other posts showing ARM-templates with errors in them.

Creating a host-key using ARM/Bicep

You need to add a hostkey to an existing function during deployment. Here is how to do it. As ever if you just want the code, scroll to the ARM-code towards the end of the post.

Why?

There might be many reasons for doing this, but mine was to allow authentication from an API. The function is already using the built in AAD functionality (see this blog post for more info). Azure Functions also have a built in functionality of using host keys for authentication. It is just like an API-key, and I would like to keep that security feature.

ALM cycles

You often might deploy a Function and API at the same time, within the same pipeline, but I encourage you to rethink this behaviour. Make your APIm a first level citizen in your integration platform. An API that calls a Function today, might call additional services tomorrow, and any update to that API would require a redeploy of the Function.

This is why you need to keep your API in its own ALM cycle.

When to deploy the key

Since we need to separate the Function and the API we cannot deploy the host key with the function. Once again: we would need to redeploy the Function every time we update the API.

I deploy the host key when I deploy the API. The API, and its authentication should be created at the same time.

You could argue that the key is deployed with the function, and the API deployment gets the key from the Function at deploy time. This hinges on the function deployment being responsible for creating all the keys that are needed for the callers. My way allows you to deploy the Function and focus on the code and leave the authorization to the APIm, which is kind of why APIm is used in the first place.

The function does not reset

If you deploy your Azure Functions in a standardized way, the key created by the API deployment will not be deleted during a redeploy. So any post release bug fix does not affect the APIs access to the function.

It will delete the key if you delete the whole function before redeploy, but why would you do that?

Deploying the Key

I use Azure Devops and ARM but I have provided a Bicep version as well. When using Azure Devops I get all the upsides of release management, which is always useful.

The ARM Template

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "FunctionAppName": {
            "type": "String"
        },
        "FunctionAppKey": {
            "type": "SecureString"
        }
    },
    "variables": { "keyName": "MyAPIName-key')]" },
    "functions": [],
    "resources": [
        {
            "type": "Microsoft.Web/sites/host/functionKeys",
            "apiVersion": "2018-11-01",
            "name": "[concat(parameters('FunctionAppName'), '/default/', variables('keyName'))]",
            "properties": {
                "name": "[variables('keyName')]",
                "value": "[parameters('FunctionAppKey')]"
            }
        }
    ],
    "outputs": {}
}

Notice that the FunctionAppKey is a SecureString. This is for keeping the value hidden during and after deployment.

The Context

Deploying the ARM using CI/CD you simply use the ever useful Deploy ARM Template step and refer to the ARM template. Here are some pointers:
– Remeber that the Function is usually in another resource group than the API manager. Update the Resource Group setting.
– Store the Function App key as a variable in the release. Either as a secret value, or as a variable group (pointing to a keyvault).
– Supply the Function App key in the “Override Template parameters” input box.

For more information about deploying values using either parameters or a Keyvault read my post “How I deploy Keyvault values”.

The conclusion

Make your APIm a first level citizen in your ALM cycle. As such, the API deploys anything relating to its access and the best way to do that is using a separate ARM template and run that template at the same time you deploy any update to the API.

Deploying the Front Door using Bicep

Bicep is a very nice language that rests on top of ARM. I have always preferred ARM to scripting as “wanted state” is a much better way of looking at anything relating to infrastructure.

But this reliance on ARM also means that whatever template definition is output by the product, that is what you must use. Case in point: Azure Front Door.

Circular reference

A circular reference in an age old problem in coding. You cannot use circual references because, logically, it does not work. It is a series of references where the last object references the first, resulting in a closed loop.

Within Azure Front Door (or AFD) we have this problem, a lot. Within AFD it refers to itself, and if you add a Rules Engine (RE), that refers to the AFD, that refers to the RE.

If you do like I do: Setup something the first time using the portal, and the extract the ARM using the Export Template, or Insert Resource in VS Code to get the Bicep Code directly, you will find that the generated ARM contains a lot of errors due to circular references.

We need to overcome this, and solve some other issues.

The first problem

The first problem comes about 5 rows in, when we need to define the (first) routing rule. You need to assign it an ID. This ID consist of the resource ID for the AFD that we are just about to create!

This is the first of many circular references.

The solution

I solved this, and all other circular references, by using variables and string interpolation. Simply put: Create the resource ID yourself. You know what the ID will be, because you are just about to create it.

Create two useful variables at the top of the bicep file.

var frontDoorName = 'contoso-afd-test'
var frontDoorResourceId = '{resourceGroup().id}/providers/Microsoft.Network/frontDoors/{frontDoorName}'

You must deploy the AFD within the context of a resource group, even if the AFD is a global service. You get the start of the resource ID by reffering to this, and the add the rest. The resulting frontDoorResourceId would be:

/subscriptions/GUIDHERE/resourcegroups/contoso-TEST-rg/providers/Microsoft.Network/frontdoors/contoso-afd-test

Now, we can use the frontDoorResourceId variable, every time a sub-resource needs to be defined, such as a routing rule or backend pool, as defined below.

routingRules: [
  {
    id: '{frontDoorResourceId}/RoutingRules/APIm'
    name: 'APIm'
    properties: {
      routeConfiguration: {
        forwardingProtocol: 'HttpsOnly'
        backendPool: {
          id: '{frontDoorResourceId}/BackendPools/ApimEurope${Env}'
        }
        '@odata.type': '#Microsoft.Azure.FrontDoor.Models.FrontdoorForwardingConfiguration'
      }
      ...
    }
  }
]

The second problem

The next problem is the rules engines. In the template definition it looks like you need to add another Resource ID, but there is a problem. The Rules Engines is a totally different resource type and it needs to be created separately. This means you simply cannot refer to it in the same way as with the frontDoorResourceId above.

The solution

It is very easy: Do not add the property to the AFD Definition. This property is not read only, but it will be assigned at deploy time by the Rules Engines Definition you add to the same Bicep File.

resource frontDoorRulesEngine 'Microsoft.Network/frontDoors/rulesEngines@2020-05-01' = {
  name: rulesEngineName
  parent: frontDoor
  properties: {
    rules: [
      {
        name: 'RuleNumberOne'
        priority: 0
        action: {
          routeConfigurationOverride: {
            forwardingProtocol: 'HttpsOnly'
            backendPool: {
              id: '${frontDoorResourceId}/BackendPools/FightClub'
            }
            '@odata.type': '#Microsoft.Azure.FrontDoor.Models.FrontdoorForwardingConfiguration'
          }
          requestHeaderActions: []
          responseHeaderActions: []
        }
  ...
  // Rule settings go here
  ...

      }
    ]
  }
}

You assign the AFD you are about to create as the rules engine’s parent. This will connect the AFD and the RE at deploytime.

Note the use of the frontDoorResourceId to refer to the Backend Pool.

The third (last) problem

The this is only a problem if you are using the Web Application Firewall (WAF) and you should. If you need to enable the WAF you must refer to a policy, or rather a FrontDoorWebApplicationFirewallPolicies resource. You do this when you create your frontendEndpoints.

A solution

This is not the solution, but I know it works for me. I define the WAF policies in a separate Bicep file as these tend to be separate from the Front Door. Remember, you can reuse the same policy for different WAFs.

Create a separate file with the resource definition. If you want to send a customized error message you need to run the base64 string function on it before deployment. This is much more useful than converting it to Base64 and pasting that sting into the Bicep-file.

Name the policy using a parameter.

param PolicyName string

resource defaultPolicy 'Microsoft.Network/FrontDoorWebApplicationFirewallPolicies@2020-11-01' = {
  name: PolicyName
   policySettings: {
     enabledState: 'Enabled'
     mode: 'Prevention'
     customBlockResponseStatusCode: 406
     customBlockResponseBody: base64('<html><head><title>You are blocked</title></head><body bgcolor="#FFB299"><p><h1>WAF custom response</h1>You are being blocked. If you need access, contact support with this reference{{azure-ref}}</p></body></html>')
   }
}

Send the policy name parameter to your AFD bicep and construct the reference using string interpolation. My naming below assumes that the policy is deployed in the same resource group as the AFD. If that is not your case, create a reference to that resource group in the bicep file, or update your bicep file for the policy to use an output parameter and send that to the AFD bicep deployment.

About the error message and the use of azure-ref, consult this documentation.

output policyResourceID string = myPolicy.id

This is how I constructed the resource reference in my AFD Bicep file. Note the liberal use of the frontDoorResourceId and frontDoorName variables.

frontendEndpoints: [
  {
    id: '{frontDoorResourceId}/FrontendEndpoints/{frontDoorName}-azurefd-net'
    name: '{frontDoorName}-azurefd-net'
    properties: {
      hostName: '{frontDoorName}.azurefd.net'
      sessionAffinityEnabledState: 'Enabled'
      sessionAffinityTtlSeconds: 0
      webApplicationFirewallPolicyLink: {
        id: '{resourceGroup().id}/providers/Microsoft.Network/frontdoorwebapplicationfirewallpolicies/{WAFPolicyName}'
      }
    }
  }
]

Conclusion

Deploying AFD is possible but also a bit of a pain using Bicep. This is notdue to the language, but the ARM template. AFD is a very old service that was in use by the XBOX network for years before being made available in Azure. I think this shows, and I hope for a revisional update at some point.

Still: This should NOT deter you from using Bicep (or ARM) and infrastructure as code for your AFD installations. Please avoid PowerShell and Azure Cli.