Upload files to blob with minimal access

Uploading files from AzureDevops

When you deploy things to Azure you often find yourself using the Azure File Copy task. Sometimes to allow an APIm deploy to read policy files, sometimes when you use ARM master templates and sometimes just to deploy files to a storage.

Using minimal access levels

When deploying from Azure DevOps you use a Service Connection. That connection is represented as an Application Registration and Enterprise Application in your Azure Tenant.

Using the Enterprose Application you set the access rights needed by Azure Devops in order to deploy ARM-templates or upload files.

One way of solving a problem is to use the big hammertm and just make the Service Connection identity subscription Owner, but, like a big hammer, that can cause big problems. I like using the minimal approach. Do not assign higher access rights than the deployment needs.

What is needed to upload

They are really strange but these are the exact, minimal access-rights (or role assignments) needed for the storage account you want to upload a file to.

Role Scope
Contributor The Storage Account
Reader The Resource Group containing the storage account
Storage Blob Contributor The Storage Account

But wait there is more

Additionally you need to create the Container you want to upload files to. If you do not create the container, you will get a very misrepresenting error along the lines of

[error]Storage account: [accountname] not found. The selected service connection 'Service Principal' supports storage accounts of Azure Resource Manager type only.

Yeah, I did this … don’t make the same … mistake. I mean, the error does say that it cannot find the storage account. It should say

[error]Storage URI: The URI [URI here] was not found. Make sure the whole URI exists and is accessable by the Service Principal.

How to deploy a host key to Azure Functions

The reason for this post is faulty documentation, and there are even some other posts showing ARM-templates with errors in them.

Creating a host-key using ARM/Bicep

You need to add a hostkey to an existing function during deployment. Here is how to do it. As ever if you just want the code, scroll to the ARM-code towards the end of the post.

Why?

There might be many reasons for doing this, but mine was to allow authentication from an API. The function is already using the built in AAD functionality (see this blog post for more info). Azure Functions also have a built in functionality of using host keys for authentication. It is just like an API-key, and I would like to keep that security feature.

ALM cycles

You often might deploy a Function and API at the same time, within the same pipeline, but I encourage you to rethink this behaviour. Make your APIm a first level citizen in your integration platform. An API that calls a Function today, might call additional services tomorrow, and any update to that API would require a redeploy of the Function.

This is why you need to keep your API in its own ALM cycle.

When to deploy the key

Since we need to separate the Function and the API we cannot deploy the host key with the function. Once again: we would need to redeploy the Function every time we update the API.

I deploy the host key when I deploy the API. The API, and its authentication should be created at the same time.

You could argue that the key is deployed with the function, and the API deployment gets the key from the Function at deploy time. This hinges on the function deployment being responsible for creating all the keys that are needed for the callers. My way allows you to deploy the Function and focus on the code and leave the authorization to the APIm, which is kind of why APIm is used in the first place.

The function does not reset

If you deploy your Azure Functions in a standardized way, the key created by the API deployment will not be deleted during a redeploy. So any post release bug fix does not affect the APIs access to the function.

It will delete the key if you delete the whole function before redeploy, but why would you do that?

Deploying the Key

I use Azure Devops and ARM but I have provided a Bicep version as well. When using Azure Devops I get all the upsides of release management, which is always useful.

The ARM Template

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "FunctionAppName": {
            "type": "String"
        },
        "FunctionAppKey": {
            "type": "SecureString"
        }
    },
    "variables": { "keyName": "MyAPIName-key')]" },
    "functions": [],
    "resources": [
        {
            "type": "Microsoft.Web/sites/host/functionKeys",
            "apiVersion": "2018-11-01",
            "name": "[concat(parameters('FunctionAppName'), '/default/', variables('keyName'))]",
            "properties": {
                "name": "[variables('keyName')]",
                "value": "[parameters('FunctionAppKey')]"
            }
        }
    ],
    "outputs": {}
}

Notice that the FunctionAppKey is a SecureString. This is for keeping the value hidden during and after deployment.

The Context

Deploying the ARM using CI/CD you simply use the ever useful Deploy ARM Template step and refer to the ARM template. Here are some pointers:
– Remeber that the Function is usually in another resource group than the API manager. Update the Resource Group setting.
– Store the Function App key as a variable in the release. Either as a secret value, or as a variable group (pointing to a keyvault).
– Supply the Function App key in the “Override Template parameters” input box.

For more information about deploying values using either parameters or a Keyvault read my post “How I deploy Keyvault values”.

The conclusion

Make your APIm a first level citizen in your ALM cycle. As such, the API deploys anything relating to its access and the best way to do that is using a separate ARM template and run that template at the same time you deploy any update to the API.

Deploying the Front Door using Bicep

Bicep is a very nice language that rests on top of ARM. I have always preferred ARM to scripting as “wanted state” is a much better way of looking at anything relating to infrastructure.

But this reliance on ARM also means that whatever template definition is output by the product, that is what you must use. Case in point: Azure Front Door.

Circular reference

A circular reference in an age old problem in coding. You cannot use circual references because, logically, it does not work. It is a series of references where the last object references the first, resulting in a closed loop.

Within Azure Front Door (or AFD) we have this problem, a lot. Within AFD it refers to itself, and if you add a Rules Engine (RE), that refers to the AFD, that refers to the RE.

If you do like I do: Setup something the first time using the portal, and the extract the ARM using the Export Template, or Insert Resource in VS Code to get the Bicep Code directly, you will find that the generated ARM contains a lot of errors due to circular references.

We need to overcome this, and solve some other issues.

The first problem

The first problem comes about 5 rows in, when we need to define the (first) routing rule. You need to assign it an ID. This ID consist of the resource ID for the AFD that we are just about to create!

This is the first of many circular references.

The solution

I solved this, and all other circular references, by using variables and string interpolation. Simply put: Create the resource ID yourself. You know what the ID will be, because you are just about to create it.

Create two useful variables at the top of the bicep file.

var frontDoorName = 'contoso-afd-test'
var frontDoorResourceId = '{resourceGroup().id}/providers/Microsoft.Network/frontDoors/{frontDoorName}'

You must deploy the AFD within the context of a resource group, even if the AFD is a global service. You get the start of the resource ID by reffering to this, and the add the rest. The resulting frontDoorResourceId would be:

/subscriptions/GUIDHERE/resourcegroups/contoso-TEST-rg/providers/Microsoft.Network/frontdoors/contoso-afd-test

Now, we can use the frontDoorResourceId variable, every time a sub-resource needs to be defined, such as a routing rule or backend pool, as defined below.

routingRules: [
  {
    id: '{frontDoorResourceId}/RoutingRules/APIm'
    name: 'APIm'
    properties: {
      routeConfiguration: {
        forwardingProtocol: 'HttpsOnly'
        backendPool: {
          id: '{frontDoorResourceId}/BackendPools/ApimEurope${Env}'
        }
        '@odata.type': '#Microsoft.Azure.FrontDoor.Models.FrontdoorForwardingConfiguration'
      }
      ...
    }
  }
]

The second problem

The next problem is the rules engines. In the template definition it looks like you need to add another Resource ID, but there is a problem. The Rules Engines is a totally different resource type and it needs to be created separately. This means you simply cannot refer to it in the same way as with the frontDoorResourceId above.

The solution

It is very easy: Do not add the property to the AFD Definition. This property is not read only, but it will be assigned at deploy time by the Rules Engines Definition you add to the same Bicep File.

resource frontDoorRulesEngine 'Microsoft.Network/frontDoors/rulesEngines@2020-05-01' = {
  name: rulesEngineName
  parent: frontDoor
  properties: {
    rules: [
      {
        name: 'RuleNumberOne'
        priority: 0
        action: {
          routeConfigurationOverride: {
            forwardingProtocol: 'HttpsOnly'
            backendPool: {
              id: '${frontDoorResourceId}/BackendPools/FightClub'
            }
            '@odata.type': '#Microsoft.Azure.FrontDoor.Models.FrontdoorForwardingConfiguration'
          }
          requestHeaderActions: []
          responseHeaderActions: []
        }
  ...
  // Rule settings go here
  ...

      }
    ]
  }
}

You assign the AFD you are about to create as the rules engine’s parent. This will connect the AFD and the RE at deploytime.

Note the use of the frontDoorResourceId to refer to the Backend Pool.

The third (last) problem

The this is only a problem if you are using the Web Application Firewall (WAF) and you should. If you need to enable the WAF you must refer to a policy, or rather a FrontDoorWebApplicationFirewallPolicies resource. You do this when you create your frontendEndpoints.

A solution

This is not the solution, but I know it works for me. I define the WAF policies in a separate Bicep file as these tend to be separate from the Front Door. Remember, you can reuse the same policy for different WAFs.

Create a separate file with the resource definition. If you want to send a customized error message you need to run the base64 string function on it before deployment. This is much more useful than converting it to Base64 and pasting that sting into the Bicep-file.

Name the policy using a parameter.

param PolicyName string

resource defaultPolicy 'Microsoft.Network/FrontDoorWebApplicationFirewallPolicies@2020-11-01' = {
  name: PolicyName
   policySettings: {
     enabledState: 'Enabled'
     mode: 'Prevention'
     customBlockResponseStatusCode: 406
     customBlockResponseBody: base64('<html><head><title>You are blocked</title></head><body bgcolor="#FFB299"><p><h1>WAF custom response</h1>You are being blocked. If you need access, contact support with this reference{{azure-ref}}</p></body></html>')
   }
}

Send the policy name parameter to your AFD bicep and construct the reference using string interpolation. My naming below assumes that the policy is deployed in the same resource group as the AFD. If that is not your case, create a reference to that resource group in the bicep file, or update your bicep file for the policy to use an output parameter and send that to the AFD bicep deployment.

About the error message and the use of azure-ref, consult this documentation.

output policyResourceID string = myPolicy.id

This is how I constructed the resource reference in my AFD Bicep file. Note the liberal use of the frontDoorResourceId and frontDoorName variables.

frontendEndpoints: [
  {
    id: '{frontDoorResourceId}/FrontendEndpoints/{frontDoorName}-azurefd-net'
    name: '{frontDoorName}-azurefd-net'
    properties: {
      hostName: '{frontDoorName}.azurefd.net'
      sessionAffinityEnabledState: 'Enabled'
      sessionAffinityTtlSeconds: 0
      webApplicationFirewallPolicyLink: {
        id: '{resourceGroup().id}/providers/Microsoft.Network/frontdoorwebapplicationfirewallpolicies/{WAFPolicyName}'
      }
    }
  }
]

Conclusion

Deploying AFD is possible but also a bit of a pain using Bicep. This is notdue to the language, but the ARM template. AFD is a very old service that was in use by the XBOX network for years before being made available in Azure. I think this shows, and I hope for a revisional update at some point.

Still: This should NOT deter you from using Bicep (or ARM) and infrastructure as code for your AFD installations. Please avoid PowerShell and Azure Cli.

How to deploy NamedValues from KeyVault in APIm using Bicep

Introduction

I recently had a bit of a … run in … with deploying Named Values in APIm. These values are connected to a keyvault. This is a very good strategy for keeping keys and passwords centralized and secret. I have done this before, but was trying to use Bicep this time, but in doing so I got an error.

The error

This is the error: Can not find Managed Identity with provided IdentityClientId : <Principal ID goes here>

What I was trying to do

I was trying to add the Named Value and KeyVault connection using this bicep:

resource ApimNamedValue 'Microsoft.ApiManagement/service/namedValues@2021-04-01-preview' =  {
  parent: ApimInstance
  name: 'BackEndAPIKey'
  properties: {
    displayName: 'BackEndAPIKey'
    secret: true
    keyVault:{
      secretIdentifier: '${MyKeyVault.properties.vaultUri}secrets/BackEndAPIKey'
      identityClientId: APIminstance.identity.principalId
    }
  }
}

resource ApimInstance 'Microsoft.ApiManagement/service@2021-04-01-preview' existing ={
  name: 'ApimInstanceName'
}

resource MyKeyVault 'Microsoft.KeyVault/vaults@2019-09-01'  existing = {
  scope: resourceGroup(KeyVaultRGName)
  name: keyVaultName
}

I am using “references” to existing instances of a KeyVault (row 19) and an APIm (row 15).
There is just one thing wrong with it: You cannot suppliy the ClientID by submitting the PrincipalID (sometimes also called ObjectID). That is what triggers the error.

The solution

I got the solution directly from Adrian Hall (thank you).
It is down to a misunderstanding of the documentation. You are supposed to supply a ClientID if you want to use another identity than the APIm identity. If you want to use the identity of the APIm you are trying to deploy a named value for: Simply omit the property.

TLDR; remove row 9 from the Bicep code above and everything will work.

Just a short thing about access

To get the APIm instance to read values from the KeyVault, you need to add it to your Access Policies for the KeyVault. You can achieve this by adding the following Bicep code to your deployment.

// APIm reference
resource APIminstance 'Microsoft.ApiManagement/service@2021-04-01-preview' existing = {
  name: 'APImInstanceName'
  scope: resourceGroup('APImRGName')
}

// Keyvault config
resource MyKeyvault 'Microsoft.KeyVault/vaults@2021-06-01-preview' existing = {
  name: 'KeyVaultName'
}

resource KeyVaultSetting 'Microsoft.KeyVault/vaults/accessPolicies@2021-06-01-preview' = {
  name: 'add'
  parent: MyKeyvault
  properties: {
    accessPolicies: [
      {
        objectId: APIminstance.identity.principalId
        tenantId: APIminstance.identity.tenantId
        permissions: {
          'keys': []
          'secrets': [
            'list'
            'get'
          ]
          'certificates': []
        }
      }
    ]
  }
}

Disallowing use of API key as a query in APIm

Introduction

I asked on Twitter if, using API Management, there is any way of not allowing the user to send the API-key as a query parameter. The feedback and discussion made me create a blog post.

As ever, if you are looking for the solution, just scroll to the end.

The problem

TLS or HTTPs encryption

You are using an API-key to protect your API. That key is what security people calls a “shared private secret”. This means that anyone getting their hands on the key can use it, regardless of who you are and where. For most scenarios this is really good, but you do not want the key to fall into the wrong hands, you need to protect it.

You can protect it using TLS (called HTTPs) here is a good and detailed overview of the communication process. In HTTPs almost everything is protected using encryption. A very good and clear explaination of this can be found here.

Unencrypted

Encrypted

So what is the problem with sending an API-key as a query? It is encrypted, right?

Logging

The issue is not that anyone can intercept a message and look at the querystring, it is down to logging. When hosting your API (in Azure API manager or not) you might be behind a firewall, an application gateway or similar. The people running that log usually hosts the TLS endpoint, meaning they can decrypt the message. These logging tools usually surface some metadata about the message, including the querystring. The Azure Application Gateway is a great example of this.

This is an example of a call to an API. See how the subscription-key is logged?

Sending the API-key as a header is much more secure, from the logging standpoint. You should also consider sending other sensitive data such as PII as headers and not query for the same reason.

Best practise

Since the person hosting the private key (the pfx-cert) theoretically can decrypt and look at the whole message, they could access the data anyway. However this is not a built in functionallity in any Azure logging product (that I am aware of). I would consider not sending the API-key as a query, being a best practise; a should and not a must.

The solution

I added a policy to the Global policy in APIm. The Global policy is always executed before any call and I think you should consider putting more stuff in there.

<choose>
    <when condition="@(context.Request.OriginalUrl.Query.GetValueOrDefault("subscription-key","") != "")">
        <return-response>
            <set-status code="400" reason="Bad request" />
            <set-header name="content-type" exists-action="override">
                <value>application/json</value>
            </set-header>
            <set-body>{"Message":"Use the header Auth option."}</set-body>
        </return-response>
    </when>
</choose>

A flaw

This means that the call was sent all the way to APIm and was logged. However, it can create better habits in your callers from now on.

Another flaw

This solution has one obvious flaw: If someone renames the API-key query from subscription-key to something-else, then the policy will accept the incoming call, but remember this is a recommendation. A should.