⚠️ This post is outdated and may contain information that is no longer accurate.
← About Jeff

Using mail2bug with Azure App Service, WebJobs and VSTS to handle support mail

February 24, 2017

mail2bug is a service that allows you to create a bug from an e-mail thread simply by adding a specific recipient to the mail thread. It also keeps the bug up-to-date with information from the mail thread by adding any subsequent replies on the thread as comments to the bug. The mail2bug README has useful information if you want to learn more.

Mail2bug was not designed for the cloud - in many cases there's a server box under someone's desk running their team's on-premise mail2bug instance, or maybe it's in a lab, or on a Hyper-V server. That works for many folks but it turns out that it's easy to run this standard .NET app in the cloud, too.

Thanks to Azure App Service, the WebJobs feature of web sites, and a simple tweak to a deployment script, the changes I made to mail2bug have enabled my team to use a cloud-hosted mail2bug for almost a year now with zero maintenance cost or effort. It just keeps ticking.

In this short post I'll cover how we use WebJobs to enable automatic deployment and execution on Azure of this simple app. It is implemented in .NET and was open sourced a few years ago.

2/27/2017 Update: My pull request contributing the changes to mail2bug was accepted. Hooray!

I am hoping that this post will help the few folks I know who have been asking how this worked for my team.

What's mail2bug?

Historically mail2bug was a Microsoft internal tool that a few engineering systems teams implemented. Mail2bug does simple things:

  • Creates a Visual Studio Team Services or TFS on-prem work item from an e-mail conversation and maintains the history
  • Integrates with Exchange or Office 365 mail accounts
  • Can reply back to the start of a thread or support incident with a link to the tracking bug and also a templated mail with helpful information or your support SLA

It's essentially a "poor man's Zen Desk", not designed to do much more. On our team this is a good solution for now, though there are features I know we want to implement that would make it even better.

Here's what an auto-response mail looks like from one of our team addresses:

And here is what a Visual Studio Team Services cloud-hosted work item query view looks like, showing the active support tickets on our team:

Azure App Service and WebJobs

Introduction

App Service is a fully-managed platform for web app hosting, it makes web apps and sites extremely easy to use and deploy at scale.

Powering the infrastructure for App Service are the "Advanced Tools", or Project Kudu. The Project Kudu wiki on GitHub has a lot of great information that you just won't find elsewhere, so if you are new to this, take a look.

A feature of App Service and Kudu is a technology called WebJobs - WebJobs are cron/schedule tasks that run either continuously or triggered based on time or REST calls. It makes a lot of simple automation tasks even easier and they're quick and easy to deploy alongside a site.

Right inside the Azure portal for an app you can explore any WebJobs setup within your app service.

You can also drill into the logs for the jobs, and that opens up the Advanced Tools management side of your app.

Default .NET console app experience

Kudu generates a deployment script automatically based on the type of repository that is connected to an App Service instance.

When you deploy a .NET Framework console app, it automatically builds the project and deploys it as a continuous WebJob.

It just works! For many "lift and shift" .NET console apps, this is a nice win, and easy.

Triggered WebJob changes

In our case we like triggered WebJobs, because you can set an interval - every 2 minutes, every 10 minutes, etc., with a cron-like syntax. This also lets us rotate the password / secret used to connect to the mail account regularly, since each time the app runs, it will grab the latest rotated account password. More on that soon.

When the WebJob runs, it will store a log of exactly what happened during the run, and by default a certain set of the last N runs will be stored. This value is configurable and documented as part of the Kudu wiki.

Inside the WebJob mini portal you can see the history for a WebJob:

And then you can select any particular instance and find out the console output and other data from that run:

Nice.

Azure KeyVault secrets

To enable cloud-based deployment and still connect to an Office 365 e-mail account to check for mail and respond, we have to store a secret: the token or password to access the e-mail account.

KeyVault scenario

Azure KeyVault is the best bet for this, since we can then authorize our App Service instance through Active Directory to be able to GET the secret at runtime. We can also handle secrets rotation very easily, since each time the triggered WebJob runs, it will refresh the latest secret to use for connecting to the mail account.

This way we can store the KeyVault secret URI inside our source code, since it is not a secret, and then the app, if authorized, will be able to resolve the secret at runtime. Our use of KeyVault is mostly about preventing developer mistakes and of course to avoid a source code evil, which is storing secrets in code (gasp!).

While I won't walk through the specifics in this post, the steps are:

  • Create a new Azure Active Directory (AAD) application that the mail2bug app will use as its identity to connect to the secret store
  • Store the secret in KeyVault, this can be set using the portal, a PowerShell or CLI script, or even our team's own KeyVault management portal that we have built
  • Authorize the AAD app to be able to GET the secret(s) in the KeyVault instance we have the secret stored in

Then we just need to point the app configuration at the KeyVault secret URI and provide the client ID and secret to the app inside App Service. There is also a slightly more secure method using certificates that could be implemented, but that I have chosen to not implement at this time for simplicity sake.

Mail2Bug secrets management

The mail2bug app was built to use DPAPI - the Windows Data Protection API - to keep the secret encrypted and safe within a Windows box sitting under a desk somewhere, or in a lab.

In the cloud this is not ideal, and for App Service it just makes sense to use KeyVault instead.

The pull request I have submitted for mail2bug alters the DPAPI code paths to alternatively be able to use Azure KeyVault if the vault and secret information is available.

Continuous deployment

Using Azure App Service we have a nice story when we want to update the configuration for a mail2bug task, tweak the e-mail template that we use for autoresponse, etc.

The standard mail2bug configuration story for on-premise users is to remote desktop into a machine or otherwise remotely work with the files, restart the service or bump the scheduled task.

For my team it's even easier, as we just have a Git repo with both the mail2bug source code and our configuration files and e-mail template in it. We can just create a branch, code review the change, and then push the master branch, at which point App Service updates the app and the WebJob and the next time that the triggered WebJob runs it will have the latest configuration. This also works for updating the mail2bug app itself.

Here's the repo:

The structure is pretty basic:

  • configuration folder has our e-mail template and various mail2bug XML config files
  • mail2bug folder contains the source code for mail2bug
  • .deployment and deploy.cmd custom deployment script that Kudu will use whenever our app is deployed instead of the standard auto-generated script
  • settings.job, the triggered WebJob configuration that we want to use for our app

The custom deployment script is up in a Gist if you need it. It just builds the project and moves it into the folder App_Data/jobs/triggered/mail2bug. This is the location that kudu looks to setup WebJobs, etc.

Configuration

We need to configure Mail2Bug - nearly the same way as any other mail2bug instance, so that is left as an exercise to the reader.

We also need to configure the App Service instance.

Mail2Bug configuration files

A Mail2Bug configuration file contains info about how to respond to e-mail, where to open bugs, and other data. Here is a sample configuration file, partially redacted. They aren't pretty, but they get the job done.

<?xml version="1.0"?>
<Config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Instances>
    <InstanceConfig Name="npmjs-admin">
      <TfsServerConfig>
        <SimulationMode>false</SimulationMode>
        <CollectionUri>https://ospo.visualstudio.com/DefaultCollection/</CollectionUri>
        <ServiceIdentityUsername>vstsusername@domain.com</ServiceIdentityUsername>
        <ServiceIdentityPatKeyVaultSecret>
          <KeyVaultPath>https://yourkeyvaultaddress.vault.azure.net/secrets/mail2bug-pat</KeyVaultPath>
          <ApplicationIdEnvironmentVariableName>DIRECTORY_APPLICATION_ID</ApplicationIdEnvironmentVariableName>
          <ApplicationSecretEnvironmentVariableName>DIRECTORY_APPLICATION_SECRET</ApplicationSecretEnvironmentVariableName>
        </ServiceIdentityPatKeyVaultSecret>
        <Project>YourVSTSProjectName</Project>
        <WorkItemTemplate>Issue</WorkItemTemplate>
        <CacheQueryFile>configuration\openedissues.wiq</CacheQueryFile>
        <NamesListFieldName>Assigned To</NamesListFieldName>
      </TfsServerConfig>
      <WorkItemSettings>
        <FieldsToCache>
          <string>State</string>
          <string>Assigned To</string>
        </FieldsToCache>
        <ApplyOverridesDuringUpdate>true</ApplyOverridesDuringUpdate>
        <Mnemonics>
          <MnemonicDefinition Mnemonic="remove" Field="State" Value="Closed" />
          <MnemonicDefinition Mnemonic="resolved" Field="State" Value="Closed" />
          <MnemonicDefinition Mnemonic="close" Field="State" Value="Closed" />
        </Mnemonics>
        <ConversationIndexFieldName>Integration Build</ConversationIndexFieldName>
        <DefaultFieldValues>
          <DefaultValueDefinition Field="Assigned To" Value="username@domain.com" />
          <DefaultValueDefinition Field="Changed By" Value="##Sender" />
          <DefaultValueDefinition Field="Description" Value="##MessageBody" />
          <DefaultValueDefinition Field="Priority" Value="3" />
          <DefaultValueDefinition Field="Area Path" Value="Path\To\Store\WorkItem\In" />
          <DefaultValueDefinition Field="Iteration Path" Value="IterationPathValue" />
        </DefaultFieldValues>
        <AttachOriginalMessage>true</AttachOriginalMessage>
        <AddEmailHeaderToItem>false</AddEmailHeaderToItem>
        <DefaultAssign>Active</DefaultAssign>
      </WorkItemSettings>
      <EmailSettings>
        <Recipients>emailAddressToKeyOffOf@microsoft.com</Recipients>
        <ReplyTemplate>configuration\template.htm</ReplyTemplate>
        <CompletedFolder>foldername</CompletedFolder>
        <ErrorFolder>processing-errors</ErrorFolder>
        <EWSKeyVaultSecret>
          <KeyVaultPath>https://yourkeyvaultaddress.vault.azure.net/secrets/mailaccount-password</KeyVaultPath>
          <ApplicationIdEnvironmentVariableName>DIRECTORY_APPLICATION_ID</ApplicationIdEnvironmentVariableName>
          <ApplicationSecretEnvironmentVariableName>DIRECTORY_APPLICATION_SECRET</ApplicationSecretEnvironmentVariableName>
        </EWSKeyVaultSecret>
        <EWSMailboxAddress>mailAccount@domain.com</EWSMailboxAddress>
        <EWSUsername>mailAccount@domain.com</EWSUsername>
        <ServiceType>EWSByRecipients</ServiceType>
        <SendAckEmails>true</SendAckEmails>
        <AckEmailsRecipientsAll>true</AckEmailsRecipientsAll>
        <AppendOnlyEmailTitleRegex>.*(bug|work item)\s*#*\s*(?&lt;id&gt;\d+)</AppendOnlyEmailTitleRegex>
        <AppendOnlyEmailBodyRegex>!!!(bug|work item)\s*#*\s*(?&lt;id&gt;\d+)</AppendOnlyEmailBodyRegex>
        <ExplicitOverridesRegex>###\s*(?&lt;fieldName&gt;[^:]*):\s*(?&lt;value&gt;.*)</ExplicitOverridesRegex>
      </EmailSettings>
    </InstanceConfig>
  </Instances>
</Config>

Application Settings

The final piece of the puzzle is to give the App Service access to our AAD app's client ID and secret. This is so that it can access the vault. The mail2bug configuration includes KeyVault secret URIs, and also the name of the environment variables that will have the app's client ID and secrets available.

We set these secrets in the Azure portal. I have chosen the variable names DIRECTORY_APPLICATION_ID and DIRECTORY_APPLICATION_SECRET.

Closing

Hope this helps you to learn about App Service and WebJobs, and I also hope that the upstream project accepts my contribution at some point!

Jeff