/ service fabric

NGINX for Service Fabric

AspNetCore and Service Fabric, both being relatively new in the market, have become fairly stable in the last couple of months. Usually, when you create a Service Fabric application with an stateless service built atop of AspNetCore, the service would use the HttpSys server (formerly WebListener). This is because the httpsys driver present on windows is production grade and can be exposed to the Internet. In fact IIS (commonly used to host AspNetCore apps and also used by Azure Web Apps), makes use of the httpsys driver. This approach works fine without port sharing across multiple host names. There are at least two other approaches to this which I will discuss later in this post.

The need

Service Fabric is a powerful platform. Besides giving you the raw power of the virtual machine, you get fine grained control on scaling. Worth mentioning is the Actor programming model which is useful in given scenarios.
In this particular scenario I needed to host only 4 AspNetCore applications in the 5-node cluster. Since this application are self hosted they need to listen for communication on their own port. These are multi-tenant application where tenants are resolved via the host name i.e. from a base domain example.com, you'd have tenants such as:

  • tenant1.example.com
  • tenant2.example.com
  • tenant3.example.com

The same would be done for the other 3 domains: example.info, example.org, example.net

Decision Making

1) Azure App Service

First, I must say this can be handled very efficiently with Azure App Service but while the app was running, I seemed to hit some limits that forced me to move to Service Fabric. Scaling up and out on App Service did not solve the issue. Further, one of the service required to connect on-premise with an old CISCO VPN device (not in my control) which can only work with policy based VPN Gateway on Azure. Azure App Service only supports route-based VPN Gateway. No need to go any further.

2) HttpSys based server only

This requires setting up the services on different ports so as to use different certificates for the different domains. THe resulting URLs would look like:

With an exception of the first one, the other look very ugly (I know what you are thinking ... but it's a matter of perspective).

The problem with this approach, is also with handling the certificate. Service Fabric requires that you supply the certificate thumbprint in the endpoint binding policy of the application manifest. I don't like having to bind the https certificate to my code. The best I could do is to make the thumbprint a parameter that I can override. However, there is no getting around the requirement to deploy the certificate to the nodes before hand using Azure Key Vault. I can be 'real song and dance' when your certificate process requires changing the certificates every so often such as when Lets Encrypt 90-day certificates start supporting wild card certificates in January 2018.

3) Azure Application Gateway

Here, the service offered will go all the heavy lifting including load balancing SSL offloading/termination, URL rewriting. There are discussions of the same here, here, here and many more you can find on Google.

This was the best option I thought I had until I realized there is no support for wildcard domains in the http listener. There is a user voice request here.

In addition, besides paying for the instance(s), once still has to pay for data transfer incurred as indicated in the pricing foot notes. Why do you do this Microsoft?

4) Azure API Management

This would work well in this scenario. The official documentation does a good job at explaining the details. However, my biggest issue here was the pricing starting at $49.

5) Nginx

Honestly, I was really trying to avoid having to write my own solution but I was tired of searching. I came across Haishi's blob post which he did so well. You basically only need to install the NuGet package in your solution Service Fabric application project and the post install scripts will set it up as a guest executable. In the post, he also explains how to set it up as service from scratch. The latter approach is what fit my scenario because adding the same to multiple applications running in service fabric would be very rather untidy. It seemed best to have Nginx run as the only service in an application of its own. Configuring Nginx is rather painless, I got it running in less than 1 hour. In the end, my URLs looked better:

Implementation

I borrowed a lot from Haishi's blog post so I'd encourage you read it first if you haven't already done so. I added this section to just explain you need to change the implementation. The folder structure remained largely the same.

Service Code

I only added a few elements to the original implementation

internal sealed class NginxWrapper : StatelessService
{
    public NginxWrapper(StatelessServiceContext context) : base(context) { }

    protected override async Task RunAsync(CancellationToken cancellationToken)
    {
        // listen to changes in configuration
        Context.CodePackageActivationContext.ConfigurationPackageModifiedEvent += OnConfigurationPackageModified;
        ServiceEventSource.Current.ServiceMessage(Context, "Service started. Configuration changes listener registered");

        while (true)
        {
            // if cancellation has been requested, attempt shutdown gracefully but force if it fails
            if (cancellationToken.IsCancellationRequested)
            {
                ServiceEventSource.Current.ServiceMessage(Context, "Cancellation requested. Graceful shutdown started.");

                GracefulShutdown(); // attempt graceful shutdown

                // gracefully shutdown the process within 5 seconds
                for (var t = 0; t < 5; t++)
                {
                    if (!IsRunning) break; // stop looping if it is no longer running
                    await Task.Delay(TimeSpan.FromSeconds(1));
                }

                // if the process is stil running forcefully close
                if (IsRunning) KillProcesses();

                return; // nothing more todo
            }

            // ensure the process is still running incase it had been mistakenly dropped
            if (!IsRunning)
            {
                ServiceEventSource.Current.ServiceMessage(
                    Context,
                    "Either the process was manually stopped or had not be started. Starting the process ...");

                SetupAndValidatePaths();
                StartProcess();
            }

            // wait a little while before working to prevent over tasking the cpu
            await Task.Delay(TimeSpan.FromSeconds(5));
        }
    }

    private string ConfigurationFilePath { get; set; }
    private string ExecutableFilePath { get; set; }
    private string RunningProcessName { get; set; }
    private bool IsRunning => !string.IsNullOrWhiteSpace(RunningProcessName) && Process.GetProcessesByName(RunningProcessName).Any();

    private void OnConfigurationPackageModified(object sender, PackageModifiedEventArgs<ConfigurationPackage> e)
    {
        ServiceEventSource.Current.ServiceMessage(Context, "Configuration package was changed. Process shall be restarted");

        GracefulShutdown();
        KillProcesses(); // kill any pending proceses

        SetupAndValidatePaths();
        StartProcess();
    }

    private void SetupAndValidatePaths()
    {
        var configPackage = Context.CodePackageActivationContext.GetConfigurationPackageObject("Config");
        var parameters = configPackage.Settings.Sections["Nginx"].Parameters;
        var configurationFileName = parameters["ConfigurationFileName"].Value;
        var executableFileName = parameters["ExecutableFileName"].Value;

        var codePackageName = Context.CodePackageActivationContext.CodePackageName;
        ExecutableFilePath = Path.Combine(
            Context.CodePackageActivationContext.GetCodePackageObject(codePackageName).Path,
            executableFileName);

        ConfigurationFilePath = Path.Combine(
            configPackage.Path,
            configurationFileName);

        ServiceEventSource.Current.ServiceMessage(
            Context,
            "Setting up service with. {0}={1} and {2}={3}",
            nameof(ConfigurationFilePath),
            ConfigurationFilePath,
            nameof(ExecutableFilePath),
            ExecutableFilePath);

        if (!File.Exists(ExecutableFilePath))
            throw new FileNotFoundException("The specified executable file must exist", ExecutableFilePath);

        if (!File.Exists(ConfigurationFilePath))
            throw new FileNotFoundException("The specified configuration file must exist", ConfigurationFilePath);
    }

    private void KillProcesses()
    {
        if (!string.IsNullOrWhiteSpace(RunningProcessName))
        {
            var processes = Process.GetProcessesByName(RunningProcessName);
            ServiceEventSource.Current.ServiceMessage(
                Context,
                "Found {0} processes with name {1} to forcefully close.",
                processes.Length,
                RunningProcessName);

            foreach (var p in processes)
            {
                ServiceEventSource.Current.ServiceMessage(Context, "Killing process with Id: {0}", p.Id);
                p.Kill();
            }
        }
    }

    private void StartProcess() => LaunchProcess("-c " + ConfigurationFilePath);
    private void GracefulShutdown() => LaunchProcess("-c " + ConfigurationFilePath + " -s quit");
    private void LaunchProcess(string arguments)
    {
        var process = new Process
        {
            StartInfo = new ProcessStartInfo
            {
                FileName = ExecutableFilePath,
                WorkingDirectory = Path.GetDirectoryName(ExecutableFilePath),
                UseShellExecute = false,
                Arguments = arguments,
            },
        };

        ServiceEventSource.Current.ServiceMessage(
            Context,
            "Starting process {0} at {1} with arguments {2}",
            Path.GetFileName(process.StartInfo.FileName),
            process.StartInfo.FileName,
            process.StartInfo.Arguments);

        process.Start();
        RunningProcessName = process.ProcessName;

        ServiceEventSource.Current.ServiceMessage(Context, "Service started with name: {0}", RunningProcessName);
    }
}

Usage

I did not want to repeat the same process setting up Nginx on another cluster, I prepared a package to allow reuse. The only thing that changes in the upstream ports (a.k.a. backend ports), certificates and host names. Therefore the only file I change is the nginx.conf file.

You can download the application package here

Extract the file and use the three simple scripts (Deploy, Undeploy, and Upgrade). Using these scripts you can deploy the pre-compiled package to your cluster. You must first call Connect-ServiceFabricCluster if you are not already connected.

To deploy the application:

.\Deploy.ps1

Before deployment you should edit the configuration file named nginx.conf in the Config folder of the service package names NginxWrapperPkg. Further details on configuration can be found here.

Disclaimer

  • The code/package provided is to be used at your own risk and there are no guarantees whatsoever
  • A managed service is preferable to this approach due to patching etc.
  • Nginx configuration provided has not been hardened, which is a task the user should undertake at their own pace and time.