Running a Windows MID in Fargate with persistent storage

[Will Hallam]

DISCLAIMER: this example is an academic exercise with a pinch of "what's possible" added in.  It comes with no support or warranty, implied or explicit.  Caveat emptor!

Intro

Here's an example of how you can run a Windows MID in AWS Fargate by applying a quick and dirty FSx Windows share.  It would need significant polishing to be viable for actual use, in the security and functional areas, but it illustrates the feasibility of such an approach to service elastic MID capacity needs.

The files and scripts used by this example can be found in the following repo: https://github.com/willhallam-sn/example-mid-win-fargate

Image

To build my image, I chose to use Azure DevOps because they provide a Windows runtime which includes Docker support.  To use AWS CodeBuild I would have to create my own build image as their pre-built Windows environment doesn't include Docker.  So I created an ADO pipeline which I linked to my Git repo.  I added a variable set named "docker-vars", in which I stored the DockerHub credentials to be used in any DockerHub interactions.  Due to the overall Windows-friendly Azure environment, that was all it took to get my image built.

The primary tweaks I applied to the standard recipe were in the Dockerfile and asset\init.ps1 files.

I had to add "-force" to the CMD directive so the config.xml would be regenerated on every launch; failing to do that made the encrypted password inaccessible on successive launches.  I removed the USER directive because the "ContainerUser" account was unable to create a symlink to the persistent storage.

Given the academic nature of the exercise, I chose to leverage a Windows share to house the entire "agent" file hierarchy, modifying the init.ps1 script so it would mount the share, detect a brand new MID launch, copy the original "agent" folder onto the share if need be, and then symlink the "agent" directory in the container image over to the share.  (NOTE: the permissions and structure of the share would need a bunch of sprucing up if someone were going to deploy this for real).

I chose to upload my image to DockerHub for each of portability, but the pipeline could be adjusted pretty easily to push the image to any public or private repo.

Directory

Once I had my image, I went over to AWS to set things up for my serverless Windows MID.  I started by creating a directory instance under Directory Service (I chose an AWS-managed AD).  I added a "mid.server" user to my new directory, setting aside the credentials to be stored in AWS Secrets Manager.

File Share

With a directory instance created, I went to FSx for Windows and provisioned a filesystem to serve as my MID server's persistent storage.  I made the "mid.server" user the owner.  I verified I could mount the share to a test EC2 instance running Windows -- this proves to be a handy debugging tool for the MID container.

Secret

For credential storage and retrieval, plus some config parameters, I set up an object in AWS Secrets Manager with the following key/value pairs:

1.  username: MID user login name
2.  password: password MID user on ServiceNow and within AD
3. instance: ServiceNow instance URL
4. share_drive: drive letter for FSx share within the Windows container, e.g. "R:"
5. share_path: IP-based UNC path to the FSx share, e.g. "\\<share ip>\share"
6. share_user: user name from my AD user

Cluster

I created an ECS virtual cluster under "Elastic Container Service", for use in hosting Fargate workloads.

Task

I defined an ECS Task Definition to launch my custom Windows MID image, adding the following environment variables, which are all used within the image to configure and launch the MID container:

• MID_SERVER_NAME should contain the MID server name you want to appear in your instance
• MID_INSTANCE_USERNAME - gets its value from secret key 1 above
• MID_INSTANCE_PASSWORD - gets its value from secret key 2 above
• MID_INSTANCE_URL - gets its value from secret key 3 above
• SHARE_DRIVE - gets its value from secret key 4 above
• SHARE_PATH - gets its value from secret key 5 above
• SHARE_USER - gets its value from secret key 6 above
• SHARE_PASSWD - gets its value from secret key 2 above

Launch

I launched my task, using my virtual ECS cluster.  I was able to monitor its progress via the cluster Tasks tab.  Once it entered the running state, I could see it populate the agent folder structure within the FSx share and then follow its activities via the agent log file.

Closing Notes and Considerations

As I mentioned previously, this was a quick and dirty proof of concept exercise.  A bunch more work would be required to make it viable for use in a real environment.  After going through what it took to get this far, my opinion is that the juice is only worth the squeeze in vary specific use cases, e.g., where maintaining VMs as standalone MIDs or ECS nodes is extremely onerous or costly and the need for MID resources is highly elastic.  Under those circumstances, the following things should be considered/addressed at a minimum:

• Instead of copying the whole agent structure, selected files only should be stored in the persistent storage (e.g. config.xml, keystore, java security files, log files perhaps -- I did like the enhanced debug ability).
• Ideally the rest of the storage should be read only -- I'm not sure if that's even possible with Windows containers though.
• To scale this to multiple MIDs, some kind of file structure would need to be added to the share or individual shares would be needed.
• This example is not tolerant of any issue with connectivity to the FSx share -- it would have to be able to deal elegantly with a hiccup in the availability of the persistent storage (e.g. just relaunch the task).
• I tested this as a simple task, it should be set up as an ECS Service so it automatically relaunches in the event of an issue, such as the MID crashing or being shut down.
• I've not researched the official support stance from AWS regarding FSx shares mounting to Fargate containers.  This example works, but I don't know if it would be supported should I need to engage AWS to help with any issues I encounter.
• In order for the image to create a symlink, I set it to run as "ContainerAdministrator" vs. "ContainerUser".  In theory there should be a way to run as "ContainerUser", I gave up trying to make it work after several hours with no progress.
• I did not test this example in a MID upgrade scenario -- I kind of feel like MID containers shouldn't be allowed to auto-upgrade...?

AWS can make this a lot easier by adding support for persistent storage in Windows Fargate containers, but I've no insight into if/when that might occur.

[stevejarman]

Hey Will - this is great information. Thanks for taking the time to experiment on this.