generate Amazon S3 pre-signed URL programmatically

[david631]

I want to use ServiceNow scripts to get a pre-signed URL from Amazon S3 storage. The AWS JavaScript SDK can be used for this, and I read the Community post describing how this SDK can be imported into ServiceNow; however that method looks extremely fragile. I'd prefer presigning a URL without the SDK, if possible. Has anyone done something like this before?

UPDATE

ServiceNow does not have the functions to generate signatures in AWS Signature Version 4. Version 4 requires a signing key that is derived from your secret access key by a series of hash-based message authentication codes (HMACs). GlideCertificateEncryption can return the HMACs in base64 format, but the AWS signing key requires an HMAC-SHA256 function that returns output in binary format:

Use the digest (binary format) for the key derivation. Most languages have functions to compute either a binary format hash, commonly called a digest, or a hex-encoded hash, called a hexdigest. The key derivation requires that you use a binary-formatted digest.

AWS provides an example of creating the necessary binary format digest in Python (amazon.com, sigv4-signed-request-examples.html😞

hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()

There seems to be a way of using the CryptoJS library to produce an equivalent to this using some custom functions for converting from a WordArray to byte array found here: https://stackoverflow.com/questions/29432506/how-to-get-digest-representation-of-cryptojs-hmacsha256.... 

Unfortunately, I've never been able to get CryptoJS working in ServiceNow Kingston as shown in Sift API and Request Signature Generation. I tested the solution from killswitch1111 on Cannot access SncAuthentication from application scope and it failed to get CryptoJS working, even after I pulled the correct version of CryptoJS from crypto-js on Google Code Archive.

The only way I can find to generate this signature is to stand up an entirely new service outside of ServiceNow that does nothing but sign S3 URLs in response to GET requests from ServiceNow. I was very much hoping to avoid that, but I can see no other option.

[Akash Rajput]

Hi Devid,

Instead on Crypto JS, I got SHA256 adn SHA512 working in SN using other lib. If you want I can send it out to you?

[david631]

Go ahead and post it. I'm interested to see what you're doing.

[bhavna6]

Can you please post the settings in SN?

[david631]

I got this working in ServiceNow using AWSRESTRequestSigningUtil instead of CryptoJS. I packaged the solution as a script include, so you can use it to create presigned S3 URLs using this syntax:

var urlUtil = new AWSPresignedURLUtil();
var presignedUrl = urlUtil.getPresignedUrl(
    'yeOldeCreds',
    604800,
    'us-east-1',
    'mfbucket420',
    'setup-32bit-1.6.3.1054.zip',
    'GET'
);

The script itself reads as follows.

var AWSPresignedURLUtil = Class.create();

AWSPresignedURLUtil.prototype = {
	/**
	 * This sets up all necessary variables for this class to do work. Note: the methods for reading key values aren't working right yet.
	 */
    initialize: function() {
		this.log.push({'version':this.version});
		this.log.push({'endpoint':this.endpoint});
		this.log.push({'queryStringAlgorithm':this.queryStringAlgorithm});

		if (this.signingUtil instanceof AWSRESTRequestSigningUtil) {
			this.log.push({'signingUtil':this.signingUtil});
		} else {
			var error = new Error('new AWSRESTRequestSigningUtil() failed');
			this.log.push({'error':error});
			throw error;
		}
	},


	/**
	 * This contains a list of objects describing the current status of the of the class instance. The list is best converted to a string via JSON.stringify(logList).
	 */
	log: [],


	/**
	 * This is the Amazon AWS algorithm used in the final presigned URL.
	 */
	queryStringAlgorithm: 'AWS4-HMAC-SHA256',


	/**
	 * This string identifies the Amazon AWS host for the presigned URL.
	 */
	endpoint: 'https://s3.amazonaws.com',


	/**
	 * This is the Semantic Versioning 2.0.0 number for this object (see https://semver.org/).
	 */
	version: '1.0.0',


	/**
	 * AWSRESTRequestSigningUtil is used to calculate the signature of the presigned URL.
	 */
	signingUtil: new AWSRESTRequestSigningUtil(),


	/**
	 * This returns a new pre-signed S3 URL suitable for sharing objects, per docs.aws.amazon.com "sigv4-create-canonical-request.html":
	 *
	 * Task 1: Create a Canonical Request for Signature Version 4
	 * Task 2: Create a String to Sign for Signature Version 4
	 * Task 3: Calculate the Signature for AWS Signature Version 4
	 * Task 4: Add the Signature to the HTTP Request
	 *
	 * @param {string} credentialName identifies the aws_credentials record which contains the access key and secret key that will be used for signing the URL.
	 * @param {number} expires provides the time period, in seconds, for which the generated presigned URL is valid. 86400 = 24 hours. 604800 (7 days is the max).
	 * @param {string} regionName (e.g., "us-east-1") is the Amazon S3 region defining the scope for which the presigned URL signature is valid and in which the bucket is hosted.
	 * @param {string} bucketName (e.g., "myreasonablebucket") is the Amazon S3 bucket containing the object which the pre-signed URL should get.
	 * @param {string} uri (e.g., "setup-32bit-1.6.3.1054.zip") is everything in the Amazon S3 object link after ".com/" to the question mark character ("?") that begins the query string, or to the end, if no query string exists.
	 * @param {string} httpRequestMethod only supports "GET" right now for allowing users to download an object from the given location, but it may support "PUT" in future to allow uploads as well
	 */
	getPresignedUrl: function(credentialName, expires, regionName, bucketName, uri, httpRequestMethod) {
		// the date and time are used in several places throughout the signing process
		var dateTimeString = this.signingUtil.getDateTimeStringForSigningV4();
		this.log.push({'dateTimeString':dateTimeString});

		var amazonawsService = this.signingUtil.getService(this.endpoint);
		this.log.push({'amazonawsService':amazonawsService});

		// the headers list is used in both the query string and the canonical request string
		var headersList = [
			{
				'name':'host',
				'value': bucketName + '.' + amazonawsService + '.amazonaws.com'
			}
		];
		this.log.push({'headersList':headersList});

		// the query string and cannical URI are used in the canonical request and the pre-signed URL
		var canonicalUri = '/' + uri;
		this.log.push({'canonicalUri':canonicalUri});

		var credential = this.getCredential(
			credentialName,
			dateTimeString.substring(0,8),
			regionName,
			amazonawsService
		);

		var signedHeadersString =
			this.signingUtil.getSignedHeadersString(headersList);

		var queryStringList = [
			{'name':'X-Amz-Algorithm','value':this.queryStringAlgorithm},
			{'name':'X-Amz-Credential','value':credential},
			{'name':'X-Amz-Date','value':dateTimeString},
			{'name':'X-Amz-Expires','value':expires},
			{'name':'X-Amz-SignedHeaders','value':signedHeadersString},
		];
		this.log.push({'queryStringList':queryStringList});

		// "Create a Canonical Request" (docs.aws.amazon.com)
		var canonicalRequest =
			this.getCanonicalRequest(
				httpRequestMethod,
				canonicalUri,
				queryStringList,
				headersList,
				'UNSIGNED-PAYLOAD'
			);
		this.log.push({'canonicalRequest':canonicalRequest});

		// "Create a String to Sign" (docs.aws.amazon.com)
		var stringToSign = this.getStringToSign(
			this.queryStringAlgorithm,
			dateTimeString,
			canonicalRequest
		);
		this.log.push({'stringToSign':stringToSign});

		// "Calculate the Signature for AWS Signature Version 4" (docs.aws.amazon.com)
		var signature = this.getSignature(
			stringToSign,
			dateTimeString.substring(0,8),
			credentialName
		);
		this.log.push({'signature':signature});

		// "Add the Signature to the HTTP Request" (docs.aws.amazon.com)
		var canonicalQueryString = this.getCanonicalQueryString(queryStringList);
		canonicalQueryString += '&X-Amz-Signature=' + signature;
		this.log.push({'canonicalQueryString':canonicalQueryString});

		var presignedUrl =
			'https://' + bucketName + '.' + amazonawsService +
			'.amazonaws.com' + canonicalUri + '?' + canonicalQueryString;

		this.log.push({'presignedUrl':presignedUrl});

		return presignedUrl;
	},

	/**
	 * This function sorts and concatenates a given 'name'-"value" list of parameters according to the rules specified by AWS for pre-signing S3 URLs. See amazon.com, "sigv4-create-canonical-request.html".
	 *
	 * Example return value: "X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIDEXAMPLE/20150830/us-east-1/iam/aws4_request"
	 *
	 * @param {list} queryStringList (e.g., [{'name':"X-Amz-Credential","value":"AKIDEXAMPLE/20150830/us-east-1/iam/aws4_request"},{'name':"X-Amz-Algorithm","value":"AWS4-HMAC-SHA256"}]) is an un-sorted list of query parameters
     */
	getCanonicalQueryString: function(queryStringList) {
		var canonicalQueryString = '';
		var queryObject = {};

		// URI-encode each parameter name and value
		for (var i = 0; i < queryStringList.length; i++) {
			queryObject[encodeURIComponent(queryStringList[i].name)] = encodeURIComponent(queryStringList[i].value);
		}

		// Sort the parameter names by character code point in ascending order.
		var queryObjectKeys = this.signingUtil.getKeys(queryObject).sort(); // JS sort() default sort order is according to string Unicode code points.

		// For each parameter, append the URI-encoded parameter name, followed by the equals sign character (=), followed by the URI-encoded parameter value.
		// Append the ampersand character (&) after each parameter value, except for the last value in the list.
		for (var j = 0; j < queryObjectKeys.length; j++) {
			// Use an empty string for parameters that have no value.
			var value = queryObject[queryObjectKeys[j]];
			if (gs.nil(value)) { value = ''; }
			canonicalQueryString += queryObjectKeys[j] + '=' + value + '&';
		}

		// return without the trailing ampersand
		return canonicalQueryString.substring(0,canonicalQueryString.length-1);
	},

	/**
	 * This function returns a string which provides scope (AWS region and service) for which the signature is valid in a pre-signed AWS S3 URL. This value must match the scope in the signature of these URLs. See amazon.com, sigv4-query-string-auth.html.
	 *
	 * @param {string} credentialName identifies the aws_credentials record which contains the access key and secret key that will be used for signing the URL.
	 * @param {string} dateString (e.g,. "20130721") should be formatted as yyyymmdd
	 * @param {string} regionname (e.g., "us-east-1") corresponds with one of the AWS S3 region identifiers on "AWS Regions and Endpoints - Amazon Web Services"
	 * @param {string} service (e.g., "s3") names the AWS service for which the credential is valid
	 */
	getCredential: function(credentialName, dateString, regionName, service) {
		var accessKey = this.getAccessKey(credentialName); // "AKIAIOSFODNN7EXAMPLE"

		return (
			accessKey + '/' +
			dateString + '/' +
			regionName + '/' +
			service +
			'/aws4_request'
		);
	},

	/**
	 * Task 1: Create a Canonical Request for Signature Version 4 as described on amazon.com, "sigv4-create-canonical-request.html". Note: this function uses "UNSIGNED-PAYLOAD" instead of a payload hash in the Canonical Request because a presigned URL doesn't know the payload content. See amazon.com, "sigv4-query-string-auth.html".
	 *
	 * @param {string} httpRequestMethod is GET, POST, etc.
	 * @param {string} canonicalUri (e.g., '/setup-32bit-1.6.3.1054.zip') is everything in the fully-qualified URL of the S3 object from the ending forward slash of the HTTP host to the question mark character ("?") that begins the query string parameters (if any).
	 * @param {string} queryStringList (e.g., [{'name':'X-Amz-Algorithm','value':this.queryStringAlgorithm},{'name':'X-Amz-Credential','value':credential}]) is a list of name-value objects specifying the query parameters
	 * @param {object} headersList (e.g., {'name':'host','value': 'mfbucket420.s3.amazonaws.com'}) is a list of objects each with exactly two properties ('name' and "value") that are used to create a "signed headers list" (e.g., "content-type;host") and a "canonical headers list" (e.g., "content-type:application/x-www-form-urlencoded; charset=utf-8\nhost:iam.amazonaws.com\nx-amz-date:20150830T123600Z\n"). See amazon.com, "sigv4-create-canonical-request.html".
	 * @param {string} payload is always 'UNSIGNED-PAYLOAD' for pre-signed S3 URLs
	 */
	getCanonicalRequest: function(httpRequestMethod, canonicalUri, queryStringList, headersList, payload) {
		var canonicalQueryString = this.getCanonicalQueryString(queryStringList);
		this.log.push({"canonicalQueryString":canonicalQueryString});

		var canonicalHeadersString = this.signingUtil.getCanonicalHeadersString(headersList);
		this.log.push({"canonicalHeadersString":canonicalHeadersString});

		var signedHeadersString = this.signingUtil.getSignedHeadersString(headersList);
		this.log.push({"signedHeadersString":signedHeadersString});

		var canonicalRequest =
			httpRequestMethod + '\n' +
			canonicalUri + '\n' +
			canonicalQueryString + '\n' +
			canonicalHeadersString + '\n' +
			signedHeadersString + '\n' +
			payload;

		return canonicalRequest;
	},

	/**
	 * Task 2: Create a String to Sign for Signature Version 4 (amazon.com, sigv4-create-string-to-sign.html)
	 *
	 * @param {string} algorithm (e.g., 'AWS4-HMAC-SHA256') identifies the key-hash message authentication code (HMAC) cryptographic algorithm AWS should use to validate the string
	 * @param {string} requestDateTime (e.g., '20150830T123600Z') must follow the ISO 8601 standard, and must be formatted with the "yyyyMMddTHHmmssZ" format. (see amazon.com, sigv4-query-string-auth.html).
	 * @param {string} canonicalRequest includes information from your request in a standardized (canonical) format (see, amazon.com, sigv4-create-canonical-request.html).
	 */
	getStringToSign: function(algorithm, requestDateTime, canonicalRequest) {
		// includes the date, the region you are targeting, the service you are requesting, and a termination string ("aws4_request") in lowercase characters
		// 20150830/us-east-1/iam/aws4_request
		var credentialScopeString = this.signingUtil.getCredentialScopeString(
			requestDateTime.substring(0,8), this.endpoint
		);

		// f536975d06c0309214f805bb90ccff089219ecd68b2577efef23edd43b7e1a59
		var hashedCanonicalRequest =
			this.signingUtil.getHashedPayloadString(
				canonicalRequest, 'SHA256'
			);

		var stringToSign =
			algorithm + "\n" +
			requestDateTime + "\n" +
			credentialScopeString + "\n" +
			hashedCanonicalRequest;

		return stringToSign;
	},


	/**
	 * This is used to return the GlideRecord corresponding with the given name in the "aws_credentials" table. This record is suitable for retrieving the secret key and access key strings.
	 *
	 * @param {string} name identifies the aws_credentials record to retrieve as a GlideRecord object.
	 */
	getAwsCredentialsGlideRecord: function(name) {
		var result;
		var record = new GlideRecord('aws_credentials');
		record.addQuery('name',name);
		record.query();
		if (record.hasNext()) {
			result = record.next();
		} else {
			var error = new Error('no aws_credentials name matches "' + name + '"');
			this.log.push({'error':error});
			throw error;
		}
		return result;
	},


	/**
	 * This returns the access key id string for the database record in the ServiceNow table "aws_credentials" matching the given name string. This access key id is used to create presigned Amazon S3 URLs.
	 *
	 * @param {string} name identifies the aws_credentials record from which to pull the access key id.
	 */
	getAccessKey: function(name) {
		var result;
		var record = this.getAwsCredentialsGlideRecord(name);
		var result = 'AKIAIOSFODNN7EXAMPLE'; // replace with your access key
		if (gs.nil(result)) {
			var error = new Error(
				'aws_credentials named "' + name + '" found, but access_key missing'
			);
			this.log.push({'error':error});
			throw error;
		}
		return result;
	},


	/**
	 * This returns the secret key string for the database record in the ServiceNow table "aws_credentials" matching the given name string. This secret key is used to derive the signing key in Amazon S3 pre-signed URLs.
	 *
	 * @param {string} name identifies the aws_credentials record from which to pull the secret key.
	 */
	getSecretKey: function(name) {
		var result;
		var record = this.getAwsCredentialsGlideRecord(name);
		var key = record.getValue('secret_key');
		result = 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY'; // replace with your secret key

		if (gs.nil(result)) {
			var error = new Error(
				'aws_credentials named "' + name + '" found, but secret_key missing'
			);
			this.log.push({'error':error});
			throw error;
		}
		return result;
	},


	/**
	 * This returns an AWS signature per "Task 3: Calculate the Signature for AWS Signature Version 4" on https://docs.aws.amazon.com. This function wouldn't normally be called by humans.
	 *
	 * @param {string} stringToSign (e.g., "AWS4-HMAC-SHA256\n20180927T163240Z\n20180927/us-east-1/s3/aws4_request\n5ee99ae38eefa1770d95cb1dc2b2c4a6b983a859117cd6a1daf79a690635980e") includes meta information about your request and about the canonical request that you created in Task 1: Create a Canonical Request for Signature Version 4. (Source: amazon.com, sigv4-create-string-to-sign.html)
	 * @param {string} dateString (e.g., "20180927") is an eight-digit string where the first four digits indicate the year, and the last four indicate the month and day, in that order.
	 * @param {string} credentialName identifies the aws_credentials record which contains the secret key string that will be used for signing.
	 */
	getSignature: function(stringToSign, dateString, credentialName) {
		var secretKey = this.getSecretKey(credentialName);
		this.log.push({'secretKey.substring(0,8)':secretKey.substring(0,8)});

		return this.signingUtil.getSignatureVersion4(
			this.endpoint, stringToSign, secretKey, dateString, 'HmacSHA256'
		);
	},


    type: 'AWSPresignedURLUtil'
};