Create an activity that executes basic SFTP commands on a remote server.

Before you begin

Roles required: activity_creator or workflow_admin

About this task

You can create a custom activity that manages files and directories on a target host or copies a file from one SFTP server to another. The file content is streamed through a MID Server, which avoids having to store the data on the hard drive of the MID Server host machine.
Note: This activity requires the credentials of a user who can execute SFTP commands on the source host. The Copy File activity, provided in the activity pack, requires separate credentials to access the target host.

Procedure

  1. Create a custom activity.
    This action creates a custom activity using a template.
  2. After setting up general properties and creating input variables, configure the SFTP Execution Command.
    OptionDescription
    Command Type of activity to create. The choices are:
    • Copy file
    • Create directory
    • Get file list
    • Remove file or directory
    • Rename file or directory
    • Set file attributes
    Note: The fields that display on the form depend on the command you select.
    Source host Name or IP address of the server containing the files targeted by the activity.
    Source port Port number to use to communicate with the source server. The default port number is 22.
    Source file path Full path to a file (/temp/test_data_file.txt) or directory (/temp/test_dir) on a source host depending on the selected command. Field available on following commands:
    • Copy file
    • Remove file or directory
    • Set file attributes
    Source directory Path to the directory where the source files are located. Field available on following commands:
    • Create directory
    • Get file list
    Source files Names of specific source files to include in the file transfer. Separate the file names with semi-colons. This field supports a semi-colon separated list of wild-card patterns. For example: *.txt; a?cd.pdf. If this field is blank, all files are included. Field available on following command: Get file list
    Note: For information about how the values in this field affect options in other fields, see the logic explanation which follows.
    Is a directory If checked, specifies the file path is a directory. Field available on following command: Remove file or directory
    Credential tag for source Specific credential tag this activity must use to run SFTP commands on the source host.
    Credential tag for target Specific credential tag this activity must use to run SFTP commands on the target host. Field available on following command: Copy file
    Excluded files Names of specific source files to exclude from the file transfer. The activity acts on all other files found in the source directory or subfolders. Separate the file names with commas. This field supports comma-separated list of wild-card patterns. For example: *.txt, a?cd.pdf.. Field available on following command: Get file list
    Note: For information about how the values in this field affect options in other fields, see the logic explanation which follows.
    Include subfolders If checked, includes the files from subfolders in the source directory. Field available on following command: Get file list
    Note: For information about how your selection affects other fields in the form, see the logic explanation which follows.
    Target host Name or IP address of the server to which the files are being transferred. Field available on following command: Copy file
    Target port Port number to use to communicate with the target host. The default port number is 22. Field available on following command: Copy file
    Target file path Full path to a file on a target host. Field available on following commands:
    • Copy file
    • Rename file or directory
    Suffix for temporary file If a file exists on a target host, this command enables a temporary suffix to use for a file name. If this field contains a value, the activity first copies the source file to a temporary file on the target host using targetFilePath + tempFileSuffix as the name. Upon completion, the activity renames the file to the actual target file name. If this field is blank, the activity copies the source file directly to the target file and overwrites it, if it exists. Field available on following command: Copy file
    UID User ID attribute to apply to a file or directory. The UID and GID values must be set together as a pair or they are ignored. The UID and GID numbers are internal values returned by the Get File List activity. Typically, you first use the Get File List activity to return a list of files and their attributes. Then you can move a file from a source host to a target host and set the source file attributes on the target file. This flow is demonstrated in the SFTP File Transfer workflow. Field available on following command: Set file attributes
    GID Group ID attribute to apply to a file or directory. The UID and GID values must be set together as a pair or they are ignored. The UID and GID numbers are internal values returned by the Get File List activity. Typically, you first use the Get File List activity to return a list of files and their attributes. Then you can move a file from a source host to a target host and set the source file attributes on the target file. This flow is demonstrated in the SFTP File Transfer workflow. Field available on following command: Set file attributes
    Permissions File or directory permissions to set for the user and group specified. This value must be expressed as an integer, such as 16877, which defines these permissions: rwxr-xr-x. The permissions numbers are internal values returned by the Get File List activity. Typically, you first use the Get File List activity to return a list of files and their attributes. Then you can move a file from a source host to a target host and set the source file attributes on the target file. This flow is demonstrated in the SFTP File Transfer workflow. Field available on following command: Set file attributes
    Access timestamp Override the timestamp when the file or directory was last accessed. The access and modification timestamps must be set together as a pair. Field available on following command: Set file attributes
    Modification timestamp Override the timestamp when the file or directory was last modified. The access and modification timestamps must be set together as a pair. Field available on following command: Set file attributes
    Size in bytes Size of a file, expressed in bytes. Field available on following command: Set file attributes
    Required MID Server capabilities MID Server with the appropriate MID Server capabilities for connecting to the source and target servers.
    The system uses this logic to determine which files to move from the source host:
    • If the Source files field is empty, the system selects all the files in the source directory. Otherwise, it only selects those files whose names match one of the file name patterns given in the field.
    • If the Excluded files field is empty, the system excludes nothing. Otherwise, it excludes those files whose names match one of the file name patterns given in the field.
    • The exclude rule has a higher preference than the include rule. If a file name matches one of the file name patterns in the Excluded files field, it does not get into the selection regardless of the include rule.
    • When the Include subfolders check box is cleared, the system looks only in the source directory for files to include or exclude. Otherwise, it looks in the source directory and any of its subfolders for files to include or exclude.
    Note: You can map parameter values in a test payload to variables in the Outputs tab automatically. See automap output variables.

What to do next

SFTP template execution parameters

You use execution parameters to create the input process script in the Preprocessing form.

For descriptions of the command fields, see Configure the SFTP execution command.
Note: You must use the executionParam. prefix with all variables in this table.
Table 1. SFTP execution parameters
Name Variable Type Usage
Command command String Identifies the function of this activity. The possible commands are:
  • Copy file
  • Create directory
  • Get file list
  • Remove file or directory
  • Rename file or directory
  • Set file attributes
Source source String Identifies the source host.
Source port source_port Integer Identifies the port used to communicate with the source host.
Source directory source_directory String Path to the source directory of the files to be moved.
Source files source_files String Names of the files on the source to be moved.
Excluded files excluded_files String Lists the files excluded from the operation.
Source file path source_file_path String Path to the source files to be moved.
Target host target_host String Identifies the target host.
Target port target_port Integer Identifies the port used to communicate with the target host.
Target directory target_directory String Full path to the target directory on the target host.
Target file path target_file_path String Full path to a file or directory on the source host or on the target host. For example, this value is used in the Rename File or Directory activity in the activity pack.
Temp file postfix temp_file_postfix String

Temporary file name extension used by the Copy File activity when moving a file. If this field contains an extension, the source file is copied to a temporary file using target file name + temp_file_postfix, before being renamed to the actual target file. If this field is blank, the source file is copied directly to the target file.
Is directory is_directory Boolean Indicates whether the given file path is a directory.
Include subfolders include_subfolders Boolean Indicates whether the system looks into subfolders of the source directory for files to include and exclude.
File attribute uid file_attribute_uid Integer User ID associated with a file or directory.
File attribute gid file_attribute_gid Integer Group ID associated with a file or directory.
File attribute permissions file_attribute_permissions Integer File or directory permissions for the user and group specified.
File attribute atime file_attribute_atime Integer Access time stamp from the file attributes.
File attribute mtime file_attribute_mtime Integer Modification time stamp from the file attributes.
File attribute size file_attribute_size Integer Size of the file, in bytes.
Source credential tag source_credential_tag String Credential aliases for Orchestration activities used to run the command on the source host.
Target credential tag target_credential_tag String Credential tag used to run the command on the target host.
MidCapabilities midCapabilities String (comma separated) List of required MID Server capabilities.

SFTP post-processing parameters and payload parsing

Use these parameters to create a post-processing script, payload parsing, and tagging.

Table 2. SFTP post-processing parameters
Name Variable Type Usage
Output output String Contains output data returned by the query.
EccSysId eccSysId String Contains the Reference ID associated with the ECC Queue input message.
ErrorMessages errorMessages String Contains the error messages returned from the query. If no error messages are returned, this value is null.
Tags tags Hashmap of tag values returned from the SSH command Contains the tags used to extract output using the SSH commands. The tag output is delimited by double percentage signs, as in %%tagname%% … %%. Set up the command using the following format:
%%tagname1%%
output1 line 1
output1 line 2
…
%%

%%tagname2%%
output2 line 1
output2 line 2
…
%%
The tags returned are JavaScript hashmap objects in which each key is prefixed with tag appended with the tagname.
{"__text__": "", 
"tagtagname1":"output1 line1\noutput1 line2\n", 
"tagtagname2":"output2 line1\noutput2 line2\n"}

Configure the SFTP execution command

Use the input variables you created to configure the command that Orchestration executes on the SFTP target.

Before you begin

Create the input variables you need in the Inputs form before you can advance to the Execution Command stage.

Role required: activity_creator, admin

Note: You can test the JDBC connection between the MID Server and the target without having to run the activity in a workflow context. For details, see test template outputs.

Procedure

  1. Drag variables from the list of inputs and drop them into command fields.
    The system formats the variable in the proper syntax for the command.
    Figure 1. SFTP execution command
    In this example, the Create directory command displays in the Command field.
  2. Complete the fields shown in the table.
    Table 3. SFTP command fields
    Field Description Command
    Command Type of activity to create. The choices are:
    • Copy file
    • Create directory
    • Get file list
    • Remove file or directory
    • Rename file or directory
    • Set file attributes
    Note: The fields displayed on the form depend on the command selected.
    		 All
    Source host Name or IP address of the server containing the files targeted by the activity. All
    Source port Port number to use to communicate with the source server. The default port number is 22. All
    Source file path Full path to a file on a source host.
    • Copy file
    • Remove file or directory
    • Set file attributes
    Source directory Path to the directory where the source files are located.
    • Create directory
    • Get file list
    Source files Names of specific source files to include in the management action. Separate the file names in this list with semi-colons. This field supports the use of wild cards. If this field is blank, all files are included. For information about how the values in this field are affected by options in other fields, see the section below the table. Get file list
    Is a directory Check box that determines if the specified file path is a directory. Remove file or directory
    Credential tag for source Specific Credential aliases for Orchestration activities this activity must use to run SFTP commands on the source host. All
    Credential tag for target Specific credential tag this activity must use to run SFTP commands on the target host. Copy file
    Excluded files Names of specific source files to exclude from the management action. The activity acts on all other files found in the source directory or subfolders. Separate the file names in this list with semi-colons. This field supports the use of wild cards. For information about how the values in this field are affected by options in other fields, see the section below the table. Get file list
    Include subfolders Check box to manage the files from subfolders in the source directory. For information about how your selection affects other fields in the form, see the section below the table. Get file list
    Target host Name or IP address of the server to which the files are being transferred. Copy file
    Target port Port number to use to communicate with the target host. The default port number is 22. Copy file
    Target file path Full path to a file on a target host.
    • Copy file
    • Rename file or directory
    Suffix for temporary file Temporary suffix to use for a file name if the file already exists on a target host. If this field contains a value, the activity deletes the duplicate target file if it exists, and then copies the source file to a temporary file using targetFilePath + tempFileSuffix as the name. Upon completion, the activity renames the file to the actual target file name. If this field is blank, the activity copies the source file directly to the target file and overwrites it, if it already exists. Copy file
    UID User ID attribute to apply to a file or directory. The UID and GID values must be set together as a pair or they are ignored. Set file attributes
    GID Group ID attribute to apply to a file or directory. The UID and GID values must be set together as a pair or they are ignored. Set file attributes
    Permissions File or directory permissions to set for the user and group specified. This value must be expressed as an integer, such as 16877, which defines these permissions: rwxr-xr-x. The permissions number is an internal value returned by the Get File List activity. Typically, you first use the Get File List activity to return a list of files and their attributes. Then you can move a file from a source host to a target host and set the source file attributes on the target file. This is demonstrated in the SFTP File Transfer workflow. Set file attributes
    Access timestamp Override the timestamp when the file or directory was last accessed. The access and modification timestamps must be set together as a pair. Set file attributes
    Modification timestamp Override the timestamp when the file or directory was last modified. The access and modification timestamps must be set together as a pair. Set file attributes
    Size in bytes Size of a file, expressed in bytes. Set file attributes
    Required MID Server capabilities MID Server with the appropriate MID Server capabilities for connecting to the source and target servers. By default, the system selects an SSH MID Server. All
    The system uses this logic to determine which files to move from the source host:
    • If the Source files field is empty, the system select all the files in the source directory. Otherwise, it only selects those files whose names match one of the file name patterns given in the field.
    • If the Excluded files field is empty, the system excludes nothing. Otherwise, it excludes those files whose names match one of the file name patterns given in the field.
    • The exclude rule has a higher preference than the include rule. A file whose name matches one of the file name patterns in the Excluded files field does not get into the selection, even though it would be selected by the include rule.
    • When the Include subfolders check box is cleared, the system looks only in the source directory for files to include or exclude. Otherwise, it looks in the source directory and any of its subfolders for files to include or exclude.
  3. Click Save.
  4. Click Continue to advance to the Outputs stage.