Skip to content
BluAge Documentation

Jobs & Steps

This page explains how to put in place the configuration of your Batch modeling, i.e. your Jobs and Steps declarations.

Introduction

Jobs and steps are the main configuration components in Batch Modeling, allowing you to encapsulate an entire batch process. In your UML model, they are both represented by Activities and have to be defined in special packages in order to be scanned by the BluAge generation process.

Here is an example of Job and Steps declarations :

BatchConfig_Intro

The following general rules apply to all Jobs and Steps in your UML model :

  • Jobs have to be stereotyped JOB and located in a package stereotyped PK_JOB;
  • Steps have to be stereotyped STEP (except for Tasklets which have their own stereotypes) and located in a package stereotyped PK_STEP;
  • PK_JOB and PK_STEP packages must be located somewhere under the root PK_TARGET package.

The creation of a Job Activity or a Step Activity can be achieved by right-clicking on your newly created PK_JOB or PK_STEP package and selecting the Activity Diagram entry of the Create Diagram menu :

BatchConfig_AD

Jobs Modeling

The Job goal is to declare all steps and the order in which to execute them for a single batch process.

Job Components

The following steps allow to describe the Job process :

  • Start by creating an Activity Initial Node;
  • For each step you have created in the PK_STEP package and want to insert in this Job, simply drag-and-drop the corresponding Step Activity Diagram in your Job Activity Diagram. Connect each newly created Call Behavior Action using Control Flows in the order you want to execute them;
  • To mark the end of the Job, connect the last Call Behavior Action to an Activity Final.

Here is a typical example of a Job Activity Diagram :

BatchConfig_Job

The following configuration can also be added to your JOB through its tagged values :

Tagged value Description
abstract (boolean) Deprecated.
parent (String) Deprecated.
record (int) Deprecated.
restartable (boolean) Allows to prevent restart execution of the Job even if it has failed. Default is true.

Conditional Flows

Conditional flows allow to trigger different steps depending on conditions. It can be seen as an if-batch equivalent, and is modeled the same way as a standard if, using Decision Nodes and Merge Nodes (shown below).

The condition on which to choose which next step to execute has to be stored in the guard of each Control Flow departing the Decision Node and is based on the ExitStatus of the previous step.

Here is an example of a Job Conditional Flow, triggering the step2 if the first step has succeeded or step3 otherwise :

BatchConfig_ForkJoin BatchConfig_ConditionalFlow

Parallel Flows

Parallel processing allow to execute multiple steps at the same time and can be useful for complex and time-consuming processes. It is modeled using Fork Nodes and Join Nodes.

Here is an example of a Job Parallel Flow, triggering step2 and step3 at the same time, and step4 at end when both step2 and step3 has finished :

BatchConfig_ForkJoin BatchConfig_ParallelFlow

Job Parameters

Job parameters can be added to your Job Activity Diagram and will be stored in the Job Context in order to be accessed in your whole process. Filling this value is the task of the BluAge Forward configuration and is not handled here, but the parameter declaration has to be modeled through the creation of parameters in the Activity Diagram, as demonstrated below :

BatchConfig_JobParams

SubJobs

Jobs can be run inside other jobs as nested jobs. You can achieve that in two steps :

  1. First, create the nested job.Subjobs must be located in a package stereotyped PK_JOB, but it is very important that Subjobs must not be stereotyped JOB;
  2. Then, drag-and-drop your newly created Subjob Activity Diagram in your main Job Activity Diagram as you would do with a regular step.

Here is an example of a Main Job having a regular Step and a call to a Subjob :

BatchConfig_SubJob1 BatchConfig_SubJob2

Steps Modeling

The Step is a process unit which encapsulates an independent, sequential phase of a batch job.

Step Components

The following elements can be found in a Step Activity Diagram :

Component Description Presence requirement
PreProcessor Allows to execute code before entering the processor, for initialization purpose for instance. Optional.
Reader Represents the Spring Batch ItemReader. Allows to read data from multiple sources such as databases, flat files, ebcdic files,... Required if no Processor is present in the Step. Otherwise, optional.
Processor Represents the Spring Batch ItemProcessor. Allows to handle the data retrieved by the Reader and/or to initialize the data to be handled by the Writer. Required if no Reader is present in the Step, otherwise optional.
Writer Represents the Spring Batch ItemWriter. Allows to write data to multiple sources such as databases, flat files, ebcdic files,.... Optional.
PostProcessor Allows to execute code after the processor, for cleaning purpose for instance. Optional.

Here is a typical example of a Step Activity Diagrams. The Reader, represented through a Call Operation Action, is shown in red, as well as the Writer, shown in blue. PreProcessors, Processes and PostProcessors are modeled as Operations, referenced through Call Behavior Actions, and shown in green.

BatchConfig_Step

NOTE : Each Component is explained in the Batch Components page. We are here covering the configuration process putting them together in the step.

The following configuration can also be added to your Step through its tagged values :

Tagged value Description
abstract (boolean) Deprecated.
allowStartIfComplete (boolean) Whether or not any step with a status of 'COMPLETED' is skipped during normal processing a restarted job. Default is false.
commitInterval (int) Specifies he number of items to be processed before the transaction is committed. Default is 1000.
isolation (ISOLATION) Deprecated.
isStandalone (boolean) Deprecated.
noRollbackExceptionsClasses (Enum) Deprecated.
parent (STEP) Deprecated.
propagation (boolean) Deprecated.
readerTransactionalQueue (boolean) Deprecated.
record (int) Deprecated.
restartable (boolean) Deprecated.
retryableExceptionsClasses (Enum) Deprecated.
retryLimit (int) Deprecated.
skipLimit (int) Deprecated.
skippableExceptionsClasses (Enum) Deprecated.
startLimit (int) Deprecated.
timeout (int) Deprecated.

Step Input & Output Types (SpringBoot Batch)

The Java configuration generated by the BluAge SpringBoot Batch stack forces the step to provide an input type and an output type.

The following rules are followed to compute the Step input type :

Present Step Components Step input type
Processor and Reader Processor input parameter type.
For lists, filling the list element type through the TemplateObject stereotype is strongly advised.
Processor only Object.
Reader only Depends on the Reader type, usually provided through one of the tagged values of the configured Reader.
Fallback is Object if no configuration has been found.

The following rules are followed to compute the Step output type :

Present Step Components Step output type
Processor (with or without Writer) Processor return parameter type.
For lists, filling the list element type through the TemplateObject stereotype is strongly advised.
For Steps without writer, the Processor can have a void return type.
Writer only For WriteTextFile / WriteTemplateLine, first Writer parameter type. Otherwise, Step input type.
No Processor nor Writer Step input type.

Partitioned Steps

Partitioning Steps is supported by the BluAge Batch Modeling. You can configure such a Step by adding the Partitioned stereotype to it, alongside the STEP stereotype.

The following tagged values can be configured for your Partitioned Step :

Tagged value (Type) Description
gridSize (int) Specifies the requests number limit for a single step to the task executor.
implementation (String) Specifies the fully qualified name of your Partitioner. This file has to be hosted by the BluAge Forward project as a specific file, and should contain an implementation of the Spring Partitioner interface, which will create the ExecutionContexts.

You can pass arguments to your Partitioned Step Implementation by creating properties in your Activity. Following properties types are supported :

  • String properties, where you can specify the String in the default value of the property;
  • Object properties, where you can specify the fully qualified name of the Object to instanciate in the default value of the property;
  • Dynamic string properties, where you can specify the name of the dynamic property in the default value of the property using the pattern $sp([propertyName]). The value of the property stored in the BluAge Forward properties files configuration will then be injected.

Here is an example of a Partitioned Step declaration with an example of each described property type :

BatchConfig_PartitionedStep1

Multi-resources Partitioned Steps

Multi-resources partitioning is a sub-case of the previous Partitioned configuration, allowing to work on multiple input files and associate their file names with execution context keys.

To enable Multi-resources partitioning, simply activate the tagged value isMultiResource of the Partitioned stereotype put on your Step. You are then allowed to pass a comma-separated list of input files in the Forward configuration.

NOTE : Passing arguments is not supported in Multi-resources Partitioned Step.

Tasklets Modeling

A Step can also be declared as a Tasklet, allowing specific data handlings which are are not covered by a Read / Process / Write combination.

Simple Tasklet (SpringBoot Batch)

The first way to model such a Tasklet is by providing the implementation of the method to be executed by the Tasklet using a process operation. Once your process operation has been created, simply reference it using a Call Behavior Action in your Step Activity Diagram, and apply the Tasklet stereotype to this Call Behavior Action, as shown in the following example :

BatchConfig_SimpleTasklet BatchConfig_SimpleTasklet

Special Tasklets

Many dedicated operations don't need a user-provided implementation and are directly provided by the BluAge Forward process. These dedicated Tasklets don't need to be stereotyped STEP, as each supported special Tasklet has its own stereotype to describe it. Moreover, no Activity Diagram is needed under the Tasklet Activity, since all configuration is made through the describing stereotype and its tagged values.
A special Tasklet still needs to be located under a PK_STEP package.

Here is an example of some BluAge-provided Tasklets declarations :

BatchConfig_Tasklet

Command Line Tasklet

The STEP_COMMAND_LINE stereotype is used to declare your Step as a Command Line Tasklet, which allows you to execute a system command.

The following tagged values can be configured for your STEP_COMMAND_LINE :

Tagged value (Type) Description
timeout (int) Specifies the time limit in ms for the command line to execute.
Default is 5000.

NOTE : The command itself is provided by the BluAge Forward process through a properties file entry.

Custom Tasklet

You can implement a Custom Tasklet by stereotyping your Step with the CUSTOM_STEP stereotype.

The following tagged values can be configured for your CUSTOM_STEP :

Tagged value (Type) Description
implementation (String) Specifies the fully qualified name of your Tasklet. This file has to be hosted by the BluAge Forward project as a specific file, and should contain an implementation of the Spring Tasklet interface.

You can pass arguments to your Custom Step by creating properties in your Activity. Following properties types are supported :

  • String properties, where you can specify the String in the default value of the property;
  • Object properties, where you can specify the fully qualified name of the Object to instanciate in the default value of the property;
  • Dynamic string properties, where you can specify the name of the dynamic property in the default value of the property using the pattern $sp([propertyName]). The value of the property stored in the BluAge Forward properties files configuration will then be injected.

Here is an example of a CUSTOM_STEP declaration with an example of each described property type :

BatchConfig_CustomStep

Email Tasklet

You can implement an Email Tasklet by stereotyping your Step with the EMAIL_STEP stereotype, which allows you to configure and send emails.

The following tagged values can be configured for your EMAIL_STEP :

Tagged value (Type) Description
bcc (String) Specifies the comma-separated list of hidden recipients in copy (blind copy carbon).
cc (String) Specifies the comma-separated list of recipients in copy (copy carbon).
encoding (String) Specifies the encoding used for the mail (Charset name).
Default is default charset.
from (String) Specifies the sender mail address.
host (String) Specifies the SMTP host name.
inputEncoding (String) Specifies the encoding used when reading the mail body Resource (Charset name).
Default is default charset.
password (String) Specifies the password to be used if authentication is required.
port (String) Specifies the SMTP host port.
Default port is 25.
subject (String) Specifies the subject of the mail.
to (String) Specifies the receiver mail address.
username (String) Specifies the username to be used if authentication is required.
useSTARTTLS (String) Whether or not to enable the SMTP startTLS for encrypted connexion.
Default value is false.

Keep in mind that those values can also be configured by the BluAge Forward process through properties file entries, for more dynamic modifications. You generally don't want to hard-code them in your UML Model unless you are certain they won't change throughout your whole project modernization.

The tagged values are prioritized over the property file entries.

Empty File Check Tasklet

You can implement an Empty File Check Tasklet by stereotyping your Step with the STEP_EMPTY_FILE_CHECK stereotype, which allows you to check whether a specified file exists and is not empty or not.

The target file is configured by the BluAge Forward process through properties file entries, hence no other configuration happens in the UML Model. This way, you can dynamically change the specified file between successive runs and reuse the same UML Activity in your model in multiple steps even if the target checked file differs.

File Operations Tasklets

You can implement File Operations Tasklets by stereotyping your Step with the FILE_OPERATION_STEP stereotype, which allows to perform several tasks on files such as copying, deleting, merging or resetting.

The following tagged values can be configured for your FILE_OPERATION_STEP :

Tagged value (Type) Description
mode (FileOperationMode) COPY : Copy one or multiple files to a directory.
DELETE : Delete one or multiple files.
MERGE : Append one or multiple files to a target file.
MERGE_COPY : Append one or multiple files to a target file, overwriting existing content.
RESET : Reset the content of one or multiple files.
strict (boolean) Whether or not the existence of each provided file must be verified and, if so, an error must be thrown.
Default is true.

Source and target files are configured by the BluAge Forward process through properties file entries.

FTP Tasklets

You can implement FTP Tasklets, which allows you to send (PUT) / receive (GET) files to/from a remote directory using the FTP protocol.

FTP GET

The FTP GET Tasklet is declared by adding the FTP_GET_STEP to your Step.

The following tagged values can be configured for your FTP_GET_STEP :

Tagged value (Type) Description
autoCreateLocalDirectory (Boolean) Whether or not to automatically create the local directory during the process.
Default is true.
clientMode (Integer) Whether or not to use an Active (0) or a Passive (2) local data connection mode.
Default is 0.
deleteLocalFiles (Boolean) Whether or not to delete local files before executing the process.
Default is true.
downloadFileAttempts (Integer) Specifies the number of attempts to make before giving up, if retry is enabled.
Default is 12.
fileType (Integer) Specifies the file type defined by org.apache.commons.net.ftp.FTP constants (ASCII_FILE_TYPE, BINARY_FILE_TYPE, EBCDIC_FILE_TYPE or LOCAL_FILE_TYPE).
Default is 0.
parent (FTP_GET_STEP) Deprecated.
retryIfNotFound (Boolean) Whether or not to retry the process if the first attempt did not succeed.
Default is false.
retryIntervalMilliseconds (Integer) Specifies the interval (in ms) before each retry attempt.
Default is 300000.

Local directory, remote directory, filename pattern as well as FTP session connection information are configured by the BluAge Forward process through properties file entries.

FTP PUT

The FTP PUT Tasklet is declared by adding the FTP_PUT_STEP to your Step.

The following tagged values can be configured for your FTP_PUT_STEP :

Tagged value (Type) Description
clientMode (Integer) Whether or not to use an Active (0) or a Passive (2) local data connection mode.
Default is 0.
fileType (Integer) Specifies the file type defined by org.apache.commons.net.ftp.FTP constants (ASCII_FILE_TYPE, BINARY_FILE_TYPE, EBCDIC_FILE_TYPE or LOCAL_FILE_TYPE).
Default is 0.
parent (FTP_GET_STEP) Deprecated.

Filename and FTP session connection information are configured by the BluAge Forward process through properties file entries.

Report Tasklet

You can implement Report Tasklets by stereotyping your Step with the REPORT_STEP stereotype, which allows to generate Jasper reports in multiple formats from a DataSource.

The following tagged values can be configured for your REPORT_STEP :

Tagged value (Type) Description
ciParamColumnKey (String) Deprecated.
ciParamColumnValue (String) Deprecated.
ciParamTable (String) Deprecated.
format (EnumReportType) PDF, XLS, HTML, CSV or TEXT.
reportImplementation (EnumReportImpl) Deprecated.

Report file and output file are configured by the BluAge Forward process through properties file entries.

You can also pass parameters to Jasper by creating parameters in your Step Activity. Each parameter name will be used as the parameter name to be passed to Jasper and the corresponding value is filled by a property file entry by the BluAge Forward process.

Here is an example of a REPORT_STEP with a parameter called parameter :

BatchConfig_ReportTasklet

Sort Tasklet

You can implement a Sort Tasklet by stereotyping your Step with the SORT_STEP stereotype. This tasklet is used to create file stort steps, with a syntax close to the DFSORT on z/OS with utilises already implemented. In addition to files, it is also possible to select lines to keep and to define transformations.

The following tagged values can be configured for your SORT_STEP :

Tagged value (Type) Description
charset (String) Specifies the source files encoding (see below).
headerSize (int) Specifies the size of the header in source files. Those lines won't be sorted nor transformed but will be kept as header in the target file.
include (String) Specifies a string describing lines to include (see below).
inrec (String) Specifies a string describing a transformation to be applied before the sort itself (see below).
isVariableBlock (boolean)
legacyMode (boolean)
omit (String) Specifies a string describing lines to exclude (see below).
outrec (String) Specifies a string describing a transformation to be applied after the sort itself (see below).
recordLength (int)
skipDuplicates (boolean) Whether or not to ignore lines with the same position (only the first one is kept) (see below).
sort (String) Specifies a string describing the order to apply to the sort, using DFSORT syntax (see below).
sortCharset (String) Specifies the encoding to use suring the sort of strings. The order can be different depending on this encoding (see below).
sum (String) Specifies a string describing fields to sum when different lines have the same position in the order defined by the sort tagged value (see below).

Sort

If the tagged value sort is filled, lines are sorted using an order described by the DFSORT syntax. It is possible to fill this value by different ways :

  • {fields}
  • '(' {fields} ')'
  • 'FORMAT=' {format} ',(' {fields} ')'
  • (' {fields} '), FORMAT=' {format}

... where {fields} is a list of fields in the DFSORT syntax and {format} a default format. Supported formats for the sort tagged value are :

  • CH, for strings;
  • ZD, for numbers, eventually with the sign encoded with the first number.

Include and omit

Before the sort itself, lines to be processed are selected using the include and omit tagged values. Both tagged values are defined using the DFSORT syntax, the same way as the sort tagged value. A line is included if it is selected by the include tagged value and not selected by the omit tagged value.

In addition to CH and ZD, the SS format is also allowed. It is used to test if a string is contained inside an other one.

Inrec and outrec

inrec and outrec tagged values define transformations to apply to the selected lines respectively before and after the sort itself. These transformations are described using a list of fields, eventually surrounded with parenthesis. Other syntaxes, such as IFTHEN,... are not supported.

Sum and skipDuplicates

By default, when two lines have the same position in the order described by sort, both lines are kept, in an unspecified order. If the skipDuplicates boolean is true, only one line is kept (sum="NONE" equivalent).

Warning : depending on the specified order, different lines can have the same position. In that case, only one will be kept in an unspecified order.

In the sum tagged value is specified, lines with the same position will be summed. The value of the sum tag describes fields which have to be summed. There is only one line printed in the final result. If a value is specified for sum, the value of the skipDuplicates tag is ignored.

Charset and sortCharset

The source file encoding is specified by the charset tagged value. If no value is provided, the default encoding is the JVM default encoding. The target file encoding is defined by the encoding workflow value.

The sortCharset tagged value is used during the sort itself for string comparisons. The order of characters is not the same depending on the encoding, typically legacy encodings such as EBCDIC which puts letters before numbers, contrary to ASCII and its equivalents.