Grid Iterator Component
Loop over values of a simple sequence.
This component implements a simple loop over the rows of a Grid Variable. It enables you to run an attached component multiple times, each time with different values set on any number of variables. Those variable can be referenced from the attached component.
To attach the iterator to another component, use the blue output connector and link to the desired component. To detach, right click on the attached component and select 'Disconnect from Iterator'.
If you need to iterate more than one component, put them into a separate orchestration or transformation job and use a Run Transformation or Run Orchestration component attached to the iterator. In this way, you can run an entire ETL flow multiple times, once for each row of Grid values.
Note: All iterator components are limited to 5000 iterations.
|Name||Text||The descriptive name for the component.|
|Grid Variable||Select||Choose an existing Grid Variable to iterate. (You may need to define a Grid Variable first. See the Grid Variables documentation for more information.)|
|Grid Variable Mapping||Grid Column||Enter the Grid column name.|
|Variable Name||Choose an existing variable name. (You may need to define them first in the Project → Edit Environment Variables screen.)|
|Break on Failure||Select||No - Attempt to run the attached component for each iteration, regardless of success or failure.
Yes - If the attached component does not run successfully, fail immediately.
Note: If a failure occurs during any iteration, the failure link is followed. This parameter controls whether it is followed immediately or after all iterations have been attempted.
Note: This is only available in Sequential mode. When running with concurrency, all iterations will be attempted.
|Concurrency||Select||Sequential - Iterations are done in sequence, waiting for each to complete before starting the next. This is the default.
Concurrent - Iterations are run concurrently. This requires all "Variables to Iterate" to be defined as Copied variables, so that each iteration gets its own copy of the variable isolated from the same variable being used by other concurrent executions.
Note: The maximum concurrency is 8 times the number of processors on your cloud instance. For example: An instance with 2 processors has a maximum concurrency of 16.
|Record Values in Task History||Select||If set to Yes: For each Variable chosen in "Variables to Iterate", record the value in the in-client Task History.|
This component makes the following values available to export into variables:
|Iteration Attempted||The number of iterations that this component attempts to reach (Max Iterations parameter).|
|Iteration Generated||The number of iterations that have been initiated. Iterators terminate after failure so this number will be the successful iterations plus any potential failure.|
|Iteration Successful||The number of iterations successfully performed. This is the max iteration number, minus failures and any unattempted iterations (since the component terminates after failure).|
Attached: At the bottom of this page you will find an attached job that can be downloaded and imported into your Matillion ETL instance. Simply select the job with the prefix appropriate for your Platform (RS, BQ, SF for Redshift, BigQuery, Snowflake respectively). This job demonstrates how to iterate over a date-range. It uses a Python script to populate a Grid variable with a list of dates before using that with a Grid Iterator component.