API v1 - Versions
    • Dark
      Light

    API v1 - Versions

    • Dark
      Light

    Article Summary

    Overview

    This is a guide to providing details on the Version API from the family of the Group API . The Version endpoint allows users to explore and manages the versions of the Matillion ETL Transformation and Orchestration Jobs.

    Versioning supports many use cases, but perhaps the most salient is to be able to capture your development at a point in time to designate as "live" or "production", etc., and to then continue working on the Default Version.

    All projects have a default version. This is the main working version. It is possible to make changes to a non-default version you have created via Project Menu Manage Versions, but it must be in an unlocked state.

    Managed Version

    The Version API provides the resource details to run the version details within the project group in the Matillion ETL instance. By resources here, we refer to explore the version, export versions, delete version, and create a new version within the ETL Transformation and Orchestration Jobs.

    Important Information

    • This document is part of a series target="_blank">API v1 - Group and the Matillion ETL API - v1.
    • Users responsible for experimenting with Matillion ETL API services require access to the Matillion ETL instance and ought to know how to make REST API calls either employing a REST API GUI client such as Postman or employing a command-line interface like cURL.
    • Matillion ETL API endpoints require authorisation to make any REST API call, so ensure a username and password for the Matillion ETL instance is configured before making any API call.


    Version API Endpoints

    API Base URL

    http(s)://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version

    API Endpoints and Function

    Version endpoint API is available on standard REST-based APIs that uses HTTP or HTTPS request to GET, POST, and DELETE data. The Version endpoint API service is accessed through the Uniform Resource Identifier (URI). The available API endpoints are listed below:

    /tr>

    MethodPathURIFunction
    GET/namehttp://<instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/nameGet the name of the current version.
    GET/idhttp://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/idGet the id of the selected version.
    GET/exporthttp://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/exportTo export the metadata of the current version within the project.
    GET/orchestrationhttp://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/orchestrationTo retrieve the details of the Orchestration within the selected project.
    GET/transformationhttp://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/transformationTo retrieve the details of the Transformation within the selected project.
    POST/createhttp://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/createTo create a new version which is a copy of the existing version.
    POST/deletehttp://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/deleteTo delete the selected version using POST HTTP method.
    DELETE/versionNamehttp://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>To delete the selected version using DELETE HTTP method.
    PATH/job
    GET/exporthttp://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/name/<jobName>/exportTo export the data of the job within the current version.
    POST/importhttp://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/importTo import the data of the job within the current version.
    POST/deletehttp://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/name/<jobName>/deleteTo delete the selected job using POST HTTP method.
    POST/runhttp://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/name/<jobName>/run?environmentName=<environmentName> WITH POST DATA arg0Run an Orchestration Job, where variable overrides can be passed in the body.
    POST/validatehttp://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/name/<jobName>/validate?environmentName=<environmentName>To validate the Job by passing Environment name in the body.
    DELETE/jobNamehttp://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/name/<jobName>To delete the selected job using DELETE HTTP method.
    PATH/scm


    Graphical Representation

    To illustrate the Version API v1, endpoints and methods to the further, below is the graphical flow of the /version endpoint showing possible PATH, GET , POST, and DELETE options.

    Version endpoint Flow

    Please Note

    For detailed description about accessing the SCM data of the project for PATH/scm part of Version API, with some general information on the SCM and Git commands, please refer to SCM Integration.




    URL Parameters and Description

    Below is the list of endpoint parameters and their brief description:

    Parameters NameDescription
    <InstanceAddress>This is the server IP address or domain name.
    <groupName>the name of the group.
    <projectName>The name of the project within the group.
    <versionName>The name of the version within the selected project.
    <JobName>The name of the job within the selected version.
    <name>To get the name of the resource.
    <export>This allows to export the metadata of the resource.
    <delete>Delete the selected resource.
    <id>The queue id of the current resource.
    <run>Run an Orchestration Job, where variable overrides can be passed in the body.
    <validate>Validate the Orchestration Job by passing Environment name.


    Endpoints and Server Response

    This chapter describes the Version API endpoints and the Job API endpoints along with some server response examples. However, PATH/job is a part of PATH/version API as shown in the Overview section. These APIs offers REST-based web service, offering ease of use and a flexible choice of programming language. These APIs can be used to access and analyse the versions, jobs and scm within the project in the Matillion ETL instance. All the APIs listed in this chapter are available to use with GET/POST/DELETE methods to retrieve the data used to get, update, or delete the versions.

    PATH/version

    List of endpoints for the /version:

    Below is the detailed description of these endpoints with the server response examples.

    GET/name

    This example is a GET method REST API request that will provide the name of the version within the project in the instance.

    • Base URL
      http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/name
    • Server Response
      default

    GET/id

    This is to get the id for the selected version within the project using GET HTTP method.

    • Base URL
      http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/id
    • Server Response
      796

    GET/export

    To export the selected version detail from the project within the Matillion instance, provide the versionName and use the /export endpoint. This example using GET method REST API call to export the jobs (Transformation/Orchestration) details available within the selected version.

    • Base URL
      http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/export
    • Server Response
      {
      "objects": [
      {
      "name": "default",
      "description": "Default Version",
      "jobs": {
      "objects": [
      {
      "jobObject": {...
      },
      "info": {
      "id": 800,
      "name": "dim_airport_setup",
      "type": "ORCHESTRATION",
      },
      "path": []
      },
      {
      "jobObject": {...
      },
      "info": {
      "id": 831,
      "name": "dim_airports",
      "type": "TRANSFORMATION",
      },
      ],
      "version": "master",
      "environment": "redshift"
      }
      ],
      "version": "master",
      "environment": "redshift"
      }

    GET/orchestration

    To retrieve the Orchestration detail from the project, provide the versionName and use the /orchestration endpoint. This example using GET method REST API call to get the Orchestration jobs details available within the selected version of the project.

    • Base URL
      http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/orchestration
    • Server Response
      [
      {
      "id": 800,
      "name": "dim_airport_setup",
      "description": null,
      "type": "ORCHESTRATION",
      "tag": "02fc1dee-99c5-4fcf-928f-852536bfebeb"
      },
      {
      "id": 827,
      "name": "Orchestration-Job-1",
      "description": null,
      "type": "ORCHESTRATION",
      "tag": "bbc43d86-e130-46c0-9a6e-73099bff2b94"
      }
      ]

    GET/transformation

    To retrieve the Transformation detail from the project, provide the versionName and use the /transformation endpoint. This example using GET method REST API call to get the Transformation jobs details available within the selected version of the project.

    • Base URL
      http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/orchestration
    • Server Response
      [
      {
      "id": 831,
      "name": "dim_airports",
      "description": null,
      "type": "TRANSFORMATION",
      "tag": "844e3ea8-af95-4b6e-8dfa-db9150a4cdfb"
      }
      ]

    POST/create

    This is to create a new which is a copy of the existing version by using /create endpoint. This API call is a POST method API call as we will have to attach the resource data, in the body as a JSON file to import into the Matillion ETL instance.

    • Base URL
          
      http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/create
    • POST Body (JSON)
      {
      "newVersionName": "name",
      "newVersionDescription" : "This is a new version",
      "newVersionIsLocked" : true
      }
    • Server Response
      {
      "success": true,
      "msg": "",
      "id": -1
      }

    POST/delete

    This will be a POST method API call. The /delete endpoint will allow to delete the selected version from the project.

    • Base URL
      http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/delete
    • Server Response
      {
      "success": true,
      "msg": "Successfully deleted the version: default",
      "id": -1
      }

    DELETE/versionName

    To remove any resource from the list, we will use DELETE API request. In this example, we will delete a version from the project.

    Please Note

    There are often multiple ways of achieving the same task within the API as resources and methods branch out and overlap.


    • Base URL
      http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>
    • Server Response
      {
      "success": true,
      "msg": "Successfully deleted version [default]",
      "id": -1
      }

    PATH/job

    PATH/job is the part of the PATH/version, which is further combined with HTTP methods GET, POST and DELETE.

    List of endpoints associated with the PATH/job

    Below is the detailed description of these endpoints with the server response examples.

    GET/export

    This is a GET HTTP method to export the details of the Jobs available within the selected version of the project. To export any "resource" data we use /export endpoint after the "resource name". For example: /jobName

    • Base URL
      http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/name/<jobName>/export
    • Server Response
      {
      "objects": [
      {
      "jobObject": {
      "JobType": ".OrchestrationJob",
      "id": 827,
      "revision": 2,
      "created": 1594599191329,
      "components": {
      "828": {
      "id": 828,
      ....
      },
      "829": {
      "id": 829,
      ....
      },
      "info": {
      "id": 827,
      "name": "Example-Job",
      "type": "ORCHESTRATION",
      "tag": "bbc43d86-e130-46c0-9a6e-73099bff2b94"
      },
      "path": []
      }
      ],
      "version": "master",
      "environment": "redshift"
      }

    POST/import

    This will import the resource data which is exported as shown in the above example using /import endpoint. If a resource of the same name already exists, you must delete the existing resource before importing the new or you can change the name of the imported resource (Job-Imported)in the POST BODY. This API call is a POST method API call as we will have to attach the resource data, in the body as a JSON file to import into the Matillion ETL instance.

    • Base URL
          
      http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/import
    • POST Body (JSON)
      {
      "objects": [
      {
      "jobObject": {
      "JobType": ".OrchestrationJob",
      "id": 827,
      "revision": 2,
      "created": 1594599191329,
      "components": {
      "828": {
      "id": 828,
      ....
      },
      "829": {
      "id": 829,
      ....
      },
      "info": {
      "id": 827,
      "name": "Example-Job",
      "type": "ORCHESTRATION",
      "tag": "bbc43d86-e130-46c0-9a6e-73099bff2b94"
      },
      "path": []
      }
      ],
      "version": "master",
      "environment": "redshift"
      }
    • Server Response
      {
      "name": "Version",
      "statusList": [
      {
      "success": true,
      "name": "Job-Imported"
      }
      ],
      "success": true
      }
    • URL Parameters

      There is an optional parameter, onConflict, which determines what should happen if an import clashes with something that already exists. For an explanation of this parameter's use, read API Import conflicts - Explanation in Matillion ETL API - v1.

    POST/delete

    This will be a POST method API call. The /delete endpoint will allow to delete the selected job from the project.

    • Base URL
      http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/name/<jobName>/delete
    • Server Response
      {
      "success": true,
      "msg": "Successfully deleted the job: Example-Job",
      "id": -1
      }

    POST/run

    The /run endpoint will allow to run an Orchestration job within the version. This will be a POST method API call as we will have to attach the variable for the job to be queued, in the body as a JSON file to run into the Matillion ETL instance.

    • Base URL
      http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/name/<jobName>/run?environmentName=<environmentName> WITH POST DATA arg0
    • POST BODY(JSON)
      { 
      "scalarVariables": {},
      "gridVariables": {}
      }
    • Server Response
      {
      "success": true,
      "msg": "Successfully queued job dim_airport_setup",
      "id": 65659
      }

    POST/validate

    The /validate endpoint will allow to validate an Orchestration job within the version. This will be a POST method API call as we will have to pass the environment name for the job to be queued, in the body as a JSON file to run into the Matillion ETL instance.

    • Base URL
      http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/name/<jobName>/validate?environmentName=<environmentName>
    • Server Response
      {
      "success": true,
      "msg": "Successfully queued validation for Orchestration Job: [API query], for Environment: [test].  Task Id: [xxxxxxx]",
      "id": xxxxxxx
      }

    DELETE/jobName

    To remove any resource from the list, we will use DELETE API request. In this example, we will delete a job from the project.

    • Base URL
      http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/name/<jobName>
    • Server Response
      {
      "success": true,
      "msg": "Successfully deleted job [dim_airport_setup]",
      "id": -1
      }