Getting Started with cURL

Dark
Light

Getting Started with cURL

Dark
Light

Article Summary

Share feedback

Thanks for sharing your feedback!

Overview

This article gives a brief overview of testing a REST API using cURL and some examples with different HTTP operations from Matillion ETL API. Basically, cURL is a command line tool for transfering data via URLs or endpoints (where c stands for "Client" and it indicates curl works with URL's.

When it comes to REST APIs, we can use Postman as a GUI (graphical user interface) and cURL as a CLI (command line interface) to do the same tasks.

In this article we will be using cURL to demonstrate GET, POST and DELETE request calls against a Matillion REST API.

Important Information and Links

Users responsible for experimenting with Matillion ETL API services require permissions to access to the Matillion ETL instance.
Check if you have cURL installed in your system using the command line prompt (also called your Terminal) curl --version. If cURL is not there is your system, please download and install the cURL.
For more information on Matillion ETL API v1, please refer to our support documentation target="_blank">Matillion ETL API - v1.
For more information on making REST API request using GUI client Postman, please refer to our support documentation target="_blank">Getting Started with Postman.

cURL CLI Arguments

Below are a few cURL arguments we will be using with our requests in this article. All requests will simply be curl followed by the argument and data to supply.

Curl Command	Description	Example
-H or --header	Submits the request header to the resource. Headers are common with REST API requests because the authorization is usually included in the header.	`curl -H "Content-Type: application/json" "http://www.example.com"` `curl -H "Accept: application/json" "http://www.example.com"`
-i or --include	This includes the response headers in the server response of API call.	`curl -i "http://www.example.com"`
-X or --request	This specifies the HTTP method to use with the request. If you use -d in the request, curl automatically specifies a POST method. With GET requests, including the HTTP method is optional, because GET is the default method used.	`curl -X POST -d "resource-to-update" "https://www.example.com"` `curl -X GET "https://www.example.com"`
-d or --data	Includes data to post to the URL, especially used in the POST requests. The data needs to be URL encoded . Data can also be passed in the request body.	`curl -X POST -d "data-to-post" "https://www.example.com"`
@filename	Loads content from a file.	`curl -X POST -d @filename.json "https://www.example.com"`
-u or --user <user:password>	Specify the user name and password to use for server authentication.	`curl -u "user:password" "https://www.example.com"`
-k or --insecure	This allows curl to procees and operate the request even for the server connections otherwise considered insecure.	`curl -k "http://www.example.com"`
-o or --output <file>	This allows curl to request to download the file.	`curl -o filename.json "http://www.example.com"`

See the curl documentation for a comprehensive list of curl commands you can use.

Authentication

Matillion ETL API require authentication to make any REST API call. Matillion API uses HTTP Basic Authentication. The details of the HTTP basic authentication are beyond the scope of this document, but it essentially requires a username and password which are hashed employing a Matillion ETL instance.
In the cURL request, add -u "user:password" before URL. For example:

curl -u "<user>:<password>"  http(s)://<InstanceAddress>/rest/v1

Working with cURL GET Request

Summary

This is the default method when making HTTP calls with curl. Get requests are used to retrieve information from the URL. There will be no changes done to the endpoint. In the example, we will be retrieving a resource information, perform a GET request for that named resource endpoint.

Whenever you reach a named resource endpoint, the API will expose API metadata for that resource, including PATH, GET and POST and DELETE method options available. In the example below, the metadata would show PATH options for the Example-project.

Base URL

curl -X GET -u "<user>:<password>" -k "http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>" -H "accept: application/json"

Server Response

{
"endpoints" : [ {
"httpMethod" : "PATH",
"name" : "ProjectInstanceService",
"children" : [ {
"httpMethod" : "PATH",
"name" : "getVariable",
"description" : "Allows accessing Project Variables APIs",
"path" : "/variable",
"children" : [ {
"httpMethod" : "GET",
"name" : "exportVariables",
"description" : "Download all variables in an export container",
"path" : "/export",
"type" : "ExportContainer<VariableExport>",
"example" : "/variable/export",
"totalPath" : "/variable/export"
}, {
"httpMethod" : "POST",
"name" : "importVariableExport",
"description" : "Import a new Variable",
"path" : "/import",
"arguments" : [ {
"style" : "QUERY",
"name" : "ignoreMissingEnvironments",
"type" : "boolean",
"defaultValue" : "true"
},
......

cURL GET Request

Please Note

The Resource names are case-sensitive and must be URL encoded where appropriate (for example, when the resource name contains a space).

Working with cURL GET/export Request

Summary

In the example, we will be exporting a job named "Example-Job" within the project in the group resource, of Matillion ETL instance.It is a GET API call to export the resource detail.

Please Note

Jobs imported/exported through the API are incompatible with the in-client import/export feature. If a job is exported through the API, it must be imported through the API and vice-versa.

Base URL

curl -X GET -u "<user>:<password>" -k  "http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/name/<jobName>/export"

Server Response

{
"objects": [
{
"jobObject": {
"JobType": ".OrchestrationJob",
"id": 2126346,
"revision": 3,
"created": 1591688363574,
"timestamp": 1591688363574,
"components": {
"2126347": {
......
},
"info": {
"id": 2126346,
"name": "Example-Job",
"type": "ORCHESTRATION",
"tag": "123564748-6997-428a-b988-2eb532fd7d78"
},
"path": []
}
],
"version": "1.44.11",
"environment": "redshift"
}

In order to export the job, we need to download the file in the JSON format. To download the file you need to add -o Filename.json before the cURL URL.
After adding the argument in the URL, it would be like:

curl -o Example-Job_New.json -X GET -u "<user>:<password>" -k  "http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/name/<jobName>/export"

Please Note

The job file name is edited and renamed to Example-Job_new.json (so as to avoid a name conflict with the existing job) before being imported back into Matillion ETL.

cURL GET/export Jobs

Working with cURL POST/import Request

Summary

Now that you have an exported a job (see previous example), this time we use the API to import that "job" into a Matillion ETL instance. Note that, when importing, there is no "merge" option. If a resource of the same name already exists, you must delete the existing resource before importing the new.This will be a POST method API call as we will have to attach the project (in JSON form, as exported), in the body as a JSON file to import into the Matillion ETL instance.

In the above example, the JSON file (Example-Job_new) has already been exported and renamed to avoid any conflict. We will be using the same JSON file to import into instance.

Base URL

curl -X POST -u "<user>:<password>" -k  "http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/import" -H "Content-Type: application/json" --data @Example-Job_new.json

Server Response

{
"name" : "Jobs",
"statusList" : [ {
"success" : true,
"name" : "Example-Job_new"
} ],
"success" : true
}

The API call will create this job (Example-Job_new) in the Matillion ETL instance. Note that we have changed the job name to "Example-Job_new" in the JSON, else we could get an error for importing a job whose name already exists (Example-Job). This project can now be switched to in the Matillion ETL instance.

cURL POST/import Job in Matillion instance

API Import conflicts - Explaination

There is an optional parameter for API Import: "onConflict", which determines what should happen if an import clashes with something that already exists, e.g. a project with a given name. The options are "ERROR", "SKIP", and "OVERWRITE".

This can happen when the user tries to import: Project Groups, Projects, Versions, Jobs, Passwords, Schedules, and Environments. Each outcome is as follows:

ERROR – Sends an error back.
SKIP – Skips importing any clashes, preserving the existing object.
OVERWRITE – Removes the existing object and replaces it with what has been imported.

Examples as given below (using cURL):

curl -X POST -u api-user:api-user "https://<InstanceAddress>/rest/v1/group/import" -H "Content-Type: application/json" --data @mtln_project_grp.json

would become:

curl -X POST -u api-user:api-user "https://<InstanceAddress>/rest/v1/group/import?onConflict=ERROR" -H "Content-Type: application/json" --data @mtln_project_grp.json

curl -X POST -u api-user:api-user "https://<InstanceAddress>/rest/v1/group/import?onConflict=SKIP" -H "Content-Type: application/json" --data @mtln_project_grp.json

curl -X POST -u api-user:api-user "https://<InstanceAddress>/rest/v1/group/import?onConflict=OVERWRITE" -H "Content-Type: application/json" --data @mtln_project_grp.json

Depending on what you want to happen when you try to import something that already exists. The default is ERROR.

Working with cURL DELETE Request

summary

In the example we will be deleting an existing job named Example-Job from within the Project of the Matillion ETL instance. This is a HTTP DELETE API call request to remove a resource from within the group of resources.

Base URL

curl -X DELETE -u "<user>:<password>" -k  "http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/name/Example-Job"

Server Response

{
"success" : true,
"msg" : "Successfully deleted Job: Example-Job_new",
"id" : 2126481
}

cURL DELETE Job

URL Parameters and Description

Below is the list of endpoint parameters (used in the guide) and their brief description:

Parameters Name	Description
<InstanceAddress>	This is the server IP address or domain name.
<version>	This is the API version (Not versions created in the tool).
<endpoint>	Endpoint is the part of the API used to make a API call.
<groupName>	The name of the group created in the Maillion ETL instance.
<projectName>	Get the Project API detail created in the group by its name.
<scheduleName>	Get the schedule API by its name.
<versionName>	Get the version API detail by its name.
<jobName>	Get the job name within the project by its name.
<delete>	Delete the resource within the the available resources.

Conclusion

We looked at utilizing the bare-minimum usefulness of cURL to test Matillion API REST services. In spite of the fact that it can do much more than what has been talked about here, for our purpose, this much ought to suffice. Feel free to type curl -h on the command line to check out all the available alternatives. The API REST benefit utilized for the exhibit is accessible at API v1 Maps document on our support.

What's Next

Managing users, passwords, groups, and permissions

Table of contents

Overview
cURL CLI Arguments
Authentication
Working with cURL GET Request
Working with cURL GET/export Request
Working with cURL POST/import Request
Working with cURL DELETE Request
URL Parameters and Description
Conclusion