Cloud Pub/Sub component allows messages to be published to a topic on GCP. Other applications can subscribe to these topics to receive said messages. The end result is fast, reliable, secure communication between GCP services and applications.
To publish messages from Matillion ETL, you must first ensure Topics are available to publish to (and ideally, relevant applications are subscribed to these topics). To view, edit and create topics, visit the Topics GCP page while logged into your GCP account and project.
Official help for starting with Cloud Pub/Sub can be found here.
|Name||Text||The descriptive name for the component.|
|Project||Text||The name of the project to use. Must be affiliated with the GCP account connected to Matillion ETL.|
|Topic||Select||The name of the topic to publish to. Matillion ETL will automatically gather topics available to your project. To view and create topics, navigate to Pub/Sub→Topics from the main upper-left menu on the GCP dashboard or follow this link.|
|Message||Text||The contents of the message.|
|Attributes||Parameter||Key-value pairs that can be included with the message to be later used by the subscribed application. These may commonly be data regarding the message itself.|
This example will demonstrate how to set up a Topic on GCP, submit a message to it and then check the message. To begin, ensure you are logged into the GCP Console and on the relevant project then visit the Cloud Pub/Sub Topic list.
Click 'Create Topic' and give it a name when prompted - we have called our example Topic 'docspub'.
Select your new topic from the list. From your Topic's page, click 'Create Subscription' and give the Subscription a name - we have called our example Subscription 'docsub'. Ensure the Delivery Type is set to 'Pull' before clicking 'Create' to complete the Subscription.
Now over in Matillion ETL, we have created an Orchestration job with the Cloud Pub/Sub component.
The component is set up as shown below. We enter the project name (the one used while creating the Topic) and the Topic name we just created. Now we can add some data to the message - in this case, a short piece of text we can use to verify our message has been delivered correctly.
Now the job is run and the message is sent to the Topic. In order to pull data from the topic, we need to use the Cloud SDK tool as described here.
When on the Cloud SDK, we use the following line to check our subscription:
gcloud beta pubsub subscriptions docsub
As can be seen, the message is now pulled from the Topic and printed to the screen, confirming its success.