Orchestration Components

import data from Google Datastore into BigQuery

is there a way to extract data from Google Datastore into BigQuery with Matillion?

Load CSV to BQ with unknown columns

Hi, I quite often need to load CSV file from Cloud Storage to BigQuery with uknown number and type of columns. So in other words I need to create table...

Cannot find Schema Copy

I am working in 1.34.3 (build 162) of Matillion. We have Matillion set up to work with BigQuery, I want to move an entire data set to another dataset, but...

Running Bash script to move files in GS

Hi All, Im trying to run a bash script that basically just moves files from a folder to another within the same bucket. I do this with gsutil within a...

Decide order to iterate files

Hi! Im iterating through a bunch of files in Google Storage. This works fine but I would like to do it in a certain order. The file names contain dates...

Restrict access to google drive by permissions

Greetings. We want to transfer a few spreadsheets from Google Drive to Big Query using Matillion. However we want to restrict access to people that have been assigned with the...

Passing additional columns to Cloud Storage Load component

Is there any way to pass additional data to the BiqQuery table while using Cloud Storage Load Component? For example, consider a CSV file with two columns: COL_1 and COL_2,...

Python Script in GCP

I'm trying to use a python script to change a filename in a GCP bucket. I'm running some test code just to confirm that my python script component has proper...

Metadata

Hello, In S3 bucket option, I can see metadata column with three different variables ( metadata$filename, Matadata$file_row_number, current_filestamp) but I am not sure how to retrieve that information into an...

Access parameter manage password

Hi All, i'm already created a variable inside matillion manage password, how do we access it's value? i'm tried ${variable_name} , create local variable, create global variable, nothing works Thanks...