Orchestration Components

Load blob to Snowflake

How to load multiple Azure Blob CSV files to Snowflake tables using "Azure Blob Storage Load" component. Each file should be loaded to dedicated table in Snowflake. I can load...

Control char as field delimiter for Azure blob load componenet

Requirement is to use SOH - ''\u0001’' as field delimiter for loading data from azure blob using Azure blob storage load component. But seems its not possible as there a...

bash script

Hi, How to update the environment variable in a result of the bash script? ex: variablename= aws s3 presign s3://path/

python script taking much time

Hi, I was written a simple python script, The script inside the iterator, import boto3 from datetime import datetime from datetime import timedelta bucket = 'bucket' context.updateVariable('syandication_s3_uri', _uri) print syandication_s3_uri...

XML files

Hi All! Any good examples of how to read xml files? We have XML files with correspondent XSD files in GCP Storage. I would like to fetch this data and...

import data from Google Datastore into BigQuery

is there a way to extract data from Google Datastore into BigQuery with Matillion?

Load CSV to BQ with unknown columns

Hi, I quite often need to load CSV file from Cloud Storage to BigQuery with uknown number and type of columns. So in other words I need to create table...

Cannot find Schema Copy

I am working in 1.34.3 (build 162) of Matillion. We have Matillion set up to work with BigQuery, I want to move an entire data set to another dataset, but...

Running Bash script to move files in GS

Hi All, Im trying to run a bash script that basically just moves files from a folder to another within the same bucket. I do this with gsutil within a...

Decide order to iterate files

Hi! Im iterating through a bunch of files in Google Storage. This works fine but I would like to do it in a certain order. The file names contain dates...