Skip to content

Transfers API Reference

PostgresToGCSOperator

Bases: BaseOperator

Operator to copy data from PostgreSQL to Google Cloud Storage.

Exports data from PostgreSQL using SQL queries and uploads the results to GCS in CSV, Parquet, or JSONL format. Supports automatic format switching when problematic characters are detected.

Parameters:

Name Type Description Default

postgres_conn_id

str

Reference to a specific Postgres hook.

required

sql

str

The SQL query to be executed.

required

where

Optional[str]

Optional WHERE clause to append to the SQL query. Mutually exclusive with providing a complete WHERE in sql.

None

bucket

str

The GCS bucket to upload to.

required

filename

str

The GCS filename to upload to (including a folder).

required

gcp_conn_id

str

The connection ID used to connect to Google Cloud. Defaults to 'google_cloud_default'.

'google_cloud_default'

export_format

str

The format to export the data to. Supported formats are 'csv', 'parquet', and 'jsonl'. Defaults to 'csv'.

CSV

schema_filename

Optional[str]

If set, a GCS file with the schema will be uploaded.

None

schema_overrides

Optional[Dict[str, str]]

BigQuery type overrides by top-level field name. pandas_chunksize: The number of rows to include in each chunk processed by pandas.

None

shard_size_mb

Optional[float]

Optional maximum size for each uploaded shard. When set, the returned filename is a wildcard matching all generated shards.

None

use_copy

bool

If True, use PostgreSQL COPY command for streaming.

False

use_temp_file

bool

If True, use a temporary file instead of streaming.

False

csv_kwargs

Optional[dict]

Arguments to pass to pandas.DataFrame.to_csv().

None

parquet_kwargs

Optional[dict]

Arguments to pass to DataFrame.to_parquet().

None

schema_detection_mode

str

Mode for detecting column types. Defaults to 'postgres'.

POSTGRES

sampling_size

int

Number of rows to sample for JSON detection. Defaults to 100.

100

sampling_threshold

float

Threshold for JSON column detection. Defaults to 0.9.

0.9

auto_switch_format

bool

If True, detect problematic characters (quotes, newlines) in text columns and automatically switch to JSONL format. If False, use requested format with proper CSV quoting options. Defaults to True.

True

dry_run

bool

If True, simulate the operation without writing data.

False

GCSToPostgresOperator

Bases: BaseOperator

Operator to load data from Google Cloud Storage to PostgreSQL.

Downloads files from GCS and loads them into PostgreSQL tables. Supports CSV, JSONL, Parquet, and Avro formats with automatic type coercion.

Parameters:

Name Type Description Default

target_table_name

str

PostgreSQL table name (schema.table or just table).

required

bucket_name

str

GCS bucket name.

required

object_name

str

GCS object path/filename.

required

postgres_conn_id

str

PostgreSQL connection ID.

required

gcp_conn_id

str

GCP connection ID. Defaults to 'google_cloud_default'.

required

conflict_columns

Columns for upsert conflict resolution. Optional.

None

replace

If True, replace table content. If False, append. Defaults to False.

False

grant_table_privileges

bool

If True, grant ALL privileges to PUBLIC. Defaults to True.

True

create_if_empty

bool

If True, create table when source is empty. Defaults to False.

False

create_if_missing

bool

If True, create table if it doesn't exist. Defaults to False.

False

partition_column

Optional[str]

Column name for partitioning. When set, creates one partition per unique value in the column. Requires replace=False. Defaults to None.

None

source_schema

Optional[dict]

Source column types for type coercion. Optional.

None

postgres_type_overrides

Optional[dict]

Per-column PostgreSQL type hints used when creating destination tables. Optional.

None

audit_cols_to_exclude

Columns to exclude from updates during upsert.

None

dry_run

bool

If True, simulate the operation without writing data.

False

PostgresToBigQueryOperator

Bases: BaseOperator

Transfer data from PostgreSQL to BigQuery via GCS staging.

This operator extracts data from PostgreSQL, uploads to a temporary GCS location, then loads into BigQuery. Includes source validation, automatic schema detection, and cleanup.

Parameters:

Name Type Description Default

postgres_conn_id

str

PostgreSQL connection ID.

required

sql

str | None

SQL query to extract data. Mutually exclusive with source_project_dataset_table.

''

where

Optional[str]

WHERE clause to filter data. Only applies when source_project_dataset_table is used. Mutually exclusive with sql.

None

source_project_dataset_table

str | None

Source table (postgres.schema.table). Used if sql is not provided.

None

destination_project_dataset_table

str

BigQuery destination table (project.dataset.table or dataset.table).

required

gcp_conn_id

str

GCP connection ID. Defaults to 'google_cloud_default'.

'google_cloud_default'

gcs_bucket

str

GCS bucket for temporary staging.

required

gcs_temp_path

Optional[str]

GCS path for temp files. Auto-generated if not provided.

None

export_format

str

Export format: 'csv' or 'jsonl'. Defaults to 'csv'.

CSV

schema_filename

Optional[str]

GCS path for BigQuery schema JSON file. Optional.

None

schema_overrides

Optional[dict[str, str]]

BigQuery type overrides by top-level field name.

None

pandas_chunksize

int

Rows per chunk for large exports. Defaults to 100000.

100000

use_copy

bool

If True, use PostgreSQL COPY for streaming. Defaults to False.

False

use_temp_file

bool

If True, use temporary file instead of streaming.

False

check_source_exists

bool

If True, validate source has data before transfer.

True

source_table_check_sql

Optional[str]

Custom SQL for source validation. Optional.

None

write_disposition

str

BigQuery write disposition. Defaults to 'WRITE_TRUNCATE'.

WRITE_TRUNCATE

create_disposition

str

BigQuery create disposition. Defaults to 'CREATE_IF_NEEDED'.

CREATE_IF_NEEDED

emit_asset

bool

If True, emit Airflow asset for lineage. Defaults to True.

True

cleanup_temp_files

bool

If True, delete GCS temp files after load. Defaults to True.

True

partition_by

Optional[str]

BigQuery partition column. Optional.

None

partition_type

str

Partition type: 'DAY', 'HOUR', 'MONTH', 'YEAR'. Defaults to 'DAY'.

DAY

cluster_fields

Optional[List[str]]

BigQuery clustering columns. Optional.

None

dataset_location

str

BigQuery dataset location. Defaults to 'us-central1'.

'us-central1'

create_if_empty

bool

If True, create empty table when source is empty.

False

auto_switch_format

bool

If True, switch to JSONL for JSON columns.

True

dry_run

bool

If True, simulate the operation without writing data.

False

BigQueryToPostgresOperator

Bases: BaseOperator

Transfer data from BigQuery to PostgreSQL via GCS staging.

This operator extracts data from BigQuery to a temporary GCS location, then loads into PostgreSQL. Includes source validation, automatic format detection, and cleanup.

Parameters:

Name Type Description Default

source_project_dataset_table

Optional[str]

BigQuery source table (project.dataset.table or dataset.table). Required if sql is not provided.

None

sql

Optional[str]

SQL query to export. Mutually exclusive with source_project_dataset_table. If provided, the query result is exported to PostgreSQL.

None

where

Optional[str]

WHERE clause to filter data. Only applies when source_project_dataset_table is used. Mutually exclusive with sql.

None

postgres_conn_id

str

PostgreSQL connection ID.

required

destination_table

str

PostgreSQL destination table (schema.table).

required

gcp_conn_id

str

GCP connection ID. Defaults to 'google_cloud_default'.

'google_cloud_default'

gcs_bucket

str

GCS bucket for temporary staging.

required

gcs_temp_path

Optional[str]

GCS path for temp files. Auto-generated if not provided.

None

export_format

str

Export format: 'parquet', 'csv', 'jsonl', or 'avro'. Defaults to 'parquet'.

PARQUET

auto_detect_json_columns

bool

If True, detect JSON columns from BigQuery. Defaults to True.

True

check_source_exists

bool

If True, validate source has data before transfer.

True

source_table_check_sql

Optional[str]

Custom SQL for source validation. Optional.

None

conflict_columns

Optional[List[str]]

Columns for upsert conflict resolution. Optional.

None

replace

bool

If True, replace table content. If False and conflict_columns provided, perform upsert. Defaults to True.

True

create_if_empty

bool

If True, create table when source is empty. Defaults to False.

False

create_if_missing

bool

If True, create table if it doesn't exist. Defaults to False.

False

partition_column

Optional[str]

Column name for partitioning. When set, creates one partition per unique value in the column. Requires replace=False. Defaults to None.

None

postgres_type_overrides

Optional[dict]

Per-column PostgreSQL type hints used when creating destination tables. Optional.

None

emit_asset

bool

If True, emit Airflow asset for lineage. Defaults to True.

True

cleanup_temp_files

bool

If True, delete GCS temp files after load. Defaults to True.

True

dry_run

bool

If True, simulate the operation without writing data.

False