Transfers API Reference¶
PostgresToGCSOperator¶
Bases: BaseOperator
Operator to copy data from PostgreSQL to Google Cloud Storage.
Exports data from PostgreSQL using SQL queries and uploads the results to GCS in CSV, Parquet, or JSONL format. Supports automatic format switching when problematic characters are detected.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
str
|
Reference to a specific Postgres hook. |
required |
|
str
|
The SQL query to be executed. |
required |
|
Optional[str]
|
Optional WHERE clause to append to the SQL query. Mutually exclusive with providing a complete WHERE in sql. |
None
|
|
str
|
The GCS bucket to upload to. |
required |
|
str
|
The GCS filename to upload to (including a folder). |
required |
|
str
|
The connection ID used to connect to Google Cloud. Defaults to 'google_cloud_default'. |
'google_cloud_default'
|
|
str
|
The format to export the data to. Supported formats are 'csv', 'parquet', and 'jsonl'. Defaults to 'csv'. |
CSV
|
|
Optional[str]
|
If set, a GCS file with the schema will be uploaded. |
None
|
|
Optional[Dict[str, str]]
|
BigQuery type overrides by top-level field name. pandas_chunksize: The number of rows to include in each chunk processed by pandas. |
None
|
|
Optional[float]
|
Optional maximum size for each uploaded shard. When set, the returned filename is a wildcard matching all generated shards. |
None
|
|
bool
|
If True, use PostgreSQL COPY command for streaming. |
False
|
|
bool
|
If True, use a temporary file instead of streaming. |
False
|
|
Optional[dict]
|
Arguments to pass to pandas.DataFrame.to_csv(). |
None
|
|
Optional[dict]
|
Arguments to pass to DataFrame.to_parquet(). |
None
|
|
str
|
Mode for detecting column types. Defaults to 'postgres'. |
POSTGRES
|
|
int
|
Number of rows to sample for JSON detection. Defaults to 100. |
100
|
|
float
|
Threshold for JSON column detection. Defaults to 0.9. |
0.9
|
|
bool
|
If True, detect problematic characters (quotes, newlines) in text columns and automatically switch to JSONL format. If False, use requested format with proper CSV quoting options. Defaults to True. |
True
|
|
bool
|
If True, simulate the operation without writing data. |
False
|
GCSToPostgresOperator¶
Bases: BaseOperator
Operator to load data from Google Cloud Storage to PostgreSQL.
Downloads files from GCS and loads them into PostgreSQL tables. Supports CSV, JSONL, Parquet, and Avro formats with automatic type coercion.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
str
|
PostgreSQL table name (schema.table or just table). |
required |
|
str
|
GCS bucket name. |
required |
|
str
|
GCS object path/filename. |
required |
|
str
|
PostgreSQL connection ID. |
required |
|
str
|
GCP connection ID. Defaults to 'google_cloud_default'. |
required |
|
Columns for upsert conflict resolution. Optional. |
None
|
|
|
If True, replace table content. If False, append. Defaults to False. |
False
|
|
|
bool
|
If True, grant ALL privileges to PUBLIC. Defaults to True. |
True
|
|
bool
|
If True, create table when source is empty. Defaults to False. |
False
|
|
bool
|
If True, create table if it doesn't exist. Defaults to False. |
False
|
|
Optional[str]
|
Column name for partitioning. When set, creates one partition per unique value in the column. Requires replace=False. Defaults to None. |
None
|
|
Optional[dict]
|
Source column types for type coercion. Optional. |
None
|
|
Optional[dict]
|
Per-column PostgreSQL type hints used when creating destination tables. Optional. |
None
|
|
Columns to exclude from updates during upsert. |
None
|
|
|
bool
|
If True, simulate the operation without writing data. |
False
|
PostgresToBigQueryOperator¶
Bases: BaseOperator
Transfer data from PostgreSQL to BigQuery via GCS staging.
This operator extracts data from PostgreSQL, uploads to a temporary GCS location, then loads into BigQuery. Includes source validation, automatic schema detection, and cleanup.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
str
|
PostgreSQL connection ID. |
required |
|
str | None
|
SQL query to extract data. Mutually exclusive with source_project_dataset_table. |
''
|
|
Optional[str]
|
WHERE clause to filter data. Only applies when source_project_dataset_table is used. Mutually exclusive with sql. |
None
|
|
str | None
|
Source table (postgres.schema.table). Used if sql is not provided. |
None
|
|
str
|
BigQuery destination table (project.dataset.table or dataset.table). |
required |
|
str
|
GCP connection ID. Defaults to 'google_cloud_default'. |
'google_cloud_default'
|
|
str
|
GCS bucket for temporary staging. |
required |
|
Optional[str]
|
GCS path for temp files. Auto-generated if not provided. |
None
|
|
str
|
Export format: 'csv' or 'jsonl'. Defaults to 'csv'. |
CSV
|
|
Optional[str]
|
GCS path for BigQuery schema JSON file. Optional. |
None
|
|
Optional[dict[str, str]]
|
BigQuery type overrides by top-level field name. |
None
|
|
int
|
Rows per chunk for large exports. Defaults to 100000. |
100000
|
|
bool
|
If True, use PostgreSQL COPY for streaming. Defaults to False. |
False
|
|
bool
|
If True, use temporary file instead of streaming. |
False
|
|
bool
|
If True, validate source has data before transfer. |
True
|
|
Optional[str]
|
Custom SQL for source validation. Optional. |
None
|
|
str
|
BigQuery write disposition. Defaults to 'WRITE_TRUNCATE'. |
WRITE_TRUNCATE
|
|
str
|
BigQuery create disposition. Defaults to 'CREATE_IF_NEEDED'. |
CREATE_IF_NEEDED
|
|
bool
|
If True, emit Airflow asset for lineage. Defaults to True. |
True
|
|
bool
|
If True, delete GCS temp files after load. Defaults to True. |
True
|
|
Optional[str]
|
BigQuery partition column. Optional. |
None
|
|
str
|
Partition type: 'DAY', 'HOUR', 'MONTH', 'YEAR'. Defaults to 'DAY'. |
DAY
|
|
Optional[List[str]]
|
BigQuery clustering columns. Optional. |
None
|
|
str
|
BigQuery dataset location. Defaults to 'us-central1'. |
'us-central1'
|
|
bool
|
If True, create empty table when source is empty. |
False
|
|
bool
|
If True, switch to JSONL for JSON columns. |
True
|
|
bool
|
If True, simulate the operation without writing data. |
False
|
BigQueryToPostgresOperator¶
Bases: BaseOperator
Transfer data from BigQuery to PostgreSQL via GCS staging.
This operator extracts data from BigQuery to a temporary GCS location, then loads into PostgreSQL. Includes source validation, automatic format detection, and cleanup.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Optional[str]
|
BigQuery source table (project.dataset.table or dataset.table). Required if sql is not provided. |
None
|
|
Optional[str]
|
SQL query to export. Mutually exclusive with source_project_dataset_table. If provided, the query result is exported to PostgreSQL. |
None
|
|
Optional[str]
|
WHERE clause to filter data. Only applies when source_project_dataset_table is used. Mutually exclusive with sql. |
None
|
|
str
|
PostgreSQL connection ID. |
required |
|
str
|
PostgreSQL destination table (schema.table). |
required |
|
str
|
GCP connection ID. Defaults to 'google_cloud_default'. |
'google_cloud_default'
|
|
str
|
GCS bucket for temporary staging. |
required |
|
Optional[str]
|
GCS path for temp files. Auto-generated if not provided. |
None
|
|
str
|
Export format: 'parquet', 'csv', 'jsonl', or 'avro'. Defaults to 'parquet'. |
PARQUET
|
|
bool
|
If True, detect JSON columns from BigQuery. Defaults to True. |
True
|
|
bool
|
If True, validate source has data before transfer. |
True
|
|
Optional[str]
|
Custom SQL for source validation. Optional. |
None
|
|
Optional[List[str]]
|
Columns for upsert conflict resolution. Optional. |
None
|
|
bool
|
If True, replace table content. If False and conflict_columns provided, perform upsert. Defaults to True. |
True
|
|
bool
|
If True, create table when source is empty. Defaults to False. |
False
|
|
bool
|
If True, create table if it doesn't exist. Defaults to False. |
False
|
|
Optional[str]
|
Column name for partitioning. When set, creates one partition per unique value in the column. Requires replace=False. Defaults to None. |
None
|
|
Optional[dict]
|
Per-column PostgreSQL type hints used when creating destination tables. Optional. |
None
|
|
bool
|
If True, emit Airflow asset for lineage. Defaults to True. |
True
|
|
bool
|
If True, delete GCS temp files after load. Defaults to True. |
True
|
|
bool
|
If True, simulate the operation without writing data. |
False
|