Airflow bigquery_to_gcs operator changing field_delimiter - google-cloud-platform

Question

I am trying to use Airflow operator BigQueryToGCSOperator & forcing field_delimiter to be pipe (|) , however output of the file is always coming comma (,) delimited.
I have also tried operator BigQueryToCloudStorageOperator which has same behavior.
Any idea what wrong I am doing here ?
from airflow.providers.google.cloud.transfers.bigquery_to_gcs import (
BigQueryToGCSOperator,
data_to_gcs = BigQueryToGCSOperator(
task_id="BigQuery_to_GoogleCloudBucket",
gcp_conn_id="google_cloud_default",
project_id=project_id,
source_project_dataset_table=f"{project_id}.{temp_dataset_id}.{temp_table}",
location="EU",
print_header=True,
destination_cloud_storage_uris=destination_uri,
export_format="csv",
field_delimiter="|",
)
Thanks in advance for your reply.

Answer 1

Normally if you set the export_format field with CSV (uppercase instead of lower case) and field_delimiter it should work :
from airflow.providers.google.cloud.transfers.bigquery_to_gcs import (
BigQueryToGCSOperator,
data_to_gcs = BigQueryToGCSOperator(
task_id="BigQuery_to_GoogleCloudBucket",
gcp_conn_id="google_cloud_default",
project_id=project_id,
source_project_dataset_table=f"{project_id}.{temp_dataset_id}.{temp_table}",
location="EU",
print_header=True,
destination_cloud_storage_uris=destination_uri,
export_format="CSV",
field_delimiter="|",
)
I saw this code snippet in the Airflow code, and I think it can cause an issue if you set the export_format with csv as lowercase value :
if self.export_format == 'CSV':
# Only set fieldDelimiter and printHeader fields if using CSV.
# Google does not like it if you set these fields for other export
# formats.
configuration['extract']['fieldDelimiter'] = self.field_delimiter
configuration['extract']['printHeader'] = self.print_header
In your case, this code snippet is not invoked and the operator take the default value for field_delimiter which is ,
Here you can see the default values used in the constructor of this operator in the Airflow code :
def __init__(
self,
*,
source_project_dataset_table: str,
destination_cloud_storage_uris: List[str],
compression: str = 'NONE',
export_format: str = 'CSV',
field_delimiter: str = ',',
print_header: bool = True,
gcp_conn_id: str = 'google_cloud_default',
bigquery_conn_id: Optional[str] = None,
delegate_to: Optional[str] = None,
labels: Optional[Dict] = None,
location: Optional[str] = None,
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
**kwargs,
)