Unable to run dataflow in DataflowRunner mode because of environment issues - google-cloud-platform

Question

I am running dataflow job which reads file and pushes data to cloudsql. Its working in local mode(DirectRunner) but failing in DataflowRunner mode. I am getting following error
I 2021-08-24T10:08:13.866094Z ERROR: Command errored out with exit status 1:
I 2021-08-24T10:08:13.866142Z command: /usr/local/bin/python3 /tmp/pip-standalone-pip-_qnajhyd/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-idcssgqm/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links /var/opt/google/dataflow -- 'setuptools>=54.0' 'setuptools_scm[toml]>=5.0' 'wheel>=0.36.2' 'Cython>=0.29.22'
I 2021-08-24T10:08:13.866202Z cwd: None
I 2021-08-24T10:08:13.866213Z Complete output (15 lines):
I 2021-08-24T10:08:13.866222Z Looking in links: /var/opt/google/dataflow
I 2021-08-24T10:08:13.866232Z Processing /var/opt/google/dataflow/setuptools-57.4.0.tar.gz
I 2021-08-24T10:08:13.866244Z Installing build dependencies: started
I 2021-08-24T10:08:13.866253Z Installing build dependencies: finished with status 'error'
I 2021-08-24T10:08:13.866262Z ERROR: Command errored out with exit status 1:
I 2021-08-24T10:08:13.866272Z command: /usr/local/bin/python3 /tmp/pip-standalone-pip-_qnajhyd/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-x9grztra/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links /var/opt/google/dataflow -- wheel
I 2021-08-24T10:08:13.866290Z cwd: None
I 2021-08-24T10:08:13.866305Z Complete output (3 lines):
I 2021-08-24T10:08:13.866314Z Looking in links: /var/opt/google/dataflow
I 2021-08-24T10:08:13.866324Z ERROR: Could not find a version that satisfies the requirement wheel (from versions: none)
I 2021-08-24T10:08:13.866335Z ERROR: No matching distribution found for wheel
I 2021-08-24T10:08:13.866344Z ----------------------------------------
I 2021-08-24T10:08:13.866359Z WARNING: Discarding file:///var/opt/google/dataflow/setuptools-57.4.0.tar.gz. Command errored out with exit status 1: /usr/local/bin/python3 /tmp/pip-standalone-pip-_qnajhyd/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-x9grztra/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links /var/opt/google/dataflow -- wheel Check the logs for full command output.
I 2021-08-24T10:08:13.866383Z ERROR: Could not find a version that satisfies the requirement setuptools>=54.0 (from versions: 57.4.0)
I 2021-08-24T10:08:13.866394Z ERROR: No matching distribution found for setuptools>=54.0
I 2021-08-24T10:08:13.866404Z ----------------------------------------
I 2021-08-24T10:08:13.869072Z WARNING: Discarding file:///var/opt/google/dataflow/pymssql-2.2.2.tar.gz. Command errored out with exit status 1: /usr/local/bin/python3 /tmp/pip-standalone-pip-_qnajhyd/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-idcssgqm/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links /var/opt/google/dataflow -- 'setuptools>=54.0' 'setuptools_scm[toml]>=5.0' 'wheel>=0.36.2' 'Cython>=0.29.22' Check the logs for full command output.
I 2021-08-24T10:08:13.869405Z ERROR: Could not find a version that satisfies the requirement pymssql (from versions: 2.2.2)
I 2021-08-24T10:08:13.869475Z ERROR: No matching distribution found for pymssql
I 2021-08-24T10:08:13.943732Z /usr/local/bin/pip failed with exit status 1
F 2021-08-24T10:08:13.943818Z Failed to install packages: failed to install requirements: exit status 1
I 2021-08-24T10:11:13.976686Z [topologymanager] RemoveContainer - Container ID: 6006a3d10b0289d5b69478c1a8189eef02db1fa0af2216bb5e7c57659498009c
I 2021-08-24T10:11:14.015846Z [topologymanager] RemoveContainer - Container ID: bb14dd1d3a5414c2ea02157ebff7e7ba227337e47ff86216a3f86847e261cdee
E 2021-08-24T10:11:14.016267Z Error syncing pod 5814435f816ec192ccc2709209670a6a ("dataflow-cloudsql-upload-2021-08-2-08240304-up8o-harness-x0f4_default(5814435f816ec192ccc2709209670a6a)"), skipping: failed to "StartContainer" for "python" with CrashLoopBackOff: "back-off 1m20s restarting failed container=python pod=dataflow-cloudsql-upload-2021-08-2-08240304-up8o-harness-x0f4_default(5814435f816ec192ccc2709209670a6a)"
I 2021-08-24T10:11:25.595721Z [topologymanager] RemoveContainer - Container ID: bb14dd1d3a5414c2ea02157ebff7e7ba227337e47ff86216a3f86847e261cdee
E 2021-08-24T10:11:25.596300Z Error syncing pod 5814435f816ec192ccc2709209670a6a ("dataflow-cloudsql-upload-2021-08-2-08240304-up8o-harness-x0f4_default(5814435f816ec192ccc2709209670a6a)"), skipping: failed to "StartContainer" for "python" with CrashLoopBackOff: "back-off 1m20s restarting failed container=python pod=dataflow-cloudsql-upload-2021-08-2-08240304-up8o-harness-x0f4_default(5814435f816ec192ccc2709209670a6a)"
I 2021-08-24T10:11:38.592275Z [topologymanager] RemoveContainer - Container ID: bb14dd1d3a5414c2ea02157ebff7e7ba227337e47ff86216a3f86847e261cdee
E 2021-08-24T10:11:38.592668Z Error syncing pod 5814435f816ec192ccc2709209670a6a ("dataflow-cloudsql-upload-2021-08-2-08240304-up8o-harness-x0f4_default(5814435f816ec192ccc2709209670a6a)"), skipping: failed to "StartContainer" for "python" with CrashLoopBackOff: "back-off 1m20s restarting failed container=python pod=dataflow-cloudsql-upload-2021-08-2-08240304-up8o-harness-x0f4_default(5814435f816ec192ccc2709209670a6a)"
I 2021-08-24T10:11:53.598346Z [topologymanager] RemoveContainer - Container ID: bb14dd1d3a5414c2ea02157ebff7e7ba227337e47ff86216a3f86847e261cdee
There are so many SO posts which suggested to see about, conflicts with dependency in requirements and even after trying multiple changes in requirements.txt (trial and error) I am unable to figure out right dependencies. I followed this and
this but was unable to debug. Following is my code files
dataflow.py
import csv
import datetime
import logging
import apache_beam as beam
from apache_beam.io.fileio import MatchFiles, ReadMatches
import argparse
import os
import json
from ldif3 import LDIFParser
import pymssql
from apache_beam.options.pipeline_options import PipelineOptions, GoogleCloudOptions
logging.basicConfig(level='INFO')
# Change the project_id
project_id = os.getenv('GOOGLE_CLOUD_PROJECT')
def get_db_connection():
mssqlhost = '127.0.0.1'
mssqluser = 'a'
mssqlpass = 'b'
mssqldb = 'usersdb'
cnx = pymssql.connect(user=mssqluser, password=mssqlpass,
host=mssqlhost, database=mssqldb)
return cnx
class SQLWriteDoFn(beam.DoFn):
# Max documents to process at a time
MAX_DOCUMENTS = 200
def __init__(self, project):
self._project = project
def setup(self):
os.system("wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy")
os.system("chmod +x cloud_sql_proxy")
os.system(f"./cloud_sql_proxy -instances=mydatabase-database=tcp:0.0.0.0:1433 &")
def start_bundle(self):
self._mutations = []
logging.info("In start_bundle")
def finish_bundle(self):
logging.info("In finish_bundle")
if self._mutations:
self._flush_batch()
def process(self, element, *args, **kwargs):
logging.info("In process")
self._mutations.append(element)
if len(self._mutations) > self.MAX_DOCUMENTS:
self._flush_batch()
def _flush_batch(self):
try:
mssqlconn = get_db_connection()
print("Connection Established to MS SQL server.")
cursor = mssqlconn.cursor()
stmt = "insert into usersdb.dbo.users_dataflow (uname, password) values (%s,%s)"
cursor.executemany(stmt, self._mutations)
mssqlconn.commit()
mssqlconn.close()
except Exception as e:
print(e)
self._mutations = []
def return_dictionary_element_if_present(dict_entry, element):
if dict_entry.get(element):
return dict_entry.get(element)[0]
return ''
class CreateEntities(beam.DoFn):
def process(self, file):
parser = LDIFParser(file.open())
arr=[]
for dn, entry in parser.parse():
# dict1 ={}
dict_entry = dict(entry)
uname = return_dictionary_element_if_present(dict_entry,'uname')
userPassword = return_dictionary_element_if_present(dict_entry,'userPassword')
arr.append(tuple((uname,userPassword)))
return arr
def dataflow(pipeline_options):
print("starting")
options = GoogleCloudOptions.from_dictionary(pipeline_options)
with beam.Pipeline(options=options) as p:
(p | 'Reading data from GCS' >> MatchFiles(file_pattern="gs://my_bucket_name/*.ldiff")
| 'file match' >> ReadMatches()
| 'Create entities' >> beam.ParDo(CreateEntities())
# | 'print to screen' >> beam.Map(print)
| 'Write to CloudSQL' >> beam.ParDo(SQLWriteDoFn(pipeline_options['project']))
)
if __name__ == '__main__':
parser = argparse.ArgumentParser(
description='dataflow options for ldif to sql')
parser.add_argument('--project', help='Project ID',
default=f'{project_id}')
parser.add_argument('--region', help='region', default='us-central1')
parser.add_argument('--runner', help='Runner', default='DirectRunner')
parser.add_argument('--staging_location',
default=f'gs://{project_id}/staging')
parser.add_argument('--temp_location',
default=f'gs://{project_id}/tmp')
args = parser.parse_args()
JOB_NAME = 'cloudsql-upload-{}'.format(
datetime.datetime.now().strftime('%Y-%m-%d-%H%M%S'))
pipeline_options = {
'project': args.project,
'staging_location': args.staging_location,
'runner': args.runner,
'job_name': JOB_NAME,
'temp_location': args.temp_location,
'save_main_session': True,
'requirements_file': 'requirements.txt',
'region': args.region,
'machine_type': 'n1-standard-8'
}
dataflow(pipeline_options)
requirements.txt
ldif3
pymssql
apache-beam[gcp]==2.31.0
Execution way
python3 dataflow.py --runner=DataflowRunner
Any help is really appreciated. Thanks in Advance.
Edit1:
I have done multiple trial and error and finally changed my requirements file as following
setuptools==57.4.0
wheel==0.37.0
setuptools_scm[toml]==6.0.1
Cython==0.29.24
ldif3
apache-beam[gcp]==2.31.0
But iam getting only following error now.
ERROR: pymssql-2.2.2-cp36-cp36m-manylinux_2_12_i686.manylinux2010_i686.whl is not a supported wheel on this platform.
The way iam running now is
python3 dataflow.py --runner=DataflowRunner --requirements_file=requirements.txt --extra_package=pymssql-2.2.2-cp36-cp36m-manylinux_2_12_i686.manylinux2010_i686.whl

Answer 1

When you run your pipeline locally, the packages that your pipeline
depends on are available because they are installed on your local
machine. However, when you want to run your pipeline remotely, you
must make sure these dependencies are available on the remote
machines.
In your execution way I see that you don't specify the requirements.txt:
so you need to start by adding --requirements_file requirements.txt
Ref: https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/

Answer 2

The error
is not a supported wheel on this platform
is probably due to trying to use a Python 3.6 wheel with a Beam pipeline that uses a different Python version. Can you try to use a wheel that matches the Python version your Beam pipeline is run with ?