I have a lambda function as part of a custom deployment stack. It calls /opt/awscli/aws s3 s3//<bucket name> /tmp/tmpfolder/
But it returns with a error 127. Presumably, because it can't find /opt/awscli/aws.
This is the code that errors:
def aws_command(*args):
aws="/opt/awscli/aws" # from AwsCliLayer
logger.info("| aws %s" % ' '.join(args))
subprocess.check_call([aws] + list(args))
This is the stacktrace:
[ERROR] 2022-12-13T06:08:25.547Z 7a8c5bee-a4ec-4ef0-aa53-6ebe2496f89d Command '['/opt/awscli/aws', 's3', 'cp', 's3://<bucket_name>/<80cd38...rest_of_zip>.zip', '/tmp/tmpblg17gcs/006ca603-6934-431e-9a96-0bd41b25d465']' returned non-zero exit status 127.
Traceback (most recent call last):
File "/var/task/index.py", line 103, in handler
s3_deploy(s3_source_zips, s3_dest, user_metadata, system_metadata, prune)
File "/var/task/index.py", line 130, in s3_deploy
aws_command("s3", "cp", s3_source_zip, archive)
File "/var/task/index.py", line 188, in aws_command
subprocess.check_call([aws] + list(args))
File "/var/lang/lib/python3.9/subprocess.py", line 373, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/opt/awscli/aws', 's3', 'cp', 's3://cdk-hnb659fds-assets-403230261384-us-west-2/80cd38c81cb1261c29a32510bb6fafd2cfbe110c40d6ab17cb9e78fc2b78202b.zip', '/tmp/tmpblg17gcs/006ca603-6934-431e-9a96-0bd41b25d465']' returned non-zero exit status 127.
Does anyone have an idea how to make sure /opt/awscli/aws it where it should be found by lambda function? Is there a way to manually install onto the ec2 instance?
Related
I am trying to execute python code on a dataproc cluster via airflow orchestration.
I am using airflow 1.10.12, and DataprocWorkflowTemplateInstantiateInlineOperator to instanciate a dataproc cluster & pass some parameters (and templated params aswell). The main objective is to run some prediction code.
Note that I upgraded this code so use airflow.providers.google.cloud.operators.dataproc and not airflow.contrib.operators.dataproc_operator to import DataprocInstantiateInlineWorkflowTemplateOperator, because the former introduced the parameters kwarg that, in theory, permits passing arguments to the cluster. Using the later in my other scripts, I have no errors, but I cannot introduce parameters to the cluster.
...
from airflow.providers.google.cloud.operators.dataproc import (
DataprocInstantiateInlineWorkflowTemplateOperator,
)
...
workflow_seg_members = make_workflow_template(
region=REGION,
dataproc_job_bucket=DATAPROC_JOB_BUCKET,
python_main_executable_path="segmentation_members/seg_members_prediction.py",
)
op_seg_members_prediction = DataprocInstantiateInlineWorkflowTemplateOperator(
task_id="seg_members_prediction",
project_id=PROJECT_ID,
region=REGION,
template=workflow_seg_members,
parameters={
"execution_date_str": "{{ds_nodash}}",
"project": "<REDACTED>",
"dataset": "<REDACTED>",
"features_table_prefix": "global_features",
"output_table_prefix": "seg_members_output",
"path_to_model": "segmentation_members/DecisionTreeClassifier.pkl",
"bucket_name": "<REDACTED>",
"model_designation": "Segmentation Members",
},
)
in seg_members_prediction.py, I use argparse to create the needed arguments.
The error I am getting is :
TypeError: Parameter to MergeFrom() must be instance of same class: expected google.cloud.dataproc.v1beta2.OrderedJob got str.
My questions are :
How do I fix this MergeFrom() exception?
Is this the right approach to pass parameters from airflow to my dataproc cluster?
Here is the complete stack :
File "/usr/local/lib/airflow/airflow/providers/google/common/hooks/base_google.py", line 373, in inner_wrapper
return func(self, *args, **kwargs)
File "/usr/local/lib/airflow/airflow/providers/google/cloud/hooks/dataproc.py", line 712, in instantiate_inline_workflow_template
metadata=metadata,
File "/opt/python3.6/lib/python3.6/site-packages/google/cloud/dataproc_v1beta2/gapic/workflow_template_service_client.py", line 488, in instantiate_inline_workflow_template
request_id=request_id,
TypeError: Parameter to MergeFrom() must be instance of same class: expected google.cloud.dataproc.v1beta2.OrderedJob got str.
[2022-06-13 12:36:34,696] {taskinstance.py:1197} INFO - Marking task as UP_FOR_RETRY. dag_id=ds_seg_members_integration, task_id=seg_members_prediction, execution_date=20220610T162234, start_date=20220613T123629, end_date=20220613T123634
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 7, in <module>
exec(compile(f.read(), __file__, 'exec'))
File "/usr/local/lib/airflow/airflow/bin/airflow", line 37, in <module>
args.func(args)
File "/usr/local/lib/airflow/airflow/utils/cli.py", line 76, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/airflow/airflow/bin/cli.py", line 561, in run
_run(args, dag, ti)
File "/usr/local/lib/airflow/airflow/bin/cli.py", line 480, in _run
pool=args.pool,
File "/usr/local/lib/airflow/airflow/utils/db.py", line 74, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/airflow/airflow/models/taskinstance.py", line 986, in _run_raw_task
result = task_copy.execute(context=context)
File "/usr/local/lib/airflow/airflow/providers/google/cloud/operators/dataproc.py", line 1748, in execute
metadata=self.metadata,
File "/usr/local/lib/airflow/airflow/providers/google/common/hooks/base_google.py", line 373, in inner_wrapper
return func(self, *args, **kwargs)
File "/usr/local/lib/airflow/airflow/providers/google/cloud/hooks/dataproc.py", line 712, in instantiate_inline_workflow_template
metadata=metadata,
File "/opt/python3.6/lib/python3.6/site-packages/google/cloud/dataproc_v1beta2/gapic/workflow_template_service_client.py", line 488, in instantiate_inline_workflow_template
request_id=request_id,
TypeError: Parameter to MergeFrom() must be instance of same class: expected google.cloud.dataproc.v1beta2.OrderedJob got str.
EDIT :
I tried running the following code but still got the same error :
op_seg_members_prediction = DataprocInstantiateInlineWorkflowTemplateOperator(
task_id="seg_members_prediction",
project_id=PROJECT_ID,
region=REGION,
template=workflow_seg_members,
)
op_seg_members_prediction.execute(context="DEBUG")
After browsing the apache-airflow-providers-google doc I discovered it isn't compatible with airflow 1.
You can install this package on top of an existing Airflow 2
installation (see Requirements below for the minimum Airflow version
supported) via pip install apache-airflow-providers-google
The package supports the following python versions: 3.7,3.8,3.9,3.10
So I upgraded to airflow 2 in my local environment, and the MergeFrom() error stopped occurring.
This still begs the question : How do you pass parameters from airflow to dataproc for airflow 1
I'm using .ebextensions to run some automation commands, but some commands are failing without a explaining the cause issue or printing any informative stacktrace, but when I run the same command manually it works like a charm with no issues:
the command:
aws logs put-retention-policy --log-group-name `{"Fn::Join":["/", ["/aws/eb", { "Ref":"AWSEBEnvironmentName" }, "var/log/app-log"]]}` --retention-in-days 7 --region `{"Ref":"AWS::Region"}`
the stack trace:
[ERROR] Command 03 (aws logs put-retention-policy --log-group-name /aws/log-group-name --retention-in-days 7 --region us-west-2) failed
2021-04-19 21:33:50,248 [ERROR] Error encountered during build of prebuild_2_squirrel: Command 03 failed
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 542, in run_config
CloudFormationCarpenter(config, self._auth_config).build(worklog)
File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 260, in build
changes['commands'] = CommandTool().apply(self._config.commands)
File "/usr/lib/python2.7/site-packages/cfnbootstrap/command_tool.py", line 117, in apply
raise ToolError(u"Command %s failed" % name)
ToolError: Command 03 failed
2021-04-19 21:33:50,250 [ERROR] -----------------------BUILD FAILED!------------------------
2021-04-19 21:33:50,250 [ERROR] Unhandled exception during build: Command 03 failed
Traceback (most recent call last):
File "/opt/aws/bin/cfn-init", line 171, in <module>
worklog.build(metadata, configSets)
File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 129, in build
Contractor(metadata).build(configSets, self)
File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 530, in build
self.run_config(config, worklog)
File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 542, in run_config
CloudFormationCarpenter(config, self._auth_config).build(worklog)
File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 260, in build
changes['commands'] = CommandTool().apply(self._config.commands)
File "/usr/lib/python2.7/site-packages/cfnbootstrap/command_tool.py", line 117, in apply
raise ToolError(u"Command %s failed" % name)
Anyone seen this before or knows what it could be the issue ?
Thanks in advance.
after checking the /var/log/cfn-init-cmd.log, I saw details about the issue, the log group gets created few seconds after, so I just need to wait
There was no issue in building the project a little while back, but it started throwing below error.
RuntimeError: Container does not exist. Cannot get logs for this
container
Normally this happens when docker cannot mount the shared directory, but in this case even adding the lambda directory manually in the docker interface didn't help!
Complete debug log of sam build --use-container
Building function 'SAListManagerUrlLambda'
Fetching lambci/lambda:build-python3.7 Docker container image......
Mounting C:\Users\xxxx\xxxx\xxxx\xxxx\functions\xxxx-xxxx\xxxx-xxxx as /tmp/samcli/source:ro,delegated inside runtime container
Container was not created. Skipping deletion
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 1292, 'exitReason': 'RuntimeError', 'exitCode': 255, 'requestId': 'cbfcd29c-16ae-xxxx-xxxx-b9ffec8de75a', 'installationId': 'fece8ccc-cb84-xxxx-xxxx-ac72820ef0c3', 'sessionId': 'e1cbc287-1850-xxxx-xxxx-3a235769f7fb', 'executionEnvironment': 'CLI', 'pyversion': '3.7.6', 'samcliVersion': '0.53.0'}}]}
HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Traceback (most recent call last):
File "D:\obj\windows-release\37amd64_Release\msi_python\zip_amd64\runpy.py", line 193, in _run_module_as_main
File "D:\obj\windows-release\37amd64_Release\msi_python\zip_amd64\runpy.py", line 85, in _run_code
File "C:\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\__main__.py", line 12, in <module>
cli(prog_name="sam")
File "C:\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "C:\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "C:\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "C:\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\decorators.py", line 73, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "C:\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "C:\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\telemetry\metrics.py", line 96, in wrapped
raise exception # pylint: disable=raising-bad-type
File "C:\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\telemetry\metrics.py", line 62, in wrapped
return_value = func(*args, **kwargs)
File "C:\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\commands\build\command.py", line 129, in cli
mode,
File "C:\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\commands\build\command.py", line 194, in do_cli
artifacts = builder.build()
File "C:\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\build\app_builder.py", line 117, in build
function.metadata)
File "C:\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\build\app_builder.py", line 271, in _build_function
options)
File "C:\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\build\app_builder.py", line 369, in _build_function_on_container
container.wait_for_logs(stdout=stdout_stream, stderr=stderr_stream)
File "C:\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\local\docker\container.py", line 197, in wait_for_logs
raise RuntimeError("Container does not exist. Cannot get logs for this container")
RuntimeError: Container does not exist. Cannot get logs for this container
In my case the reason was different, Action Center's Focus Assist was set to Alarms Only.
This caused the share directory notification to fail, causing the build failure.
So, make sure your Focus Assist is set to OFF.
It seems that many situations can trigger the same error. For more information the --debug option can be used like this:
sam build --use-container --debug
I see that you are using it, because you got extra information like this:
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 1292, 'exitReason': 'RuntimeError', 'exitCode': 255, 'requestId': 'cbfcd29c-16ae-xxxx-xxxx-b9ffec8de75a', 'installationId': 'fece8ccc-cb84-xxxx-xxxx-ac72820ef0c3', 'sessionId': 'e1cbc287-1850-xxxx-xxxx-3a235769f7fb', 'executionEnvironment': 'CLI', 'pyversion': '3.7.6', 'samcliVersion': '0.53.0'}}]}
HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Traceback (most recent call last):
In my case I did suppose that the error was sending the telemetry.
My guess is that somehow the build process need pass the region. In my case it is not us-west-2.
Anyway, I disabled it as specified in the documentation and it now works.
In my case local disk in my cloud9 was almost full, so I had to delete some docker images that comes pre-installed with cloud9.
To remove an image use
docker rmi Image
This will clear up space and your build will not fail the next time.
I need to make requests from a python script which is hosted on an already created Google Cloud Compute engine instance, to the google cloud NLP API. So i followed the steps that were needed to accomplish this task.
I activated the NLP API for my project which also includes the VM instance
I downloaded the service key(as JSON) for the default service account that was created with the VM(Because im trying to authenticate with Application Default Credentials)
I set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the downloaded keys path
Installed google cloud client library for python
I then proceeded to check if the credentials work by running the example with the google storage API and it returned an empty list which means it worked as intended.
But i get the following error when trying to make calls to the NLP API
Traceback (most recent call last):
File "/home/user/anaconda3/lib/python3.7/site-
packages/google/api_core/grpc_helpers.py", line 57, in
error_remapped_callable
return callable_(*args, **kwargs)
File "/home/user/anaconda3/lib/python3.7/site-
packages/grpc/_channel.py", line 549, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/home/user/anaconda3/lib/python3.7/site-
packages/grpc/_channel.py", line 466, in _end_unary_response_blocking
raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.PERMISSION_DENIED
details = "Request had insufficient authentication scopes."
debug_error_string = "
{"created":"#1552931720.241957477","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Request had insufficient authentication scopes.","grpc_status":7}">
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/user/anaconda3/lib/python3.7/site-
packages/google/cloud/language_v1/gapic/language_service_client.py",
line 230, in analyze_sentiment
request, retry=retry, timeout=timeout, metadata=metadata
File "/home/user/anaconda3/lib/python3.7/site-
packages/google/api_core/gapic_v1/method.py", line 143, in __call__
return wrapped_func(*args, **kwargs)
File "/home/user/anaconda3/lib/python3.7/site-
packages/google/api_core/retry.py", line 270, in retry_wrapped_func
on_error=on_error,
File "/home/user/anaconda3/lib/python3.7/site-
packages/google/api_core/retry.py", line 179, in retry_target
return target()
File "/home/usere/anaconda3/lib/python3.7/site-
packages/google/api_core/timeout.py", line 214, in func_with_timeout
return func(*args, **kwargs)
File "/home/user/anaconda3/lib/python3.7/site-
packages/google/api_core/grpc_helpers.py", line 59, in
error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.PermissionDenied: 403 Request had
insufficient authentication scopes.
Any Idea why this is happening?
Below is the simple code snippet, which I'm trying to run, it gives exception.
I've configured AWS on my local and I'm able to describe the same stack in the AWS Opsworks UI. Can someone help here, with what could be the reason:
import boto3
client=boto3.client('opsworks')
response=client.describe_stack_summary(
StackId="6efce529-0b77-43dc-981b-ff20b906c4ae"
)
print(response)
Stacktrace for error:
Traceback (most recent call last):
File "botoTest.py", line 9, in <module>
StackId="6efce529-0b77-43dc-981b-ff20b906c4ae"
File
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-
packages/botocore/client.py", line 320, in _api_call
return self._make_api_call(operation_name, kwargs)
File
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-
packages/botocore/client.py", line 623, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.ResourceNotFoundException: An error occurred
(ResourceNotFoundException) when calling the DescribeStackSummary
operation: Unable to find stack with ID 6efce529-0b77-43dc-981b-
ff20b906c4ae