Broken DAG : Bad response from application: 429 - google-cloud-platform

I started getting this error when I started to develop the awin_integration.py dag.
Broken DAG: [/home/airflow/gcs/dags/media/awin_integration.py] Bad response from application: 429 / {'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Pragma': 'no-cache', 'Expires': '0', 'X-Frame-Options': 'DENY', 'Content-Type': 'application/json', 'Content-Length': '105', 'Date': 'Sun, 17 Jan 2021 19:38:09 GMT', 'Connection': 'keep-alive'} / {"error":"request.limit.exceeded","description":"Requests limit '20' times per '1' minutes' is exceeded"}
What does this error mean?

Related

Unable to trigger Airflow using REST Api in google cloud composer

I have tried running the 'Call Airflow REST API using client_id' from the link https://cloud.google.com/composer/docs/access-airflow-api#curl. I have triggered the code as below
data_1 = "{'key':'value'}"
trigger_dag(data_1)
I have been facing the error as below
> Exception: Bad response from application: 400 / {'Date': 'Wed, 20 Apr
> 2022 18:58:00 GMT', 'Content-Type': 'application/problem+json',
> 'Content-Length': '231', 'Server': 'gunicorn',
> 'Access-Control-Allow-Headers': '', 'Access-Control-Allow-Methods':
> '', 'Access-Control-Allow-Origin': '', 'X-Frame-Options': 'DENY',
> 'Via': '1.1 google'} / '{\n "detail": "\\"{\'key\':\'value\'}\\" is
> not of type \'object\' - \'conf\'",\n "status": 400,\n "title": "Bad
> Request",\n "type":
> "https://airflow.apache.org/docs/apache-airflow/2.1.4/stable-rest-api-ref.html#section/Errors/BadRequest"\n}\n'

Can't receive text message using SNS python Lambda

I've read everything possible about SNS and I can't figure out why any text messages are sent from an aws lambda function.
My function can't be more simple:
def send_sms(message):
print ("sending message...")
print(message)
client = boto3.client("sns")
response = client.publish(
PhoneNumber="+336xxxxxxxxx",
Message=message
)
print (response)
I'm using an SNSFullAccess for this lambda function.
And my log for the print (response) are:
{'MessageId': '6e9aa592-4ff7-5cf1-be92-ffe34ca16120',
'ResponseMetadata': {'RequestId':
'5d73d693-b584-5ecb-926f-8ce2b6d2831a', 'HTTPStatusCode': 200,
'HTTPHeaders': {'x-amzn-requestid':
'5d73d693-b584-5ecb-926f-8ce2b6d2831a', 'content-type': 'text/xml',
'content-length': '294', 'date': 'Sun, 30 Jan 2022 21:20:22 GMT'},
'RetryAttempts': 0}}

Fail to deploy Django application with standard environment to Google App Engine

I deployed my Django app last Wednesday and everything was ok. On the next day, I tried to deploy changes, and App Engine ran into an error: INTERNAL: Internal error encountered. Also, I ran the deployment command today using the same app.yml config and nothing has changed. The same error has occurred.
My app.yml:
runtime: python37
instance_class: F1
env: standard
entrypoint: gunicorn -b :$PORT MyProject.wsgi --timeout 600
beta_settings:
cloud_sql_instances: "<project-name>:<region>:<database-name>"
handlers:
- url: /.*
script: auto
secure: always
redirect_http_response_code: 301
Here are the logs:
DEBUG: Running [gcloud.app.deploy] with arguments: [--quiet: "True", --verbosity: "debug", DEPLOYABLES:1: "[u'app.yaml']"]
DEBUG: Loading runtimes experiment config from [gs://runtime-builders/experiments.yaml]
INFO: Reading [<googlecloudsdk.api_lib.storage.storage_util.ObjectReference object at 0x7f463bc2cd10>]
DEBUG:
Traceback (most recent call last):
File "/opt/hostedtoolcache/gcloud/290.0.1/x64/lib/googlecloudsdk/api_lib/app/runtime_builders.py", line 269, in _Read
with contextlib.closing(storage_client.ReadObject(object_)) as f:
File "/opt/hostedtoolcache/gcloud/290.0.1/x64/lib/googlecloudsdk/api_lib/storage/storage_api.py", line 303, in ReadObject
object_=object_ref, err=http_exc.HttpException(err)))
BadFileException: Could not read [<googlecloudsdk.api_lib.storage.storage_util.ObjectReference object at 0x7f463bc2cd10>]. Please retry: HTTPError 404: No such object: runtime-builders/experiments.yaml
DEBUG: Experiment config file could not be read. This error is informational, and does not cause a deployment to fail. Reason: Unable to read the runtimes experiment config: [gs://runtime-builders/experiments.yaml], error: Could not read [<googlecloudsdk.api_lib.storage.storage_util.ObjectReference object at 0x7f463bc2cd10>]. Please retry: HTTPError 404: No such object: runtime-builders/experiments.yaml
Traceback (most recent call last):
File "/opt/hostedtoolcache/gcloud/290.0.1/x64/lib/surface/app/deploy.py", line 133, in _ServerSideExperimentEnabled
runtimes_builder_root)
File "/opt/hostedtoolcache/gcloud/290.0.1/x64/lib/googlecloudsdk/api_lib/app/runtime_builders.py", line 524, in LoadFromURI
.format(uri, e))
ExperimentsError: Unable to read the runtimes experiment config: [gs://runtime-builders/experiments.yaml], error: Could not read [<googlecloudsdk.api_lib.storage.storage_util.ObjectReference object at 0x7f463bc2cd10>]. Please retry: HTTPError 404: No such object: runtime-builders/experiments.yaml
DEBUG: API endpoint: [https://appengine.googleapis.com/], API version: [v1]
Services to deploy:
descriptor: [/home/runner/work/MyProject/MyProject/app.yaml]
source: [/home/runner/work/MyProject/MyProject]
target project: [<project-name>]
target service: [default]
target version: [20201018t154518]
target url: [https://<site-url>]
DEBUG: No bucket specified, retrieving default bucket.
DEBUG: Using bucket [gs://staging.<bucket-name>.appspot.com].
Beginning deployment of service [default]...
INFO: Using ignore file at [/home/runner/work/MyProject/MyProject/.gcloudignore].
DEBUG: Skipping file [README.md]
DEBUG: Skipping file [static]
DEBUG: Skipping file [.github]
DEBUG: Skipping file [tests]
DEBUG: Skipping file [.git]
DEBUG: Skipping file [misc]
DEBUG: Skipping file [static-root]
DEBUG: Skipping upload of [my_file_1.py]
...
DEBUG: Skipping upload of [my_file_n.py]
INFO: Incremental upload skipped 85.92% of data
DEBUG: Uploading 164 files to Google Cloud Storage
DEBUG: Using [16] threads
#============================================================#
#= Uploading 164 files to Google Cloud Storage =#
#INFO: Uploading [/home/runner/work/MyProject/MyProject/my_file_1.py] to [staging.<project-name>.appspot.com/1373be4811b24b04f2c1f0aaef730bdc76346e4a]
...
#INFO: Uploading [/home/runner/work/MyProject/MyProject/my_file_n.py] to [staging.<project-name>.appspot.com/1373be4811b24b04f2c1f0aaef730bdc76346e4a]
File upload done.
INFO: Manifest: [{...}]
DEBUG: Converted YAML to JSON: "{
"betaSettings": {
"cloud_sql_instances": "<project-name>:<region>:<database-name>"
},
"entrypoint": {
"shell": "gunicorn -b :$PORT MyProject.wsgi --timeout 600"
},
"env": "standard",
"handlers": [
{
"redirectHttpResponseCode": "REDIRECT_HTTP_RESPONSE_CODE_301",
"script": {
"scriptPath": "auto"
},
"securityLevel": "SECURE_ALWAYS",
"urlRegex": "/.*"
}
],
"instanceClass": "F1",
"runtime": "python37"
}"
DEBUG: Response returned status 500, retrying
DEBUG: Retrying request to url https://appengine.googleapis.com/v1/apps/<project-name>/services/default/versions?alt=json after exception HttpError accessing <https://appengine.googleapis.com/v1/apps/<project-name>/services/default/versions?alt=json>: response: <{'status': '500', 'content-length': '109', 'x-xss-protection': '0', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF', '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Sun, 18 Oct 2020 15:45:29 GMT', 'x-frame-options': 'SAMEORIGIN', 'alt-svc': 'h3-Q050=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"', 'content-type': 'application/json; charset=UTF-8'}>, content <{
"error": {
"code": 500,
"message": "Internal error encountered.",
"status": "INTERNAL"
}
}
>
DEBUG: Response returned status 500, retrying
DEBUG: Retrying request to url https://appengine.googleapis.com/v1/apps/<project-name>/services/default/versions?alt=json after exception HttpError accessing <https://appengine.googleapis.com/v1/apps/<project-name>/services/default/versions?alt=json>: response: <{'status': '500', 'content-length': '109', 'x-xss-protection': '0', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF', '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Sun, 18 Oct 2020 15:45:29 GMT', 'x-frame-options': 'SAMEORIGIN', 'alt-svc': 'h3-Q050=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"', 'content-type': 'application/json; charset=UTF-8'}>, content <{
"error": {
"code": 500,
"message": "Internal error encountered.",
"status": "INTERNAL"
}
}
>
DEBUG: Response returned status 500, retrying
DEBUG: Retrying request to url https://appengine.googleapis.com/v1/apps/<project-name>/services/default/versions?alt=json after exception HttpError accessing <https://appengine.googleapis.com/v1/apps/<project-name>/services/default/versions?alt=json>: response: <{'status': '500', 'content-length': '109', 'x-xss-protection': '0', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF', '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Sun, 18 Oct 2020 15:45:29 GMT', 'x-frame-options': 'SAMEORIGIN', 'alt-svc': 'h3-Q050=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"', 'content-type': 'application/json; charset=UTF-8'}>, content <{
"error": {
"code": 500,
"message": "Internal error encountered.",
"status": "INTERNAL"
}
}
>
DEBUG: Response returned status 500, retrying
DEBUG: Retrying request to url https://appengine.googleapis.com/v1/apps/<project-name>/services/default/versions?alt=json after exception HttpError accessing <https://appengine.googleapis.com/v1/apps/<project-name>/services/default/versions?alt=json>: response: <{'status': '500', 'content-length': '109', 'x-xss-protection': '0', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF', '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Sun, 18 Oct 2020 15:45:29 GMT', 'x-frame-options': 'SAMEORIGIN', 'alt-svc': 'h3-Q050=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"', 'content-type': 'application/json; charset=UTF-8'}>, content <{
"error": {
"code": 500,
"message": "Internal error encountered.",
"status": "INTERNAL"
}
}
>
DEBUG: (gcloud.app.deploy) INTERNAL: Internal error encountered.
Traceback (most recent call last):
File "/opt/hostedtoolcache/gcloud/290.0.1/x64/lib/googlecloudsdk/calliope/cli.py", line 983, in Execute
resources = calliope_command.Run(cli=self, args=args)
File "/opt/hostedtoolcache/gcloud/290.0.1/x64/lib/googlecloudsdk/calliope/backend.py", line 807, in Run
resources = command_instance.Run(args)
File "/opt/hostedtoolcache/gcloud/290.0.1/x64/lib/surface/app/deploy.py", line 117, in Run
default_strategy=flex_image_build_option_default))
File "/opt/hostedtoolcache/gcloud/290.0.1/x64/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 643, in RunDeploy
ignore_file=args.ignore_file)
File "/opt/hostedtoolcache/gcloud/290.0.1/x64/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 433, in Deploy
extra_config_settings)
File "/opt/hostedtoolcache/gcloud/290.0.1/x64/lib/googlecloudsdk/api_lib/app/appengine_api_client.py", line 172, in DeployService
extra_config_settings)
File "/opt/hostedtoolcache/gcloud/290.0.1/x64/lib/googlecloudsdk/api_lib/app/appengine_api_client.py", line 249, in _CreateVersion
return self.client.apps_services_versions.Create(create_request)
File "/opt/hostedtoolcache/gcloud/290.0.1/x64/lib/googlecloudsdk/third_party/apis/appengine/v1/appengine_v1_client.py", line 827, in Create
config, request, global_params=global_params)
File "/opt/hostedtoolcache/gcloud/290.0.1/x64/lib/third_party/apitools/base/py/base_api.py", line 729, in _RunMethod
http, http_request, **opts)
File "/opt/hostedtoolcache/gcloud/290.0.1/x64/lib/third_party/apitools/base/py/http_wrapper.py", line 346, in MakeRequest
check_response_func=check_response_func)
File "/opt/hostedtoolcache/gcloud/290.0.1/x64/lib/third_party/apitools/base/py/http_wrapper.py", line 402, in _MakeRequestNoRetry
check_response_func(response)
File "/opt/hostedtoolcache/gcloud/290.0.1/x64/lib/googlecloudsdk/api_lib/util/apis.py", line 281, in _CheckResponseForApiEnablement
http_wrapper.CheckResponse(response)
File "/opt/hostedtoolcache/gcloud/290.0.1/x64/lib/third_party/apitools/base/py/http_wrapper.py", line 223, in CheckResponse
raise exceptions.BadStatusCodeError.FromResponse(response)
BadStatusCodeError: HttpError accessing <https://appengine.googleapis.com/v1/apps/<project-name>/services/default/versions?alt=json>: response: <{'status': '500', 'content-length': '109', 'x-xss-protection': '0', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF', '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Sun, 18 Oct 2020 15:46:08 GMT', 'x-frame-options': 'SAMEORIGIN', 'alt-svc': 'h3-Q050=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"', 'content-type': 'application/json; charset=UTF-8'}>, content <{
"error": {
"code": 500,
"message": "Internal error encountered.",
"status": "INTERNAL"
}
}
>
ERROR: (gcloud.app.deploy) INTERNAL: Internal error encountered.
How can I fix this problem?

Thumbnail upload - Request forbidden by administrative rules

Following this tutorial -> https://developer.vimeo.com/api/upload/thumbnails
I have setup a time of 180s and the request of step 2 has returned status 201.
all the links have a similar format:
https://i.vimeocdn.com/video/<id>_<size>.jpg?r=pad'
Here is the request:
{ protocol: 'https:',
host: 'i.vimeocdn.com',
port: 443,
method: 'PUT',
headers:
{ 'Content-Type': 'image/jpg',
Accept: 'application/vnd.vimeo.*+json;version=3.4',
'User-Agent': 'Vimeo.js/2.1.1',
Authorization: 'Bearer 9a3918a750089766b46f0e8a59a79f49',
'Content-Length': 189 },
body:
'_readableState=&readable=true&_events=&_eventsCount=1&_maxListeners=&path=%2Ffile.jpg&fd=&flags=r&mode=438&start=&end=&autoClose=true&pos=&bytesRead=0&closed=false',
path: '/video/<id>_1920x1080.jpg' }
}
I've tried with that URI but allways get this result:
<html><body><h1>403 Forbidden</h1>
Request forbidden by administrative rules.
</body></html>
{ connection: 'close',
'cache-control': 'no-cache',
'content-type': 'text/html',
'accept-ranges': 'bytes, bytes',
via: '1.1 varnish, 1.1 varnish',
'access-control-allow-origin': '*',
date: 'Wed, 10 Jun 2020 16:59:05 GMT',
'x-served-by': 'cache-dfw18638-DFW, cache-ewr18125-EWR',
'x-cache': 'MISS, MISS',
'x-cache-hits': '0, 0',
'x-timer': 'S1591808345.129097,VS0,VE49',
'transfer-encoding': 'chunked' }
The token has the following access:
private create edit upload video_files public
I've tried to do the PUT request to the api hostname, but the page couldn't be found.
Any ideas?
Thanks in advance.

I've used the wrong link, the PUT request must be to a link that looks like this:
https://i.cloud.vimeo.com/video/<id>?expires=<timestamp>&sig=<hash>

aws boto3 paginator of list_images

can someone please help me understand and do paginator codehere in this list_images code:
When i run this to get list of images for ec2imagebuilder, in the responce i got the nexttoken so how to use this to list all images in next page/until end.
client = boto3.client('imagebuilder')
response = client.list_images(owner='Amazon')
print(response)
Response (Trucated result):
{'ResponseMetadata': {'RequestId': 'f4b9e178-b959-4e23-be57-0c234fbec69d', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Sun, 10 May 2020 21:53:55 GMT', 'content-type': 'application/json', 'content-length': '8709', 'connection': 'keep-alive', 'x-amzn-requestid': 'f4b9e178-b959-4e23-be57-0c234fbec69d', 'x-amz-apigw-id': 'MVet5Hp0PHcFeaw=', 'x-amzn-trace-id': 'Root=1-5eb877f2-9bec8ecc200f1394f6b0d340;Sampled=1'}, 'RetryAttempts': 0}, 'requestId': 'f4b9e178-b959-4e23-be57-0c234fbec69d', 'imageVersionList': [{'arn': 'arn:aws:imagebuilder:us-west-2:aws:image/amazon-linux-2-x86/2019.11.21', 'name': 'Amazon Linux 2 x86', 'version': '2019.11.21', 'platform': 'Linux', 'owner': 'Amazon', 'dateCreated': '2019-11-30T07:37:51.495Z'}, {'arn': 'arn:aws:imagebuilder:us-west-2:aws:image/windows-server-2012-r2-rtm-english-core-x86/2019.11.19', 'name': 'Windows Server 2012 R2 RTM English Core x86', 'version': '2019.11.19', 'platform': 'Windows', 'owner': 'Amazon', 'dateCreated': '2019-11-30T07:38:07.177Z'}], 'nextToken': 'eyxxxMS4xOSIsICJBY2NvdW50SWQiOiAiNTgwMDg3NjIzMDA1In0sICJtYXhfcmVzdWx0cyI6IDI1LCAia2V5X2NvbmRpdGlvbnMiOiB7IkFjY291bnRJZCI6IHsiQXR0cmlidXRlVmFsdWVMaXN0IjogWyI1ODAwODc2MjMwdddiOiBmYWxzZSwgInNjYW5faW5kZXhfZm9yd2FyZCI6IHRydWUsICJleHBpcmF0aW9uX2RhdGUiOiAxNTg5MjM0MDM1fQ=='}

Based on the documentation, you can use the following code snippet to list all images owned by Amazon.
client = boto3.client('imagebuilder')
response = client.list_images(owner='Amazon')
print(response['imageVersionList'])
while 'nextToken' in response:
response = client.list_images(owner='Amazon', nextToken=response['nextToken'])
print(response['imageVersionList'])