I added login: Admin to app.yaml like documentation is described, but I still have a 302 erro when runing a task with cron in google app engine.
I have this app.yaml
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /.*
script: main.app
login: admin
And this cron.yaml
cron:
- description: "Dashboard"
url: /processdate?from=2016-03-01&until=2016-03-31
schedule: every day 23:46
timezone: Europe/Madrid
I obtain this error
0.1.0.1 - - [26/Oct/2018:00:49:40 +0200] "GET /processdate?from=2016-03-01&until=2016-03-31 HTTP/1.1" 302 355 - "AppEngine-Google; (+http://code.google.com/appengine)" "p20000.appspot.com" ms=74 cpu_ms=12 cpm_usd=3.9674e-8 loading_request=0 instance=00c61b117c78f767097d6896daa1f8967a815c14a94d54578ac19efa9d50a5077d5a app_engine_release=1.9.65 trace_id=3c92edad090b5a57d249bd92be246e58
httpRequest: {
status: 302
}
insertId: "5bd248840005a3aae7fa2111"
labels: {
clone_id: "00c61b117c78f767097d6896daa1f8967a815c14a94d54578ac19efa9d50a5077d5a"
}
logName: "projects/p201309/logs/appengine.googleapis.com%2Frequest_log"
operation: {
first: true
id: "5bd2488400ff047fe69ec5d94d0001657e62692d70682d3230313330390001323031383130323674303033383339000100"
last: true
producer: "appengine.googleapis.com/request_id"
}
protoPayload: {
#type: "type.googleapis.com/google.appengine.logging.v1.RequestLog"
appEngineRelease: "1.9.65"
appId: "e~myappname"
cost: 3.9674e-8
endTime: "2018-10-25T22:49:40.369327Z"
finished: true
first: true
host: "p200000.appspot.com"
httpVersion: "HTTP/1.1"
instanceId: "00c61b117c78f767097d6896daa1f8967a815c14a94d54578ac19efa9d50a5077d5a"
instanceIndex: -1
ip: "0.1.0.1"
latency: "0.074441s"
megaCycles: "12"
method: "GET"
requestId: "5bd2488400ff047fe69ec5d94d0001657e62692d70682d3230313330390001323031383130323674303033383339000100"
resource: "/processdate?from=2016-03-01&until=2016-03-31"
responseSize: "355"
startTime: "2018-10-25T22:49:40.294886Z"
status: 302
taskName: "25ed634cde05b07d9a7906f2161d2b16"
taskQueueName: "__cron"
traceId: "3c92edad090b5a57d249bd92be246e58"
traceSampled: true
urlMapEntry: "main.app"
userAgent: "AppEngine-Google; (+http://code.google.com/appengine)"
versionId: "20181026t003839"
}
receiveTimestamp: "2018-10-25T22:49:40.376251430Z"
resource: {
labels: {
module_id: "default"
project_id: "myappname"
version_id: "20181026t003839"
zone: "eu2"
}
type: "gae_app"
}
timestamp: "2018-10-25T22:49:40.294886Z"
trace: "projects/myappname/traces/3c92edad090b5a57d249bd92be246e58"
traceSampled: true
}
Any idea about what could be the problem or how to solve it?
I don't have any problem runing this script in my local machine.
------ UPDATED WITH A MUCH MORE SIMPLE APP ------
I tested something much more simple. This is working when I run https://myappname.appspot.com/hellocron and http://myappname.appspot.com/hellocron
With this the result it's what expected.
But when I run it with cron error 302 it's returned again
----------- app.yaml
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /hellocron
script: main.app
login: admin
secure: always
----------- cron.yaml
cron:
- description: "hellocron"
url: /hellocron
schedule: every day 23:46
timezone: Europe/Madrid
------------ I had the same error result
0.1.0.1 - - [01/Nov/2018:12:29:49 +0100] "GET /hellocron HTTP/1.1" 302 267 - "AppEngine-Google; (+http://code.google.com/appengine)" "myappname.appspot.com" ms=8 cpu_ms=9 cpm_usd=2.9839e-8 loading_request=0 instance=00c61b117cb863320ce80ff59a2b5b4b20ee440529428f43d612baa0e980733727302b27 app_engine_release=1.9.65 trace_id=9b9fa8ad127ad41f7907529c3863a0a9
{
httpRequest: {
status: 302
}
insertId: "5bdae3ad0004cfe52d64d457"
labels: {
clone_id: "00c61b117cb863320ce80ff59a2b5b4b20ee440529428f43d612baa0e980733727302b27"
}
logName: "projects/myappname/logs/appengine.googleapis.com%2Frequest_log"
operation: {
first: true
id: "5bdae3ad00ff04ac4decaecea60001657e62692d70682d3230313330390001323031383131303174313135363535000100"
last: true
producer: "appengine.googleapis.com/request_id"
}
protoPayload: {
#type: "type.googleapis.com/google.appengine.logging.v1.RequestLog"
appEngineRelease: "1.9.65"
appId: "e~myappname"
cost: 2.9839e-8
endTime: "2018-11-01T11:29:49.315161Z"
finished: true
first: true
host: "myappname.appspot.com"
httpVersion: "HTTP/1.1"
instanceId: "00c61b117cb863320ce80ff59a2b5b4b20ee440529428f43d612baa0e980733727302b27"
instanceIndex: -1
ip: "0.1.0.1"
latency: "0.008908s"
megaCycles: "9"
method: "GET"
requestId: "5bdae3ad00ff04ac4decaecea60001657e62692d70682d3230313330390001323031383131303174313135363535000100"
resource: "/hellocron"
responseSize: "267"
startTime: "2018-11-01T11:29:49.306253Z"
status: 302
taskName: "b0467e8a57f53a8ee2b827ca35db275f"
taskQueueName: "__cron"
traceId: "9b9fa8ad127ad41f7907529c3863a0a9"
traceSampled: true
urlMapEntry: "main.app"
userAgent: "AppEngine-Google; (+http://code.google.com/appengine)"
versionId: "20181101t115655"
}
receiveTimestamp: "2018-11-01T11:29:49.321937019Z"
resource: {
labels: {
module_id: "default"
project_id: "myappname"
version_id: "20181101t115655"
zone: "eu2"
}
type: "gae_app"
}
timestamp: "2018-11-01T11:29:49.306253Z"
trace: "projects/myappname/traces/9b9fa8ad127ad41f7907529c3863a0a9"
traceSampled: true
}
The code for /hellocron in python is this one basically:
decorator = OAuth2DecoratorFromClientSecrets(
os.path.join(os.path.dirname(__file__), 'client_secrets.json'),
scope='https://www.googleapis.com/auth/bigquery')
class hellocron (webapp2.RequestHandler):
#decorator.oauth_required
def get(self):
self.response.write('hellocron')
app = webapp2.WSGIApplication([
('/hellocron', hellocron),
(decorator.callback_path, decorator.callback_handler()) ], debug=True)
You want to drop the #decorator.oauth_required from the cron url handler code.
The cron service does not have any user credentials (it doesn't run as a user) so that decorator will cause a re-direction to a login service - hence the 302 response. You should be able to verify this by re-trying your manual check but from an incognito browser window.
To secure the cron service URLs you can't use regular user authentication for this reason. But you can use login: admin in the app.yaml file and, if you want, also check for the X-Appengine-Cron: true header or the 0.1.0.1 source IP address, see Securing URLs for cron.
Similarly you might need to drop the secure: always from the respective app.yaml handler definition (I don't have it enabled for my app): I'm not sure if the cron service makes its GET request using http or https. If it makes it using http the secure: always config will also cause a redirection to a https URL. You can easily check if this is correct or not following my earlier comment after you drop the decorator.
My name is Dan I'm from google cloud support.
As I could understand, you're receiving a 302 response when launching Cron jobs.
Have you a custom domain with SSL enabled? I ask you this because we have received this report from some customers that are using HTTP instead of HTTPS.
Are you running your application on App Engine flexible? This information will help me to better understand your current scenario.
I will be awaiting your reply.
I wanted to post this solution as I was pulling my hair out yesterday trying to figure out why my GAE cron jobs kept failing and this thread helped me fine tune the solution that worked.
I am using Python, but the idea seems the same. In my cron script (i.e. the page visited with the cron logic), I had a redirect after successful execution of a report, due to this the cron would state it failed in GAE, but the logic would still execute correctly.
To fix the issue, all I did was return a simple HTTP response so the cron thought everything loaded correctly on the page instead of the redirect or not returning an HTTP response all together.
I was trying to execute a django admin endpoint from gcloud app engine. It seems there is no way to make gcloud crons login with any credentials before running so here is what you have to do if you find yourself in this situation (receiving 302 in your crons executions):
Make the url public, that means: no credentials needed to run it.
Gcloud crons send request including X-AppEngine-Cron in it's headers. They strip 'X-' from all their headers so you may trust this particular one. All you have to do is check for this header in your endpoint request as this guy does.
Doc: https://cloud.google.com/appengine/docs/flexible/go/scheduling-jobs-with-cron-yaml?hl=es-419#validating_cron_requests
Related
I am using aws Instance and I am trying to run promtail in order to fetch logs and forward it to loki server. Promtail, Loki and Grafana are being run through Docker.
The Loki Server is runing on port 3100, Promtail on 3400 and Loki on 8001. Since It is an AWS platform what needs to be done so that It stops throwing error at http://43.206.43.87:3100/loki/api/v1/push end point.
here is my promtail-config.yaml
server:
http_listen_port: 3400
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://43.206.43.87:3100/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- 43.206.43.87
labels:
job: varlogs
__path__: /var/log/*log
Here is my loki-config.yaml
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 0
common:
path_prefix: /tmp/loki
storage:
filesystem:
chunks_directory: /tmp/loki/chunks
rules_directory: /tmp/loki/rules
replication_factor: 1
ring:
instance_addr: 43.206.43.87
kvstore:
store: inmemory
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
ruler:
alertmanager_url: http://localhost:9093
Please help me out
All I had to do was change the loki-config.yaml file in that
I had to write
instance_addr: localhost
and everything worked
Hi to all I created and used openAPI by yaml and I created endpoint that maps 2 cloud functions which use path templating to route the call no error by google sdk cli.
Now I call by POST https://myendpointname-3p5hncu3ha-ew.a.run.app/v1/setdndforrefcli/12588/dnd?key=[apikey] because it's mapped by below open api and reply me "Path does not match any requirement URI template.".
I don't know why path template in endpoint not work I added path_translation: APPEND_PATH_TO_ADDRESS to avoid google to use CONSTANT_ADDRESS default which append id in query string with brutal [name of cloud function]?GETid=12588 and overwrite query parameters with same name.
Somebody can tell me how can I debug the endpoint or the error in openAPI (that have green check ok icon in endpoint)?
# [START swagger]
swagger: '2.0'
info:
description: "Get data "
title: "Cloud Endpoint + GCF"
version: "1.0.0"
host: myendpointname-3p5hncu3ha-ew.a.run.app
# [END swagger]
basePath: "/v1"
#consumes:
# - application/json
#produces:
# - application/json
schemes:
- https
paths:
/setdndforrefcli/{id}/dnd:
post:
summary:
operationId: setdndforrefcli
parameters:
- name: id # is the id parameter in the path
in: path # is the parameter where is in query for rest or path for restful
required: true
type: integer
format: int64
minimum: 1
security:
- api_key: []
x-google-backend:
address: https://REGION-PROJECT-ID.cloudfunctions.net/mycloudfunction
path_translation: APPEND_PATH_TO_ADDRESS
protocol: h2
responses:
'200':
description: A successful response
schema:
type: string
# [START securityDef]
securityDefinitions:
# This section configures basic authentication with an API key.
api_key:
type: "apiKey"
name: "key"
in: "query"
# [END securityDef]
I had the same error, but after did some test I was able to use successfully the path templating (/endpoint/{id}). I resolved this issue as follows:
1 .- gcloud endpoints services deploy openapi-functions.yaml \
--project project
Here you will get a new Service Configuration that you will to use in the next steps.
2.-
chmod +x gcloud_build_image
./gcloud_build_image -s SERVICE \
-c NEWSERVICECONFIGURATION -p project
Its very important change the service configuration with every new deployment of the managed service.
3.- gcloud run deploy SERVICE \
--image="gcr.io/PROJECT/endpoints-runtime-serverless:SERVICE-NEW_SERVICE_CONFIGURATION" \
--allow-unauthenticated \
--platform managed \
--project=PROJECT
I'm using Cloud Run, Endpoints and Cloud Functions to build an API service. There are multiple endpoints running completely fine, but I'm no longer able to deploy any new endpoints.
The Cloud Run environment has an error that prevents it from making a call to the corresponding Cloud Function. Oddly enough, all other endpoints work fine, but I'm unable to create new endpoints.
I found this article: https://cloud.google.com/endpoints/docs/openapi/troubleshoot-response-errors but it's only for the BAD_GATEWAY error code.
All code is deployed completely fine. No errors in deploying the Cloud Function, Cloud Run or Open API yaml file.
Error in response:
{
"code": 13,
"message": "INTERNAL_SERVER_ERROR",
"details": [
{
"#type": "type.googleapis.com/google.rpc.DebugInfo",
"stackEntries": [],
"detail": "application"
}
]
}
Error in Cloud Run:
5#5: *33 invalid URL prefix in "", client: xxxxx, server: , request: "GET /user HTTP/1.1", host: "[my cloud run host]"
GET 500 404 B4 ms python-requests/2.22.0 [cloud run host]/user
The main.py file:
def user(request):
return "Ok"
The yaml file:
/user:
x-google-backend:
address: https://[cloud functions host]/user
get:
summary: Retrieves a user.
operationId: getUser
responses:
'200':
description: A successful response
'400':
description: BAD_REQUEST
If we look at your YAML:
/user:
x-google-backend:
address: https://[cloud functions host]/user
get:
summary: Retrieves a user.
operationId: getUser
responses:
'200':
description: A successful response
'400':
description: BAD_REQUEST
... pay particular attention to the x-google-backend section. Notice that this exists within the /user path section. Now notice that the address is a URL with a path. You don't want a path part in the URL, just the address of the host (and optional port). Change your start of your YAML to:
/user:
x-google-backend:
address: https://[cloud functions host]
(The /user portion was removed from the address line)
I'm kinda of new in EG. I have followed the documentation about issuing a jwt token instead of opaque but still receiving an opaque access token. Not sure what I'm missing to change.
This is my system.config file
db:
redis:
host: localhost
port: 6379
namespace: EG
crypto:
cipherKey: sensitiveKey
algorithm: aes256
saltRounds: 10
session:
secret: keyboard cat
resave: false
saveUninitialized: false
accessTokens:
timeToExpiry: 7200000
tokenType: 'jwt'
issuer: 'express-gateway'
audience: 'something'
subject: 'test'
secretOrPrivateKey: 'ssssst'
refreshTokens:
timeToExpiry: 7200000
authorizationCodes:
timeToExpiry: 300000
Already added a user and app with their own credentials (oauth2, basic-auth, jwt) without changing anything on the models.
this is my gateway.config file
http:
port: 8080
admin:
port: 9876
hostname: localhost
apiEndpoints:
api:
host: localhost
paths: '/api/*'
serviceEndpoints:
httpbin:
url: 'https://httpbin.org'
policies:
- jwt
- oauth2
- proxy
- rate-limit
pipelines:
default:
apiEndpoints:
- api
policies:
- oauth2:
action:
jwt:
issuer: express-gateway
audience: something
subject: test
secretOrPublicKey: ssssst
checkCredentialExistence: false
- proxy:
- action:
serviceEndpoint: httpbin
changeOrigin: true
the request for login is like this and secret is the keySecret generated with jwt credential of the app.
http://localhost:8080/oauth2/authorize?response_type=token&client_id=ae921ba9-7b4b-4c53-aaba-354bd6398e52&redirect_uri=http://localhost:3002/explorer&client_secret=0qOpBZkwO2ayQ8dO18yRuh
and the token response looks like this.
14070f7c4ffc49efb1fc1709cc4a7267|90c6efd2cf8342859756d8e3705417a3
Thanks for anyone who can help me.
The configuration looks correct. I've also personally tried the configuration you provided and I got back a JWT
Maybe you can try to set up a test project on Glitch.me so it's going to be easier to replicate locally.
I am running the predict function through the API Call against a model deployed in Cloud MLEngine and I have enabled logging so that I can see the logs in Stackdriver Logging.
When I go to the logs I see the logs generated are like this
{
httpRequest: {
latency: "0.002585s"
remoteIp: "X.X.X.X"
requestMethod: "POST"
requestSize: "259"
requestUrl: "ml.googleapis.com/v1/projects/<project-id>/models/pricing_model/versions/v1:predict"
responseSize: "56"
status: 200
userAgent: "google-cloud-sdk gcloud/202.0.0 command/gcloud.beta.ml-engine.predict invocation-id/XXXXXXXXX environment/None environment-version/None interactive/True from-script/False python/2.7.14 (Macintosh; Intel Mac OS X 16.7.0),gzip(gfe)"
}
insertId: "qd41"
jsonPayload: {
#type: "type.googleapis.com/google.cloud.ml.v1.PredictionLogEntry"
modelLatency: "0.000507s"
numInstances: "1"
requestType: "LOADED"
}
logName: "projects/<project-id>/logs/ml.googleapis.com"
receiveTimestamp: "2018-05-23T20:51:42.980326893Z"
resource: {
labels: {…}
type: "cloudml_model_version"
}
severity: "INFO"
timestamp: "2018-05-23T20:51:42Z"
}
This does not include the request and response. Is there a way I can add that parameter in so that whenever I run predict against the model in ML Engine, I can also see the request and response?
I would like to see two more fields in the logs like this
request:[1068,34,45,89,3245,543543]
prediction(or Response):[34354.43]