I'm trying to upload a file to GCS using cloud function. Api gateway is used to invoke it (POST).
The file size may vary but they are less than 32MB.
When I try to upload a file with size 10MB or above (less than 32MB), it is giving an error as 413 Request Entity Too Large: The data value transmitted exceeds the capacity limit.
Based on this article, max HTTP Request size is 32MB for Gen2 functions.
But it still fails to read the uploaded file.
Below is the sample code,
import functions_framework
#functions_framework.http
def main(request):
"""HTTP Cloud Function.
Args:
request (flask.Request): The request object.
<https://flask.palletsprojects.com/en/1.1.x/api/#incoming-request-data>
Returns:
The response text, or any set of values that can be turned into a
Response object using `make_response`
<https://flask.palletsprojects.com/en/1.1.x/api/#flask.make_response>.
"""
try:
request_json = request.get_json(silent=True)
request_args = request.args
print(request.files.get('file').content_type)
print(request.headers['Fullfilepath'])
if request_json and 'name' in request_json:
name = request_json['name']
elif request_args and 'name' in request_args:
name = request_args['name']
else:
name = 'World'
return 'Hello {}!'.format(name)
except Exception as e:
print(e)
return "Something went wrong", 500
if __name__ == "__main__":
main()
Note: If a file size is below 10MB, this works
Could someone explain why its not working with Gen2 cloud function?
EDIT:
I use postman make api call.
Below is the cURL snippet from postman for the request,
curl --location --request POST 'https://<gen2-function-url>' \
--header 'Fullfilepath: myFile.xlsx' \
--form 'file=#"/C:/Users/jithin/Downloads/myFile.xlsx"'
CURL OUTPUT (less than 10MB - working):
* Trying 216.249.35.23:443...
* TCP_NODELAY set
* Connected to <gen2-fn>
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=*.a.run.app
* start date: Sep 26 08:18:32 2022 GMT
* expire date: Dec 19 08:18:31 2022 GMT
* subjectAltName: host "<gen2-fn>" matched cert's "*.a.run.app"
* issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1C3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x6587b7c0)
> POST / HTTP/2
> Host: <gen2-fn>
> user-agent: curl/7.68.0
> accept: */*
> fullfilepath:myFile.xlsx
> content-length: 133134
> content-type: multipart/form-data; boundary=------------------------4aa0493a972a819f
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
* We are completely uploaded and fine
< HTTP/2 200
< content-type: text/html; charset=utf-8
< x-cloud-trace-context: ac66dub6c7d036958f95o05;o=1
< date: Wed, 02 Nov 2022 08:48:59 GMT
< server: Google Frontend
< content-length: 12
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
<
* Connection #0 to host <gen2-fn> left intact
Hello World!%
CURL OUTPUT (more than 10MB - NOT working):
* Trying 216.249.35.23:443...
* TCP_NODELAY set
* Connected to <gen2-fn>
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=*.a.run.app
* start date: Oct 17 08:15:34 2022 GMT
* expire date: Jan 9 08:15:33 2023 GMT
* subjectAltName: host "<gen2-fn>" matched cert's "*.a.run.app"
* issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1C3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5432697c0)
> POST / HTTP/2
> Host: <gen2-fn>
> user-agent: curl/7.68.0
> accept: */*
> fullfilepath:myFile1.xlsx
> content-length: 16492439
> content-type: multipart/form-data; boundary=------------------------d75d043c4ffd6fce
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
* We are completely uploaded and fine
< HTTP/2 500
< content-type: text/html; charset=utf-8
< x-cloud-trace-context: 464039806234sdf503247ec892803a20;o=1
< date: Wed, 02 Nov 2022 08:48:21 GMT
< server: Google Frontend
< content-length: 20
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
<
* Connection #0 to host <gen2-fn> left intact
Something went wrong%
Related
I created a bucket in s3. Static website hosting, choose Enable. I upload two html file.
page1.html
This is page1
page2.html
This is page2
I added metadata x-amz-website-redirect-location = /page2.html into page1 object in s3 website console.
When I visit http://bucket-name.s3-website.Region.amazonaws.com/page1.html on chrome. it's not redirect(it's page1 content not page2). I followed the documentation and search about this question. https://docs.aws.amazon.com/AmazonS3/latest/userguide/how-to-page-redirect.html
thanks in advance.
screenshot of page1 metadata
My bucket settings.
curl -v the site
$ curl -v https://aws-redirect-test.s3.ap-northeast-1.amazonaws.com/page1.html
* Trying 3.5.154.185...
* TCP_NODELAY set
* Connected to aws-redirect-test.s3.ap-northeast-1.amazonaws.com (3.5.154.185) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=*.s3-ap-northeast-1.amazonaws.com
* start date: Dec 9 00:00:00 2021 GMT
* expire date: Dec 2 23:59:59 2022 GMT
* subjectAltName: host "aws-redirect-test.s3.ap-northeast-1.amazonaws.com" matched cert's "*.s3.ap-northeast-1.amazonaws.com"
* issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
* SSL certificate verify ok.
> GET /page1.html HTTP/1.1
> Host: aws-redirect-test.s3.ap-northeast-1.amazonaws.com
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< x-amz-id-2: jWBm/e0Rdb2BB3R/nFffH8/YS2+f1AgXFHQfT6bUzmMK9tMZDtSNYprUp4Ka6m9xMKookshlWwo=
< x-amz-request-id: T4JG7K11X2FTBCA8
< Date: Thu, 09 Jun 2022 02:15:03 GMT
< Last-Modified: Thu, 09 Jun 2022 02:12:42 GMT
< ETag: "a12ac1ca5226842e56871deaa4d9ef9c"
< x-amz-website-redirect-location: /page2.html
< Accept-Ranges: bytes
< Content-Type: text/html
< Server: AmazonS3
< Content-Length: 14
<
This is page1
* Connection #0 to host aws-redirect-test.s3.ap-northeast-1.amazonaws.com left intact
* Closing connection 0
You are not using the website endpoint. I tested the following url and it works.
http://aws-redirect-test.s3-website-ap-northeast-1.amazonaws.com/page1.html
How we can get the messages from AWS Active MQ queue using curl command ?
curl -v -u -XGET username:password 'https://hostname:8162/api/message/lte_ap_kpi_sci?type=queue'
curl -v -XGET http://username:password#hostname:8162/api/message?destination=queue://lte_ap_kpi_sci&json=true&oneShot=true
Used above command but wasn't able to get any messages from queue.
Please find below logs
lte_ap_kpi_sci'
Note: Unnecessary use of -X or --request, GET is already inferred.
* Trying IP:PORT...
* TCP_NODELAY set
* Connected to hostname
(IP) port PORT (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=*.mq.region.amazonaws.com
* start date: Apr 18 00:00:00 2021 GMT
* expire date: May 17 23:59:59 2022 GMT
* subjectAltName: host "hostname" matched cert's "*.mq.region.amazonaws.com"
* issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
* SSL certificate verify ok.
* Server auth using Basic with user 'admin'
> GET /api/message?destination=queue://lte_ap_kpi_sci HTTP/1.1
> Host: hostname:PORT
> Authorization: Basic token==
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Date: Wed, 23 Feb 2022 18:51:08 GMT
< X-FRAME-OPTIONS: SAMEORIGIN
< Location: https://hostname:PORT/
< Transfer-Encoding: chunked
< Server: Jetty(9.4.43.v20210629)
<
* Connection #0 to host hostname left intact
Is there any possible way to read the messages from queue using curl command ?
I try to upload file to S3 by pre-singed url with curl.
It returns success when I run following command
❯ curl -v -X PUT --upload-file [file directory] '[pre-sined url]'
* Trying [port]...
* TCP_NODELAY set
* Connected to bucket-name.s3.region.amazonaws.com (ip address) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=Washington; L=Seattle; O=Amazon.com, Inc.; CN=*.region.amazonaws.com
* start date: Nov 9 00:00:00 2019 GMT
* expire date: Dec 10 12:00:00 2020 GMT
* subjectAltName: host "bukcet-name.s3.region.amazonaws.com" matched cert's "*.s3.region.amazonaws.com"
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert Baltimore CA-2 G2
* SSL certificate verify ok.
> PUT [pre-signed url] HTTP/1.1
> Host: bukcet-name.s3.region.amazonaws.com
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Type: image/png
> Content-Length: 145701
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< x-amz-id-2: hogehoge
< x-amz-request-id:hugahuga
< Date: Mon, 17 Feb 2020 08:09:01 GMT
< ETag: "hogehuga"
< Content-Length: 0
< Server: AmazonS3
<
* Connection #0 to host bukcet-name.s3.region.amazonaws.com left intact
* Closing connection 0
But When I look at S3, file is not uploaded.
I want to know how to upload file correctly to S3.
[Update]
I added x-amz-acl: bucket-owner-full-control header in curl and set <AllowedHeader>x-amz-acl</AllowedHeader> in S3 bucket CORS.
curl -v -X PUT -H 'x-amz-acl: bucket-owner-full-control' --upload-file [file directory] '[pre-sined url]'
but It returns error.
<Error><Code>AccessDenied</Code><Message>There were headers present in the request which were not signed</Message><HeadersNotSigned>x-amz-acl</HeadersNotSigned>
Also I wonder my presigned url does not have file name in directry path. Is it correct presigned url?
My implementaion to generate pre-signed url is like this:
req, _ := svc.PutObjectRequest(&s3.PutObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(key),
})
url, err := req.Presign(expires)
DO I need to add ACL inside PutObjectInput struct?
This isssue is resolved by adding file name in the end of s3 directory when it is generated.
For example(Golang):
req, _ := svc.PutObjectRequest(&s3.PutObjectInput{
Bucket: aws.String("hogehoge/fugafuga/filename"),
Key: aws.String(key),
})
url, err := req.Presign(expires)
I have a very simple lambda function that facilitates short URL redirection. Like so...
var env = process.env.NODE_ENV
exports.handler = async function (event) {
var mappings = {
"": "https://example.com",
"/": "https://example.com",
"/article1": "https://example.com/articles/article-title",
"/podcasts": "https://example.com/podcasts"
}
return {
body: null,
headers: {
"Location": mappings[event.path] || "https://example.com/four-oh-four"
},
isBase64Encoded: false,
statusCode: 301
}
}
The URL redirects just fine for all routes except the homepage (with or without a slash). Instead of the homepage, I get a "Missing Authentication Token" error from API Gateway (or Cloudfront rather).
Curling doesn't appear to reveal anything... (Updated the curl code, my bad I left the redirect).
$ curl -v https://short.url/
* Trying xxx.xx.xxx.xx...
* TCP_NODELAY set
* Connected to short.url (xxx.xx.xxx.xx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /path/to/ca-certificates.crt
CApath: /path/to/certs
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / xxxxxxxxxxxx-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=*.ib.run
* start date: Apr 5 00:00:00 2019 GMT
* expire date: May 5 12:00:00 2020 GMT
* subjectAltName: host "short.url" matched cert's "short.url"
* issuer: xxx; O=xxx; OU=xxx; CN=xxx
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle xxxxxxxx)
> GET / HTTP/2
> Host: short.url
> User-Agent: curl/7.58.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 403
< content-type: application/json
< content-length: 42
< date: Sat, 20 Jul 2019 03:51:44 GMT
< x-amzn-requestid: xxxxxxxxxx-xxxxxxxxxx-xxxxxxxxxx
< x-amzn-errortype: MissingAuthenticationTokenException
< x-amz-apigw-id: xxxxxxxxxxxxxx_
< x-cache: Error from cloudfront
< via: 1.1 xxxxxxxxxxxxxxxxxxxxxx.cloudfront.net (CloudFront)
< x-amz-cf-pop: xxxxx-xx
< x-amz-cf-id: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx===
<
* Connection #0 to host short.url left intact
{"message":"Missing Authentication Token"}
The response "Missing Authentication Token" is misleading.
It suggests that you need to provide an Token.
The real error is, that your routes in Api gateway are not setup properly.
So it is basically an Route not found from api-gateway.
You need to provide a Route for "/" with a method or the any method and redirect it to the Lambda function. You probably setup an subroute but no route for "/"
At the moment the curl is hitting the url "/" with the method GET and Api-Gateway does not know how to route this call so it answers with: "Missing Authentication Token".
You can reproduce this behavior with every non existent route. Try: /sdfsdfsdf for example. You will get the same error.
Setup the route and you shoud be fine.
I hope I could help you!
Dominik
I have a VCenter Server version 5.5. I am trying to generate a session ID to authenticate to ReST API requests using the following command:
curl -kv -X POST -H 'Accept: application/json' --basic -u me#abc.co.in:myPass! $VCENTER/rest/com/vmware/cis/session
where $VCENTER=https://vc
Here is the output I get
* Hostname was NOT found in DNS cache
* Trying 1.2.3.4...
* Connected to vc (1.2.3.4) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES256-SHA
* Server certificate:
* subject: O=VMware, Inc.; OU=vCenterServer_2014.12.24_203443; CN=VMware default certificate; emailAddress=support#vmware.com
* start date: 2014-12-24 04:44:30 GMT
* expire date: 2024-12-22 04:44:32 GMT
* issuer: O=VMware, Inc.; OU=vCenterServer_2014.12.24_203443; CN=VC.xyz.co.in; emailAddress=support#vmware.com
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Server auth using Basic with user 'me#xyz.co.in'
> POST /rest/com/vmware/cis/session HTTP/1.1
> Authorization: Basic YWthbmtzaGFfamFpbkBwZXJzaXN0ZW50LmNvLmluOmFra2FTZXAyMDE3IQ==
> User-Agent: curl/7.35.0
> Host: pt-vc
> Accept: application/json
>
< HTTP/1.1 400 Bad Request
< Date: Thu, 13 Jul 2017 09:33:40 GMT
< Connection: close
< Content-Type: text; charset=plain
< Content-Length: 0
<
* Closing connection 0
* SSLv3, TLS alert, Client hello (1):
Looking at the output I am not sure what's going wrong and where. Is this because my password has an ! which needs to be converted into its hexadecimal equivalent?
The exclamation point should be fine to pass as a password.
The issue could be due to vCenter 5.5 not having any RESTful endpoints available. That was introduced in vSphere/vCenter 6.0.
Your output would look closer to the following if you were on a 6.0 or 6.5 environment:
curl -kv -X POST -H 'Accept: application/json' --basic -u user#domain.lab:VMware1! https://vcsa01.domain.lab/rest/com/vmware/cis/session
* Trying 10.159.13.52...
* Connected to vcsa01.domain.lab (10.159.13.52) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate: VCSA01
* Server auth using Basic with user 'user#domain.lab'
> POST /rest/com/vmware/cis/session HTTP/1.1
> Host: vcsa01.domain.lab
> Authorization: Basic ZWNrQGNwYnUubGFiOlZNd2FyZTEh
> User-Agent: curl/7.43.0
> Accept: application/json
>
< HTTP/1.1 200 OK
< Date: Thu, 13 Jul 2017 18:47:18 GMT
< Set-Cookie: vmware-api-session-id=37e6921e6a3905b47ba356aaad19d3d6;Path=/rest;Secure;HttpOnly
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
<
* Connection #0 to host vcsa01.domain.lab left intact
{"value":"37e6921e6a3905b47ba356aaad19d3d6"}