```curl: (52) Empty reply from server``` ERROR response from aws ECR - amazon-web-services

I am trying to make a curl request to get ECR Authorization token instead of using aws cli.
I referred https://docs.aws.amazon.com/AmazonECR/latest/APIReference/API_GetAuthorizationToken.html
And to generate signature, I referred this : https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html
curl -v --location --request POST 'https://ecr.us-east-1.amazonaws.com:443' -H 'Accept-Encoding: identity' -H 'Content-Length: 1590' -H 'X-Amz-Target: AmazonEC2ContainerRegistry_V20150921.GetAuthorizationToken' -H 'X-Amz-Date: 20151129T221940Z' -H 'User-Agent: aws-cli/1.17.14 Python/2.7.5 Linux/3.10.0-957.1.3.el7.x86_64 botocore/1.14.14' -H 'Content-Type: application/x-amz-json-1.1' -H 'Authorization: AUTHPARAMS {AWS4-HMAC-SHA256 Credential=AKIA*******/20200514/ap-south-1/ecr/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=ffe96d25bba760d7502993a1dbf0*********************}'
Response returned after few minutes is :
* Empty reply from server
* Connection #0 to host ecr.ap-south-1.amazonaws.com left intact
curl: (52) Empty reply from server
I basically want to use this token in making a curl request to return ECR image tags:
curl -i -s -H "Authorization: Basic TOKEN" https://90********.dkr.ecr.ap-south-1.amazonaws.com/v2/image-name/tags/list
If i can request the image tags using above command without Token generation....please provide the answer.

this error means that you are receiving a 502 response error, bad gateway. I had a similar when tried to run a curl command via ssh inside an instance hosted on EC2, so I figured out that I had deployed my image with the port mappings incorrectly. I would recommend you to check the following:
Check if your security group allows traffic on port 433
Check if you have deployed your container with the port mappings correctly to ECR, like this:
suppose you have an image called list
To run in your local machine you should execute the following command
docker run -d -p 80:433 list
make sure you have mapped these ports correctly on your Dockerfile, then tag your image
docker tag list 90********.dkr.ecr.ap-south-1.amazonaws.com/v2/image-name/tags/list
and push to ECR
docker push 90********.dkr.ecr.ap-south-1.amazonaws.com/v2/image-name/tags/list
once you have your image pushed to ECR you can run it
docker run -d -p 80:433 90********.dkr.ecr.ap-south-1.amazonaws.com/v2/image-name/tags/list
and then you can curl your image
curl http://ecr-image/your/path

Related

How to check for Jupyter active notebooks through command line

I have an AWS EMR running Jupyterhub version 0.8.1+ that I want to check if there are any active notebooks that are running any code.
I've tried the below commands but they don't seem to output what I'm looking for here since the users server is always running and notebooks can be running without any code being executed.
# only lists running servers and jovyan is always running.
sudo docker exec jupyterhub jupyter notebook list
# No useful information outputted
curl -k -i -H "Accept: application/json" "https://localhost:9443/api/sessions"
# always lists processes regardless of running notebooks
ps aux | grep ipykernel
# The last_activity only updates when a user creates a new file or folder in the ui.
curl -k https://localhost:9443/hub/api/users/$user -H "Authorization: token $admin_token" | jq -r .last_activity
curl -k https://localhost:9443/hub/api/users -H "Authorization: token $admin_token" | jq -r .last_activity
Im following this AWS blog to check if the entire EMR is idle before terminating the cluster but they never seemed to have fully implemented the jupyter checks.
https://aws.amazon.com/blogs/big-data/optimize-amazon-emr-costs-with-idle-checks-and-automatic-resource-termination-using-advanced-amazon-cloudwatch-metrics-and-aws-lambda/
Most of the files referenced can be found in Github https://github.com/septian-putra/emr-monitoring
To see if a notebook is "idle" for "busy" you can run curl -ks https://localhost:9443/user/jovyan/api/kernels -H "Authorization: token ${admin_token}"
With this command all you need to do is put it in a simple if statement with a grep -q in order to get a true false idle value.
if [ $(curl -ks https://localhost:9443/user/jovyan/api/kernels -H "Authorization: token ${admin_token}" | grep -q "busy") ]; then
JUPYTER_BUSY_NOTEBOOKS=1
else
JUPYTER_BUSY_NOTEBOOKS=0
fi
(curl -ks For a silent output and to ignore ssl. jovyan being my admin user)
Documentation
https://jupyter-kernel-gateway.readthedocs.io/en/latest/websocket-mode.html#http-resources
/api/sessions might also be useful to look at.

wso2 mutual ssl curl command APIM 3.2.0

I tried setup ALLINONE in my local, followed the documentation https://apim.docs.wso2.com/en/latest/learn/api-security/api-authentication/secure-apis-using-mutual-ssl/
it worked, but what will be the curl command for request because the document talks about testing only through postman
You can use the following curl commands when you want to use the header-based approach.
curl -X GET -H "X-WSO2-CLIENT-CERTIFICATE: (Base64 encoded public cert)" "https://localhost:8243/mock/v1" -v
curl -X GET -H "X-WSO2-CLIENT-CERTIFICATE: (Base64 encoded public cert)" "http://localhost:8280/mock/v1" -v
In order to work this, you need to add the following configuration to the deployment.toml in wso2am-3.2.0/repository/conf location.
[apimgt.mutual_ssl]
enable_client_validation = false
You can use the following curl commands if you are using the cert and key.
curl -k --cert int.ext.wso2.com.crt --key int.ext.wso2.com.key -X GET "https://localhost:8243/mock/v1" -v

How to establish a connection to AWS Device farm Remote session endpoint

I tried to create AWS Device farm remote session, which i am able to do it successfully. The response JSON of the created remote session has an endpoint (wss) and the hostAddress(IP). Rather then login into AWS device farm to interact with the device. I wanted to provide remote access directly on my own web page (I am not sure whether its possible). Hoping it can be rendered under a canvas tag.
Though i do not have experience on the socket, i just tried some sample code to connect with the received Web Socket URL.
var wsUri = "wss://devicefarm-interactive.us-west-2.amazonaws.com/?X-Amz-Date=*&X-Amz-Credential=*&X-Amz-Algorithm=*&X-Amz-SignedHeaders=host&X-Amz-Signature=*&X-Amz-Security-Token=*";
var websocket = new WebSocket(wsUri);
websocket.send('ping');
Below is the error the console.
Connection closed before receiving a handshake response.
Any sample links to implement would be helpful
I'm able to reproduce that error. I looked for examples of using the web socket connection using the networking tab after inspecting the page in chrome(since it has a web socket debugger built into it) which will show the following:
Basically how that appears to be working is it's sending us a constant stream of images.
When I copy as cURL command this is the result
curl 'wss://devicefarm-interactive.us-west-2.amazonaws.com/?X-Amz-Date=20190518T211708Z&X-Amz-Credential=ASIAIGY76PSQXN5NZ3UA%2F20190518%2Fus-west-2%2Fdevicefarm%2Faws4_request&X-Amz-Algorithm=AWS4-HMAC-SHA256&arn=arn%3Aaws%3Adevicefarm%3Aus-west-2%111122223333%3Asession%3A8f4af46d-8f86-4dcc-8324-e691ce3723f3%2F6b4dc632-188f-45b9-be32-3e9aa6881ed3%2F00000&X-Amz-SignedHeaders=host&X-Amz-Signature=39b45e1489d44dfcb904d58e59e985844416061a828aa75750d5a67db36c55dd&X-Amz-Security-Token=someTokenValue&path=video' -H 'Pragma: no-cache' -H 'Origin: https://us-west-2.console.aws.amazon.com' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en-US,en;q=0.9' -H 'Sec-WebSocket-Key: 60RGfCwWuULib6NmeoC2fA==' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36' -H 'Upgrade: websocket' -H 'Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits' -H 'Cache-Control: no-cache' -H 'Connection: Upgrade' -H 'Sec-WebSocket-Version: 13' --compressed
When I did the following CLI command I get the endpoint back:
aws devicefarm get-remote-access-session --arn arn:aws:devicefarm:us-west-2:111122223333:session:8f4af46d-8f86-4dcc-8324-e691ce3723f3/6b4dc632-188f-45b9-be32-3e9aa6881ed3/00000 --region us-west-2 --query remoteAccessSession.endpoint
"wss://devicefarm-interactive.us-west-2.amazonaws.com/?X-Amz-Date=20190518T212323Z&X-Amz-Credential=ASIAIU5CA7GBP5IEBR6Q%2F20190518%2Fus-west-2%2Fdevicefarm%2Faws4_request&X-Amz-Algorithm=AWS4-HMAC-SHA256&arn=arn%3Aaws%3Adevicefarm%3Aus-west-2%3A111122223333%3Asession%3A8f4af46d-8f86-4dcc-8324-e691ce3723f3%2F6b4dc632-188f-45b9-be32-3e9aa6881ed3%2F00000&X-Amz-SignedHeaders=host&X-Amz-Signature=aaaaqaqbbbb4e9f09b7312715c295a11b77bc0d9e7b21dcb61422a61f78a1f&X-Amz-Security-Token=FQoGZXIvYXdzEFcaDD2E90%2Bsp3i%2F%2F8cBbyKDA2EGKkFYSvXDR%2Fb7LfS%2FpQEPCWFVhe9eCOTSussvshjldx69CEFvVgV3JYtOvm2yu0UMVAxlDYlujvpMfSNwLx7FH%2B42k9qGYuvy5dQbVLg%2F%2BCRuyK9OjCxpD5pUfQ9b81U6LawcI2I1CekXeTgapRuTK9tCPcGtNOlxAvWQUVlyDGTtmqjz7vRlostquMoenNr9UB1v8jx0NSo1YIlrgY8YvZV2o5pcbYiI9I9CBD0%2F3snJZAyQtmPZkMT9gr9hI0jgX1X5MlOuarFmm%2F2Sn%2FH8L3ewMQXhvuho3OTNZTISBmUgJAbZSmQcazuDmjXqPkoNpYYcUb92vd2w5MbRfFSa5SHHXUMVcE5Wsop3BzwJyj%2FNyl59BdjFWdo82NgSFP6OBjYLjiux3hR2dx86ILJ9tfNMNfq0WXzL3Z%2BqecwMTxlxrLfZmPftsUDaO5RPtOP9uuI%2BPjfIOWOV7uFy9GjKG4HKFY%2BZVGgWhb1fVVG7%2BYHbPxgMaAKI3YJqmM9IIy8%2FdCjL74HnBQ%3D%3D%7CMjA1LjI1MS4yMzMuMTc5"
which appears to be signed. Using wscat I tried to connect to it.
endpoint=$(aws devicefarm get-remote-access-session --arn arn:aws:devicefarm:us-west-2:111122223333:session:8f4af46d-8f86-4dcc-8324-e691ce3723f3/6b4dc632-188f-45b9-be32-3e9aa6881ed3/00000 --region us-west-2 --query remoteAccessSession.endpoint)
wscat -c $endpoint
Error:
/usr/local/lib/node_modules/wscat/node_modules/ws/lib/websocket.js:455
throw new Error(Invalid URL: ${this.url});
I'll need to spend some more time on this later but I think some of this content is helpful so I posted it.

Cannot access /encrypt endpoint of PCF p-config-server service

I have followed the instructions from https://github.com/spring-cloud-services-samples/cook and managed to install and run Config Server in PCF environment (SERVICE: Config Server, PLAN: standard).
I'm now trying to hit /encrypt endpoint of the p-config-server service, in order to encrypt new value. I'm following the instructions at http://docs.run.pivotal.io/spring-cloud-services/config-server/configuring-with-git.html#encryption-and-encrypted-values:
TOKEN=$(curl -k ACCESS_TOKEN_URI -u CLIENT_ID:CLIENT_SECRET -d
grant_type=client_credentials | jq -r .access_token); curl -k -H
"Authorization: bearer $TOKEN" -H "Accept: application/json"
URI/encrypt -d 'VALUE'
...but I always get:
{
"error": "access_denied",
"error_description": "Access is denied"
}
On the other side, if I try to get standard endpoint, to get config for an app, I'm able to retrieve JSON containing app properties. E.g.
TOKEN=$(curl -k ACCESS_TOKEN_URI -u CLIENT_ID:CLIENT_SECRET -d
grant_type=client_credentials | jq -r .access_token); curl -k -H
"Authorization: bearer $TOKEN" -H "Accept: application/json"
URI/my-app/default
... gives result like:
{"name":"my-app","profiles":["default"],"label":null,"version":"bb6e64592ced731ebba272430291a595e0f14a77","state":null,"propertySources":[{"name":"https://github.com/some-user/config/my-app.yml","source":{"my-property.name":"Test123"}}]}
Any idea why I can not access /encrypt endpoint?
Thank you.
Btw, I'm executing the command in CentOS Linux release 7.4.1708, with installed jq (command-line JSON processor).
I've got the answer from CloundFoundry IT support. In my CF environment, "encrypt" endpoint should have a trailing slash (/). So it should be ...URI/encrypt/ -d 'VALUE'. Maybe it helps someone.
One more hint I've got: There is a CF CLI plugin for the Spring-Cloud-Services which you could use for convenience.
https://github.com/pivotal-cf/spring-cloud-services-cli-plugin
cf install-plugin -r CF-Community "Spring Cloud Services"
cf help config-server-encrypt-value
Hi Actually you need to hit cf env command first and take note of configuration values from that which for sample looks like below:
{
"VCAP_SERVICES": {
"p-config-server": [
{
"credentials": {
"access_token_uri": "https://p-spring-cloud-services.uaa.cf.wise.com/oauth/token",
"client_id": "p-config-server-876cd13b-1564-4a9a-9d44-c7c8a6257b73",
"client_secret": "rU7dMUw6bQjR",
"uri": "https://config-86b38ce0-eed8-4c01-adb4-1a651a6178e2.apps.wise.com"
},
[...]
and then use those values in your curl bash script. for example
TOKEN=$(curl -k https://config-86b38ce0-eed8-4c01-adb4-1a651a6178e2.apps.wise.com -u p-config-server-876cd13b-1564-4a9a-9d44-c7c8a6257b73:rU7dMUw6bQjR -d
grant_type=client_credentials | jq -r .access_token); curl -k -H
"Authorization: bearer $TOKEN" -H "Accept: application/json"
URI/ENDPOINT | jq
Basically following values are required:
ACCESS_TOKEN_URI with the value of credentials.access_token_uri
CLIENT_ID with the value of credentials.client_id
CLIENT_SECRET with the value of credentials.client_secret
URI with the value of credentials.uri
Replace ENDPOINT with the relevant endpoint:
application/profile to retrieve configuration from a Config Server service instance
eureka/apps to retrieve the registry from a Service Registry service instance
Then I think you will no more get access denied response.

How to test web service using command line curl

I am building a web service for a web application, and I would like a simple tool to test this as I am developing. I have tried some firefox plug-ins (Poster, 'REST Client'), and even though these work fine I have been unable to upload files with them.
Also, I would rather have a command-line tool that I can use to easily write a set of integration tests for this web service and that I can send to consumers of this web service as an example.
I know that curl can work for this but would like a few examples, especially around authentication (using HTTP Basic) and file uploads.
Answering my own question.
curl -X GET --basic --user username:password \
https://www.example.com/mobile/resource
curl -X DELETE --basic --user username:password \
https://www.example.com/mobile/resource
curl -X PUT --basic --user username:password -d 'param1_name=param1_value' \
-d 'param2_name=param2_value' https://www.example.com/mobile/resource
POSTing a file and additional parameter
curl -X POST -F 'param_name=#/filepath/filename' \
-F 'extra_param_name=extra_param_value' --basic --user username:password \
https://www.example.com/mobile/resource
In addition to existing answers it is often desired to format the REST output (typically JSON and XML lacks indentation). Try this:
$ curl https://api.twitter.com/1/help/configuration.xml | xmllint --format -
$ curl https://api.twitter.com/1/help/configuration.json | python -mjson.tool
Tested on Ubuntu 11.0.4/11.10.
Another issue is the desired content type. Twitter uses .xml/.json extension, but more idiomatic REST would require Accept header:
$ curl -H "Accept: application/json"
From the documentation on http://curl.haxx.se/docs/httpscripting.html :
HTTP Authentication
curl --user name:password http://www.example.com
Put a file to a HTTP server with curl:
curl --upload-file uploadfile http://www.example.com/receive.cgi
Send post data with curl:
curl --data "birthyear=1905&press=%20OK%20" http://www.example.com/when.cgi