I want to create an index pattern using Opensearch API. I tried to replicate what could be made graphically in the following image window, using as index pattern name cwl-* and then as time field #timestamp.
My domain has OpenSearch 1.2 installed.
Using curl (directly modifiend the command in kibana doc):
curl -u '****:*****' -X POST "https://******.eu-central-1.es.amazonaws.com/api/index_patterns/index_pattern" -H 'osd-xsrf: true' -H 'Content-Type: application/json' -d'
{
"index_pattern": {
"title": "cwl-*",
"timeFieldName": "#timestamp"
}
}'
but I receive
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Rejecting mapping update to [api] as the final mapping would have more than 1 type: [_doc, index_patterns]"}],"type":"illegal_argument_exception","reason":"Rejecting mapping update to [api] as the final mapping would have more than 1 type: [_doc, index_patterns]"},"status":400}
curl -u '****:*****' -X POST "https://******.eu-central-1.es.amazonaws.com/api/index_patterns/cwl-*" -H 'osd-xsrf: true' -H 'Content-Type: application/json' -d'
{
"index_pattern": {
"title": "cwl-*",
"timeFieldName": "#timestamp"
}
}'
change api/index_patterns/index_pattern to api/index_patterns/cwl-* and try again?
It worked for me in OpenSearch 1.3 when I added an ID in the URI and used saved_objects instead of index_patterns.
So your cURL-request should work when looking like this.
curl -u '****:*****' -X POST "https://<opensearch-dashboards-host>.eu-central-1.es.amazonaws.com/api/saved_objects/index-pattern/<ID>"
-H 'osd-xsrf: true'
-H 'Content-Type: application/json'
-d
'{
"index_pattern": {
"title": "cwl-*",
"timeFieldName": "#timestamp"
}
}'
I am experiencing major performance issues when calling a stored procedure via PostgREST compared when calling it via psql. Both methods complete successfully, but the PostgREST takes ~1 min compared to 2 seconds with psql. There are many statement levels and a few row-level triggers fired as a result of this procedure.
This procedure updates the database, another update procedure has the same issues,
Calling from psql:
set role myrole;
start transaction;
WITH pgrst_source AS (WITH pgrst_args AS (SELECT NULL) SELECT "api"."docket_creation"('{"dockets":[{"docket":{},"transactions":[{"transaction":{"batch_id":null,"best_before_date":null,"docket_id":1417,"minimum_price":1.5,"note":"","price":3,"product_code":"TL12","product_cost":1,"product_description":"","product_group_id":null,"product_id":217,"product_name":"12 Tray Large","quantity":1,"vat_rate_id":4,"vat_rate_percentage":0,"return_reason_id":null,"transaction_type_id":2},"transaction_discounts":[{"discount_percentage":5,"discount_type_id":10,"discount_value_per_unit":null,"minimum_quantity":null,"note":""}],"transaction_returns":[],"transaction_commissions":[{"commission_id":null,"commission_percentage":0,"commission_type_id":2,"customer_id":1,"delivery_agent_id":0,"effective_from":null,"effective_to":null,"is_cumulative":true,"owner":2,"product_id":217}]}],"docket_buy_x_get_y_discounts":[]}]}'::json) AS pgrst_scalar)
SELECT
null::bigint AS total_result_set,
pg_catalog.count(_postgrest_t) AS page_total,
(json_agg(_postgrest_t.pgrst_scalar)->0)::character varying AS body,
coalesce(nullif(current_setting('response.headers', true), ''), '[]') AS response_headers
FROM (SELECT "pgrst_source".* FROM "pgrst_source" ) _postgrest_t;
Or simply:
SELECT "api"."docket_creation"('{"dockets":[{"docket":{},"transactions":[{"transaction":{"batch_id":null,"best_before_date":null,"docket_id":1417,"minimum_price":1.5,"note":"","price":3,"product_code":"TL12","product_cost":1,"product_description":"","product_group_id":null,"product_id":217,"product_name":"12 Tray Large","quantity":1,"vat_rate_id":4,"vat_rate_percentage":0,"return_reason_id":null,"transaction_type_id":2},"transaction_discounts":[{"discount_percentage":5,"discount_type_id":10,"discount_value_per_unit":null,"minimum_quantity":null,"note":""}],"transaction_returns":[],"transaction_commissions":[{"commission_id":null,"commission_percentage":0,"commission_type_id":2,"customer_id":1,"delivery_agent_id":0,"effective_from":null,"effective_to":null,"is_cumulative":true,"owner":2,"product_id":217}]}],"docket_buy_x_get_y_discounts":[]}]}')
This takes about 2 secs.
When calling from a webapp or via curl with the call
curl 'https://<server>/hhdb_api/rpc/docket_creation' \
-X 'POST' \
-H 'Connection: keep-alive' \
-H 'sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"' \
-H 'Prefer: params=single-object' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'Authorization: Bearer $bearer_token' \
-H 'Content-Type: application/json' \
-H 'Accept: application/json, text/plain, */*' \
-H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36' \
-H 'DNT: 1' \
-H 'Origin: https://hh-tincan-dev.srv.handheld.ie' \
-H 'Sec-Fetch-Site: same-site' \
-H 'Sec-Fetch-Mode: cors' \
-H 'Sec-Fetch-Dest: empty' \
-H 'Referer: https://hh-tincan-dev.srv.handheld.ie/' \
-H 'Accept-Language: en-GB,en-US;q=0.9,en;q=0.8' \
--data-raw '{"dockets":[{"docket":{},"transactions":[{"transaction":{"batch_id":null,"best_before_date":null,"docket_id":1428,"minimum_price":1.5,"note":"","price":3,"product_code":"TL12","product_cost":1,"product_description":"","product_group_id":null,"product_id":217,"product_name":"12 Tray Large","quantity":1,"vat_rate_id":4,"vat_rate_percentage":0,"return_reason_id":null,"transaction_type_id":2},"transaction_discounts":[{"discount_percentage":5,"discount_type_id":10,"discount_value_per_unit":null,"minimum_quantity":null,"note":""}],"transaction_returns":[],"transaction_commissions":[{"commission_id":null,"commission_percentage":0,"commission_type_id":2,"customer_id":1,"delivery_agent_id":0,"effective_from":null,"effective_to":null,"is_cumulative":true,"owner":2,"product_id":217}]}],"docket_buy_x_get_y_discounts":[]}]}' \
--compressed
This will take over a minute.
I would expect both of these invocations to take a similar time.
I have enabled extensive logging to try to track this down the difference between the two methods seems to be the delays when entering a nested trigger
~
I have raised this as an issue on the PostgREST's GitHub with some configuration details:
https://github.com/PostgREST/postgrest/issues/1836
This issue is now solved please refer to the PostgREST github issue for details
I have a curl link which is successfully run with terminal but i want to convert it as a POSTMAN request where link is,
curl -v --silent -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X GET -u <registry-user>:<registry-password> https://sc-docker-registry.eic.fullstream.ai/v2/<image-name>/manifests/<tag> 2>&1 --insecure | grep Docker-Content-Digest | awk '{print ($3)}'
I have set header and auth but unable to set 2>&1 --insecure | grep Docker-Content-Digest | awk '{print ($3)}'
Where do I set 2>&1 --insecure | grep Docker-Content-Digest | awk '{print ($3)}' command in postman
I had the issue when using "Copy all as cURL (bash)" instead of just "Copy as cURL (bash)".
A few concatenated curl commands were copied. And Postman's import didn't like that.
I had the same issue but chrome now gives you the option to copy curl as (bash) or (cmd). I had the issue when using copy as cmd but worked when I used copy as bash
I had the same error and I think it's unrelated to the pipeline arguments you sent at the end of the curl. Enclosing url part into double quotes helped with my case. Looks like its a bug in curl-to-postman library
Personally for me the problem was a semicolon at the end of the cURL request. Try to remove all the noise you can until you find what sort of character is making postman have that error
curl 'http://catalog.data.gov/api/3/' \
-H 'authority: 1fzqk3npw4.execute-api.us-east-1.amazonaws.com' \
-H 'accept: */*' \
-H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36' \
--compressed ;
Remove the semicolon at the end
I removed all the single quotes to Double quotes and it worked !!!
Not Working CURL
curl --location --request POST 'http://localhost/data/?type=MOBILE&value=77700023656' \
--header 'accept: */*'
WORKING CURL (changed single quotes --> double quotes)
curl --location --request POST "http://localhost/data/?type=MOBILE&value=77700023656" \
--header "accept: */*"
remove character \ in the end of all lines
for who is using vscode, replace all by regex
I got
error: {"error":"Unexpected token ‘"}
with the curl command below.
What gives?
curl -X POST \
-H "X-Parse-Application-Id:SomeID" \
-H "Content-Type: application/json" \
-d ‘{“number”:"3016524500"}’ \
http://somedomain.com/parse/functions/testFunction
Please suggest.
I think the issue is that you're using different characters than the actual ' and " characters. Try replacing ” with " and ‘ with '.
I am currently trying to develop an application to upload files to an Amazon S3 bucket using cURL and c++. After carefully reading the S3 developers guide I have started implementing my application using cURL and forming the Header as described by the Developers guide and after lots of trials and errors to determine the best way to create the S3 signature, I am now facing a 501 error. The received header suggests that the method I'm using is not implemented. I am not sure where I'm wrong but here is the HTTP header that I'm sending to amazon:
PUT /test1.txt HTTP/1.1
Accept: */*
Transfer-Encoding: chunked
Content-Type: text/plain
Content-Length: 29
Host: [BucketName].s3.amazonaws.com
Date: [Date]
Authorization: AWS [Access Key ID]:[Signature]
Expect: 100-continue
I have truncated the Bucket Name, Access Key ID and Signature for security reasons.
I am not sure what I'm doing wrong but I think that the error is generating because of the Accept and Transfer-Encoding Fields (Not Really Sure). So can anyone tell me what I'm doing wrong or why I'm getting a 501.
The game changed significantly since the question was asked, the simple authorization headers no longer apply, yet it is still feasible to perform with a UNIX shell script, as follows.
Ensure 'openssl' and 'curl' are available at the command line. TIP: double check the openSSL argument syntax as these may vary with different versions of the tool; e.g. openssl sha -sha256 ... versus openssl sha256 ...
Beware, a single extra newline or space character, else the use of CRLF in place of the NewLine char alone would defeat the signature. Note too that you may want to use content types possibly with encodings to prevent any data transformation through the communication media. You may then have to adjust the list of signed headers at several places; please refer to AMAZON S3 API docs for the numerous conventions to keep enforced like alphabetical-lowercase ordering of header info used in hash calculations at several (redundant) places.
# BERHAUZ Nov 2019 - curl script for file upload to Amazon S3 Buckets
test -n "$1" || {
echo "usage: $0 <myFileToSend.txt>"
echo "... missing argument file ..."
exit
}
yyyymmdd=`date +%Y%m%d`
isoDate=`date --utc +%Y%m%dT%H%M%SZ`
# EDIT the next 4 variables to match your account
s3Bucket="myBucket.name.here"
bucketLocation="eu-central-1"
s3AccessKey="THISISMYACCESSKEY123"
s3SecretKey="ThisIsMySecretKeyABCD1234efgh5678"
#endpoint="${s3Bucket}.s3-${bucketLocation}.amazonaws.com"
endpoint="s3-${bucketLocation}.amazonaws.com"
fileName="$1"
contentLength=`cat ${fileName} | wc -c`
contentHash=`openssl sha256 -hex ${fileName} | sed 's/.* //'`
canonicalRequest="PUT\n/${s3Bucket}/${fileName}\n\ncontent-length:${contentLength}\nhost:${endpoint}\nx-amz-content-sha256:${contentHash}\nx-amz-date:${isoDate}\n\ncontent-length;host;x-amz-content-sha256;x-amz-date\n${contentHash}"
canonicalRequestHash=`echo -en ${canonicalRequest} | openssl sha256 -hex | sed 's/.* //'`
stringToSign="AWS4-HMAC-SHA256\n${isoDate}\n${yyyymmdd}/${bucketLocation}/s3/aws4_request\n${canonicalRequestHash}"
echo "----------------- canonicalRequest --------------------"
echo -e ${canonicalRequest}
echo "----------------- stringToSign --------------------"
echo -e ${stringToSign}
echo "-------------------------------------------------------"
# calculate the signing key
DateKey=`echo -n "${yyyymmdd}" | openssl sha256 -hex -hmac "AWS4${s3SecretKey}" | sed 's/.* //'`
DateRegionKey=`echo -n "${bucketLocation}" | openssl sha256 -hex -mac HMAC -macopt hexkey:${DateKey} | sed 's/.* //'`
DateRegionServiceKey=`echo -n "s3" | openssl sha256 -hex -mac HMAC -macopt hexkey:${DateRegionKey} | sed 's/.* //'`
SigningKey=`echo -n "aws4_request" | openssl sha256 -hex -mac HMAC -macopt hexkey:${DateRegionServiceKey} | sed 's/.* //'`
# then, once more a HMAC for the signature
signature=`echo -en ${stringToSign} | openssl sha256 -hex -mac HMAC -macopt hexkey:${SigningKey} | sed 's/.* //'`
authoriz="Authorization: AWS4-HMAC-SHA256 Credential=${s3AccessKey}/${yyyymmdd}/${bucketLocation}/s3/aws4_request, SignedHeaders=content-length;host;x-amz-content-sha256;x-amz-date, Signature=${signature}"
curl -v -X PUT -T "${fileName}" \
-H "Host: ${endpoint}" \
-H "Content-Length: ${contentLength}" \
-H "x-amz-date: ${isoDate}" \
-H "x-amz-content-sha256: ${contentHash}" \
-H "${authoriz}" \
http://${endpoint}/${s3Bucket}/${fileName}
I must acknowledge that, for someone a bit involved in cryptography like me, the Amazon signature scheme deserves numerous critics:
there's much redundancy in the information being signed,
the 5 step HMAC cascade is almost inverting semantics between key seed and data where 1 step would suffice with proper usage and same security
the last 12 characters of the secret key are useless here, because the significant key length of a SHA256 HMAC is ... 256 bits, hence 32 bytes, of which the first 4 always start with "AWS4" for just no purpose.
overall AWS S3 API re-invents standards where a S/MIME payload would have done
Apologize for the critics, I was not able to resist. Yet acknowledge: it is working reliably, useful for many companies, and an interesting service with a rich API.
You could execute a bash file. Here is an example upload.sh script which you could just run as: sh upload.sh yourfile
#!/bin/bash
file=$1
bucket=YOUR_BUCKET
resource="/${bucket}/${file}"
contentType="application/x-itunes-ipa"
dateValue=`date -R`
stringToSign="PUT\n\n${contentType}\n${dateValue}\n${resource}"
s3Key=YOUR_KEY_HERE
s3Secret=YOUR_SECRET
echo "SENDING TO S3"
signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -vv -X PUT -T "${file}" \
-H "Host: ${bucket}.s3.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Content-Type: ${contentType}" \
-H "Authorization: AWS ${s3Key}:${signature}" \
https://${bucket}.s3.amazonaws.com/${file}
more on: http://www.jamesransom.net/?p=58
http://www.jamesransom.net/?p=58
Solved: was missing an CURLOPT for the file size in my code and now everything is working perfectly