I am using below cURL command to get DataPower files from applaince to a remote Solaris server.
/usr/local/bin/curl -s --insecure --data-binary #getFile.xml -u username:password https://ip:port/service/mgmt/current
Content of getFile.xml is as below.
<?xml version="1.0"?>
<env:Envelope xmlns:env="http://schemas.xmlsoap.org/soap/envelope/">
<env:Body>
<dp:request xmlns:dp="http://www.datapower.com/schemas/management">
<dp:get-file name="config:///unicenter.cfg"/>
</dp:request>
</env:Body>
</env:Envelope>
When I am running the cURL metioned above on Solaris, I am getting long base64 encoded string. But I wish to get the complete file copied to Solaris.
The long Base64-encoded string is your file. You need to do a little work to extract it.
This curl command is using the DataPower XML Management interface, and they call it that because all requests and responses are XML-formatted. You may not have seen it as the long string flew by, but it was wrapped in XML. Here's a sample response with a small payload:
<?xml version="1.0" encoding="UTF-8"?>
<env:Envelope xmlns:env="http://schemas.xmlsoap.org/soap/envelope/">
<env:Body>
<dp:response xmlns:dp="http://www.datapower.com/schemas/management">
<dp:timestamp>2014-10-23T17:12:39-04:00</dp:timestamp>
<dp:file name="local:///testfile.txt">VGhpcyBpcyBub3QgYW4gYWN0dWFsIGVtZXJnZW5jeS4K</dp:file>
</dp:response>
</env:Body>
</env:Envelope>
So, you have two jobs to do. First, get the Base64 string out of its XML wrapper, and second, decode it. There are a million ways to do this -- I'll give you one of them. Get a copy of XmlStarlet to do the extraction, and OpenSSL to do the Base64 decoding.
Then, pipe the curl output like so:
/usr/local/bin/curl -s --insecure --data-binary #getFile.xml -u username:password https://ip:port/service/mgmt/current \
| (xmlstarlet sel -T -t -v "//*[local-name()='file']" && echo) \
| fold -w 64 \
| openssl enc -d -base64 >this-is-the-real-file
Two quick notes -- the "&& echo" is to add a trailing newline, and the "fold" is to split the Base64 string into lines. A less finicky Base64 decoder wouldn't need these. I just picked "openssl" because most people already have it.
Related
There are three variants of format selection:
curl -uadmin:admin "http://localhost:8080/alfresco/service/hellouser.json"
curl -uadmin:admin "http://localhost:8080/alfresco/service/hellouser?format=json"
curl -uadmin:admin -H "Accept: text/html" "http://localhost:8080/alfresco/service/hellouser"
But this is unclear from the DOC what format would be selected for next query:
curl -uadmin:admin -H "Accept: text/html" "http://localhost:8080/alfresco/service/hellouser.xml?format=json"
I expect json here.
May someone provide links to relevant specifications or documentation which describes priority how {format} negotiated? like this is described for Rails:
Rails picks up the expected format from the query parameter format, or if not there from the URL path suffix, or it not there from the Accept header
UPD
The controller can handle all supplied formats: json, xml, html
UPD
Another corner case:
curl -uadmin:admin "http://localhost:8080/alfresco/service/hellouser.pl?format=json"
curl -uadmin:admin "http://localhost:8080/alfresco/service/hellouser.pl?format=xml"
I'd believe you wouldn't have a 200 response, only an error with content negotiation.
The code shows that:
?format=json(format_query_param) will be discarded by the .xml (format_suffix)
filter available renderers leaving only the XMLRenderer left
then it will loop on the accept header but none will match text/html
finally this will be down to the exception
I am attempting to extract the IPv4 Addresses and URLs for each product in the Office 365 portfolio from http://go.microsoft.com/fwlink/?LinkId=533185. I would like to create an input file for updating Firewall and Proxy server rules.
Snippet from XML File
<?xml version="1.0" encoding="utf-8"?>
<products updated="8/31/2016">
<product name="WAC">
<addresslist type="IPv6">
<address>2a01:111:f406:8800::/64</address>
<address>2a01:111:f406:400::/64</address>
<address>2a01:111:f406:1c01::/64</address>
<address>2a01:111:f406:9400::/64</address>
<address>2a01:111:f406:2402::/64</address>
<address>2a01:111:f406:a804::/64</address>
<address>2a01:111:f406:b401::/64</address>
<address>2620:1ec:c11::204</address>
<address>2a01:111:202c::204</address>
<address>2620:1ec:c11::205</address>
<address>2a01:111:202c::205</address>
</addresslist>
<addresslist type="IPv4">
<address>13.69.187.20/32</address>
<address>13.70.184.242/32</address>
<address>13.71.155.176/32</address>
<address>13.75.153.216/32</address>
<address>13.76.140.48/32</address>
<address>13.78.114.39/32</address>
<address>13.85.84.102/32</address>
<address>13.88.248.161/32</address>
<address>13.88.254.212/32</address>
<address>13.94.209.165/32</address>
<address>23.103.183.15/32</address>
<address>40.68.166.51/32</address>
<address>40.74.130.243/32</address>
<address>40.74.138.42/32</address>
<address>40.76.54.124/32</address>
<address>40.86.230.88/32</address>
<address>40.114.192.209/32</address>
<address>40.117.226.146/32</address>
<address>40.126.236.216/32</address>
<address>40.127.79.139/32</address>
<address>52.169.109.48/32</address>
<address>52.172.13.171/32</address>
<address>52.172.153.104/32</address>
<address>52.175.25.142/32</address>
<address>52.232.128.169/32</address>
<address>104.40.225.204/32</address>
<address>104.41.62.54/32</address>
<address>104.211.103.207/32</address>
<address>104.211.229.230/32</address>
<address>104.214.38.136/32</address>
<address>104.215.194.17/32</address>
<address>134.170.27.86/32</address>
<address>134.170.48.20/32</address>
<address>134.170.48.22/32</address>
<address>134.170.65.86/32</address>
<address>134.170.170.86/32</address>
<address>137.116.172.39/32</address>
<address>137.135.65.72/32</address>
<address>191.235.87.181/32</address>
<address>191.237.40.220/32</address>
</addresslist>
<addresslist type="URL">
<address>*.officeapps.live.com</address>
<address>*.cdn.office.net</address>
</addresslist>
</product>
<product name="Sway">
I have figured out out to list the products which can be used as to cycle through the products to get the end result but I can seem to get past filtering on a particular product and IPv4 address.
./xmlstarlet sel -t -m '/products/product' -v #name -n Downloads/O365IPAddresses.xml
WAC
Sway
Planner
Yammer
OneNote
OfficeiPad
OfficeMobile
ProPlus
RCA
LYO
SPO
Office365Video
identity
EXO
CRLs
o365
EOP
I have tried many variations of the following and no output.
./xmlstarlet sel -t -m '//root/products/product[name="WAC"]/addresslist[type="IPv4"]' Downloads/O365IPAddresses.xml
Ultimately I would like to get to a text file that has something like the following.
WAC,IPv4,13.69.187.20/32
WAC,IPv4,13.70.184.242/32
.....
.....
WAC,URL,*.officeapps.live.com
I can then use this csv file to update the collection of Firewalls and Proxy servers in my company.
Many thanks in advance for the help.
I want to log varnish backend request which matches specified IP (for example 127.0.0.1).
So i have
"varnishlog -b -I BereqHeader:X-Forwarded-For: 127.0.0.1'"
Which actualy logs only the "BereqHeader:X-Forwarded-For:" part. I want to log full request, not only IP part.
That was first question, the second one is: how to disable loging empty request? I mean, if i have regex filter then i have a lot of request looking like this "* << BeReq >> 307454" and i obviously dont want to see them.
I have a solution. Log the data by
varnishlog -b -I BereqHeader:'X-Forwarded-For: 123.215.32.76' -i [other tags to log] > file.varnishlog
and then grep it by
cat file.varnishlog | grep -Pzo '* {3}<< BeReq {4}>>.\n- BereqHeader.+\n(-.\n)*'
which'll give us expected results.
I have a KML file which is a list of places around the world with coordinates and some other attributes. It looks like this for one place:
<Placemark>
<name>Albania - Durrës</name>
<open>0</open>
<visibility>1</visibility>
<description>(Spot ID: 275801) show <![CDATA[forecast]]></description>
<styleUrl>#wgStyle001</styleUrl><Point>
<coordinates>19.489747,41.277806,0</coordinates>
</Point>
<LookAt><range>200000</range><longitude>19.489747</longitude><latitude>41.277806</latitude></LookAt>
</Placemark>
I would like to remove everything except the name of the place. So in this case that would mean I would like to remove everything except
<name>Albania - Durrës</name>
The problem is, this KML file includes more than 1000 of these places. Doing this manually obviously isn't an option, so then how can I remove all tags except for the name tags for all of the items in the list? Can I use some kind of program for that?
Use a specialized command line tool that understands XML documents.
One such tool is xmlstarlet, which is available here for Linux, Windows and Solaris.
To address your particular problem, I used the xmlstarlet executable xml.exe like this (on Windows):
xml.exe sel -N ns=http://www.opengis.net/kml/2.2 -t -v /ns:kml/ns:Document/ns:Placemark/ns:name places.kml
This produces this output:
Albania - Durrës
Second Name
Third Name
...
Final Name
If you can guarantee that <name> occurs only as a child of <Placemark>, then this abbreviated version will produce the same result:
xml.exe sel -N ns=http://www.opengis.net/kml/2.2 -t -v //ns:name places.kml
(This is because this shorter version finds all <name> elements no matter where they occur in the document.)
If you really want an XML document, you'll need to do a little post-processing. Here's an example of a complete XML document:
<?xml version='1.0' encoding='utf-8'?>
<items>
<item>Albania - Durrës</item>
<item>Second Name</item>
<item>Third Name</item>
<!-- ... -->
<item>Final Name</item>
</items>
This first line is the XML declaration. It declares the Unicode encoding utf-8. You'll need to include this line so that XML processors recognize that your document includes Unicode characters. (As in Durrës.)
More: Here's an enhanced 'xmlstarlet' command that will produce the XML document above:
xml.exe sel -N ns=http://www.opengis.net/kml/2.2 -T -t -o "<?xml version='1.0' encoding='utf-8'?>" -n -t -v "'<items>'" -n -t -m //ns:Placemark -v "concat('<item>',ns:name,'</item>')" -n -t -o "</items>" -n places.kml
If you are on linux or similar:
grep "<name>" your_file.kml > file_with_only_name_tags
On windows, see What are good grep tools for Windows?
I am currently trying to develop an application to upload files to an Amazon S3 bucket using cURL and c++. After carefully reading the S3 developers guide I have started implementing my application using cURL and forming the Header as described by the Developers guide and after lots of trials and errors to determine the best way to create the S3 signature, I am now facing a 501 error. The received header suggests that the method I'm using is not implemented. I am not sure where I'm wrong but here is the HTTP header that I'm sending to amazon:
PUT /test1.txt HTTP/1.1
Accept: */*
Transfer-Encoding: chunked
Content-Type: text/plain
Content-Length: 29
Host: [BucketName].s3.amazonaws.com
Date: [Date]
Authorization: AWS [Access Key ID]:[Signature]
Expect: 100-continue
I have truncated the Bucket Name, Access Key ID and Signature for security reasons.
I am not sure what I'm doing wrong but I think that the error is generating because of the Accept and Transfer-Encoding Fields (Not Really Sure). So can anyone tell me what I'm doing wrong or why I'm getting a 501.
The game changed significantly since the question was asked, the simple authorization headers no longer apply, yet it is still feasible to perform with a UNIX shell script, as follows.
Ensure 'openssl' and 'curl' are available at the command line. TIP: double check the openSSL argument syntax as these may vary with different versions of the tool; e.g. openssl sha -sha256 ... versus openssl sha256 ...
Beware, a single extra newline or space character, else the use of CRLF in place of the NewLine char alone would defeat the signature. Note too that you may want to use content types possibly with encodings to prevent any data transformation through the communication media. You may then have to adjust the list of signed headers at several places; please refer to AMAZON S3 API docs for the numerous conventions to keep enforced like alphabetical-lowercase ordering of header info used in hash calculations at several (redundant) places.
# BERHAUZ Nov 2019 - curl script for file upload to Amazon S3 Buckets
test -n "$1" || {
echo "usage: $0 <myFileToSend.txt>"
echo "... missing argument file ..."
exit
}
yyyymmdd=`date +%Y%m%d`
isoDate=`date --utc +%Y%m%dT%H%M%SZ`
# EDIT the next 4 variables to match your account
s3Bucket="myBucket.name.here"
bucketLocation="eu-central-1"
s3AccessKey="THISISMYACCESSKEY123"
s3SecretKey="ThisIsMySecretKeyABCD1234efgh5678"
#endpoint="${s3Bucket}.s3-${bucketLocation}.amazonaws.com"
endpoint="s3-${bucketLocation}.amazonaws.com"
fileName="$1"
contentLength=`cat ${fileName} | wc -c`
contentHash=`openssl sha256 -hex ${fileName} | sed 's/.* //'`
canonicalRequest="PUT\n/${s3Bucket}/${fileName}\n\ncontent-length:${contentLength}\nhost:${endpoint}\nx-amz-content-sha256:${contentHash}\nx-amz-date:${isoDate}\n\ncontent-length;host;x-amz-content-sha256;x-amz-date\n${contentHash}"
canonicalRequestHash=`echo -en ${canonicalRequest} | openssl sha256 -hex | sed 's/.* //'`
stringToSign="AWS4-HMAC-SHA256\n${isoDate}\n${yyyymmdd}/${bucketLocation}/s3/aws4_request\n${canonicalRequestHash}"
echo "----------------- canonicalRequest --------------------"
echo -e ${canonicalRequest}
echo "----------------- stringToSign --------------------"
echo -e ${stringToSign}
echo "-------------------------------------------------------"
# calculate the signing key
DateKey=`echo -n "${yyyymmdd}" | openssl sha256 -hex -hmac "AWS4${s3SecretKey}" | sed 's/.* //'`
DateRegionKey=`echo -n "${bucketLocation}" | openssl sha256 -hex -mac HMAC -macopt hexkey:${DateKey} | sed 's/.* //'`
DateRegionServiceKey=`echo -n "s3" | openssl sha256 -hex -mac HMAC -macopt hexkey:${DateRegionKey} | sed 's/.* //'`
SigningKey=`echo -n "aws4_request" | openssl sha256 -hex -mac HMAC -macopt hexkey:${DateRegionServiceKey} | sed 's/.* //'`
# then, once more a HMAC for the signature
signature=`echo -en ${stringToSign} | openssl sha256 -hex -mac HMAC -macopt hexkey:${SigningKey} | sed 's/.* //'`
authoriz="Authorization: AWS4-HMAC-SHA256 Credential=${s3AccessKey}/${yyyymmdd}/${bucketLocation}/s3/aws4_request, SignedHeaders=content-length;host;x-amz-content-sha256;x-amz-date, Signature=${signature}"
curl -v -X PUT -T "${fileName}" \
-H "Host: ${endpoint}" \
-H "Content-Length: ${contentLength}" \
-H "x-amz-date: ${isoDate}" \
-H "x-amz-content-sha256: ${contentHash}" \
-H "${authoriz}" \
http://${endpoint}/${s3Bucket}/${fileName}
I must acknowledge that, for someone a bit involved in cryptography like me, the Amazon signature scheme deserves numerous critics:
there's much redundancy in the information being signed,
the 5 step HMAC cascade is almost inverting semantics between key seed and data where 1 step would suffice with proper usage and same security
the last 12 characters of the secret key are useless here, because the significant key length of a SHA256 HMAC is ... 256 bits, hence 32 bytes, of which the first 4 always start with "AWS4" for just no purpose.
overall AWS S3 API re-invents standards where a S/MIME payload would have done
Apologize for the critics, I was not able to resist. Yet acknowledge: it is working reliably, useful for many companies, and an interesting service with a rich API.
You could execute a bash file. Here is an example upload.sh script which you could just run as: sh upload.sh yourfile
#!/bin/bash
file=$1
bucket=YOUR_BUCKET
resource="/${bucket}/${file}"
contentType="application/x-itunes-ipa"
dateValue=`date -R`
stringToSign="PUT\n\n${contentType}\n${dateValue}\n${resource}"
s3Key=YOUR_KEY_HERE
s3Secret=YOUR_SECRET
echo "SENDING TO S3"
signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -vv -X PUT -T "${file}" \
-H "Host: ${bucket}.s3.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Content-Type: ${contentType}" \
-H "Authorization: AWS ${s3Key}:${signature}" \
https://${bucket}.s3.amazonaws.com/${file}
more on: http://www.jamesransom.net/?p=58
http://www.jamesransom.net/?p=58
Solved: was missing an CURLOPT for the file size in my code and now everything is working perfectly