I'm trying to get to grips with the amazonka library. I want to pre-sign a PUT request for S3. Looking at the example here, I put together the code below. It doesn't work when trying to send up a JPEG image from the client side. AWS complains that the signature is not correct.
My client-side code works with the nodejs aws sdk, so I can only guess that I am configuring things incorrectly here.
<Error>
<Code>
SignatureDoesNotMatch
</Code>
<Message>
The request signature we calculated does not match the signature you provided. Check your key and signing method.
</Message>
<CanonicalRequest>
PUT
/my-bucket/test.jpg
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-
Credential=[MY_AWS_ACCESS_KEY]%2F20171010%2Feu-west-
1%2Fs3%2Faws4_request&X-Amz-Date=20171010T084724Z&X-Amz-
Expires=100000&X-Amz-SignedHeaders=expect%3Bhost
expect:
host:s3-eu-west-1.amazonaws.com
expect;host
UNSIGNED-PAYLOAD
</CanonicalRequest>
...
Here's my code:
getS3UrlR :: Handler Value
getS3UrlR = do
y <- getYesod
let settings = appSettings y
env <- newEnv $ FromKeys (AccessKey $ encodeUtf8 $ appAwsAccessKey settings) (SecretKey $ encodeUtf8 $ appAwsSecretKey settings)
ts <- liftIO $ getCurrentTime
let b = BucketName $ appS3Bucket settings
k = ObjectKey "profile-pic-test"
url <- liftIO $ runResourceT . runAWST env . within Ireland $ presignURL ts 100000 (putObject b k (toBody ("" :: ByteString)))
return $ toJSON $ decodeUtf8 url
How can I go about debugging this?
Note - putObject requires a RqBody argument, but I don't have one to provide (as the client is going to supply the request body). So I just supplied an empty string. Could that be a problem?
Edit - I tried signing a GET request to see if that would work. It works fine! But PUT does not. This is the changed line that works fine:
url <- liftIO $ runResourceT . runAWST env . within Ireland $ presignURL ts 100000 (getObject b k)
I found an issue from Feb which led me to a workround. It seems that presignURL is adding an extraneous header to the signed URL. When I rewrite the request to remove the Expect header before signing, the resulting URL works with a simple PUT.
My workaround is included in my comment on the github issue.
Related
This AWS page suggests the following invocation:
$ aws iam update-role -–role-name ADFS-Production -–max-session-duration 14400
When I run this (substituting the correct role name) I get
aws: error: the following arguments are required: --role-name
This is pretty confusing, because the argument is right there!
$ aws --version
on my machine gives
aws-cli/1.14.69 Python/3.6.4 Darwin/16.3.0 botocore/1.9.22
I tried stepping through the arg parsing with the debugger, and it looked like maybe the problem was that the list of option_string_actions isn't being populated right:
/usr/local/Cellar/python/3.6.4_3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/argparse.py(2093)_parse_optional()
-> if arg_string in self._option_string_actions:
(Pdb) l
2088 # if it doesn't start with a prefix, it was meant to be positional
2089 if not arg_string[0] in self.prefix_chars:
2090 return None
2091
2092 # if the option string is present in the parser, return the action
2093 -> if arg_string in self._option_string_actions:
2094 action = self._option_string_actions[arg_string]
2095 return action, arg_string, None
2096
2097 # if it's just a single character, it was meant to be positional
2098 if len(arg_string) == 1:
(Pdb) self._option_string_actions
{}
but it's pretty complicated, so it's hard to be sure. Any ideas?
I ran your command and something is not right with your --. Can you please try the following command (I changed nothing but your --:
aws iam update-role --role-name ADFS-Production --max-session-duration 14400
I've been trying to set up CloudFront with signed cookies using elixir. From what I learned the signature must use sha1 encoding. According to AWS in the terminal that would look like this:
cat policy.json | openssl sha1 -sign CloudFront_Key.pem |base64 |tr '+=/' '-_~'
I can't figure out how to do that in elixir. I've been looking into this dependency and tried using :sha1 instead in it's sign method. I did read through here and did see sha, not sha1. Is that how erlang calls it? (I have close to no knowledge about erlang). Tried :sha but I do not get the same key as in the terminal and it does not seem to work. I also noticed that base64 usually adds a few extra chars at the end than the terminal. Not sure what to do.. Resort to try using System.cmd ?
This is the AWS doc I'm following: create & verify signed cookies and create signature for signed cookie
You can use :public_key.sign/3 for this.
For reference,
$ cat pk.pem
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEAknEbD26dg4w2JlaN2pmtz7sEBSEjFcfsgcUuV100L60Lz36s
EDySjiTMbhfhKRa46gBvGSEzb8G1l4iD1O5M5mfzEVg/shSOgwVfyvq5XShJBaz6
PcKJ4hsLh/aUIkvnfYWg5zdcPj5FBYT0I+jLnWvNevsolfGwN4rY5S/PuBCUwtVC
m1L3p5bVHvR33tjI4qenHkuRJUnJh4TMLCOTCGHdmZ+yyWJ2rLcYq7yciI0WlRh5
cCI1eYEehUsbcifOyazKNOAmO/9DzXZlo9S0TSYjxrG626mt+j9yD473t/FSfUeg
6Hn67wfGBWAk0HezJ3QGRl9jBAeYhSLbaX7DHQIDAQABAoIBAHo+Blu8d6oe+fjI
2cM389pq/7EUd0gwSmINamCtQenmZux/jjxDhAc5+piQQHlfKV7Um+j7SQeqSN7E
q1+syO6wqTu6UflipZADhXJYFzIHdeVR/tZdNWJUNyz5DbEPcZ7bVHSORucCbfVs
hawQISA4pB9b1wZL6VCEDAhM//ViRgBA3k6tBu5jNi+I1WXE9nrat4DSDV3uklfo
f5KfHLhbHgrl7R7jnAyWBya6O6Pw0r/3FeWrjo6W0wht5BprX4HSgJvpWtqjhodA
iucWjmWV7Gt++HBRX0NEQozajQ7fdfnCOpSD2g8QzVw/5whRBidYngJWLBNt1YQV
UMi7toECgYEAzw4i+m0PnqiExMOSLsvPcEQu0wScOe3vwXTBOcfOS0Uds0kWzZP2
BHQL7IATtN+OAMdtFi1xyN+auikTsZpy1hezUtamfeo75btFhgSrtRXQki5o3pX8
HgDh+31+Ox1RM/Qhm4DSPQ/YImw+HQD+lK9sw3yNZlmPmXwyCARxkCECgYEAtQ75
M5HcBG+zPbPHc4mrOgtQzaUTEs1onu/zDCuFZwYtyBB0RKSLLmacgh/Qcdu13Lli
fdTqego5asKztQfsB88aqkYtdY51dnm+H2sxQ2ef+1D2gbV+sdrsM7gUuVvxBMwv
itFizQ1rgZn/Vee4zfFXEoURjkKfTKT27Hn7A30CgYEAx4ugOiiRPR67lcXFREQ3
jsKnPcbbqRieT5rt/XmKXxAlJ3vw9f76wh/0veBRHae1exq3DwCNAEI/I9oimK94
rMv6joM/wWnUf/qTbi1iLgrwD3Gar6lsaJ4BLBYtaVs/vwowuWTVOPPkIIig8+LZ
dwH5mAyZWWJG+myu6vsdVwECgYAJqT/g4ZKU5gTxcOteneT2Fu572qgW48EGYhVc
++GFas38k+wwUXtfwXfudZYgzTF6EqZPwpG0a2E+8h62tTKCBCoPFemNEUnxRXPA
p26cgyYFOf+9UhrtkJnz9Imejmpg8ChFRwD3ohSveLEoO1IgIxWbVmBmb+WiKFdI
rQWY3QKBgA6Gw/8KB7LxqR3N9/1xgW5b02WVhG7gvFNlSYZMmdd8H3w8MdXCdk22
kJBfrIfkDS6nbF6w4+8q309NPwOqdkt2QNPq6Il1ugWuxFMiNEGHjjT21PrsgEJS
q6K6c+znrnI7O/wijdvEC2n+q+S9h8yv1bT6QfC1Vx88SC9GCBuq
-----END RSA PRIVATE KEY-----
$ cat policy.json
{"Statement":[{"Resource":"https://mycloudfront.net/a.png","Condition":{"DateLessThan":{"AWS:EpochTime":1512086400}}}]}
First, read the key:
iex(1)> key = File.read!("pk.pem") |> :public_key.pem_decode |> hd |> :public_key.pem_entry_decode
...
Then sign the data using :public_key.sign/3:
iex(2)> File.read!("policy.json") |> :public_key.sign(:sha, key) |> Base.encode64
"QjLmx3LASRb1zt9eW/EMywGMXB1SwX/0JrTnLOFulYjcRJ1dpacUZBB/AYI1zwaXPEQTgQ8crNDFgje6fqbLKoNwgcpE9mOK/RdDKi963ztJnD6EmtM60YbROSpjQ/LDupEYgipPNZbjCnRCJcqDX43BadbVR75G3B5mFmAwtRSPdslJ5irVnt9PjoDMdi9DYe1wGhgQkoym1tiKEyaTrH5lyrw+KPdAi1tpzuZ60ZEcQFJJbKqYYdA0SslbUFL71mdLLkQ9xz95JPNpsSY3ZJyJsKpRGFJuaL1aMsdNLxlLD91PpNW15FitBpBnAwuiiEfPrwU14zIxsfFszaM6KA=="
The output is identical to openssl:
$ cat policy.json | openssl sha1 -sign pk.pem | base64
QjLmx3LASRb1zt9eW/EMywGMXB1SwX/0JrTnLOFulYjcRJ1dpacUZBB/AYI1zwaXPEQTgQ8crNDFgje6fqbLKoNwgcpE9mOK/RdDKi963ztJnD6EmtM60YbROSpjQ/LDupEYgipPNZbjCnRCJcqDX43BadbVR75G3B5mFmAwtRSPdslJ5irVnt9PjoDMdi9DYe1wGhgQkoym1tiKEyaTrH5lyrw+KPdAi1tpzuZ60ZEcQFJJbKqYYdA0SslbUFL71mdLLkQ9xz95JPNpsSY3ZJyJsKpRGFJuaL1aMsdNLxlLD91PpNW15FitBpBnAwuiiEfPrwU14zIxsfFszaM6KA==
I'm trying to follow a tutorial for using spark from RStudio on DSX, but I'm running into the following error:
> library(sparklyr)
> sc <- spark_connect(master = "CS-DSX")
Error in spark_version_from_home(spark_home, default = spark_version) :
Failed to detect version from SPARK_HOME or SPARK_HOME_VERSION. Try passing the spark version explicitly.
I took the above code snippet from the connect to spark dialog in RStudio:
So I took a look at SPARK_HOME:
> Sys.getenv("SPARK_HOME")
[1] "/opt/spark"
Ok, Lets check that dir exists:
> dir("/opt")
[1] "ibm"
I'm guessing this is the cause of the problem?
NOTE: there are a few similar questions on stackoverflow, but none of them are about IBM's Data Science Experience (DSX).
Update 1:
I tried the following:
> sc <- spark_connect(config = "CS-DSX")
Error in config$spark.master : $ operator is invalid for atomic vectors
Update 2:
An extract from my config.yml. Note that I have many more spark services in my, I've just pasted the first one:
default:
method: "shell"
CS-DSX:
method: "bluemix"
spark.master: "spark.bluemix.net"
spark.instance.id: "7a4089bf-3594-4fdf-8dd1-7e9fd7607be5"
tenant.id: "sdd1-7e9fd7607be53e-39ca506ba762"
tenant.secret: "xxxxxx"
hsui.url: "https://cdsx.ng.bluemix.net"
Note that my config.yml was generated for me.
Update 3:
My .Rprofile looks like this:
# load sparklyr library
library(sparklyr)
# setup SPARK_HOME
if (nchar(Sys.getenv("SPARK_HOME")) < 1) {
Sys.setenv(SPARK_HOME = "/opt/spark")
}
# setup SparkaaS instances
options(rstudio.spark.connections = c("CS-DSX","newspark","cleantest","4jan2017","Apache Spark-4l","Apache Spark-3a","ML SPAAS","Apache Spark-y9","Apache Spark-a8"))
Note that my .Rprofile was generated for me.
Update 4:
I uninstalled sparklyr and restarted the session twice. Next I tried to run:
library(sparklyr)
library(dplyr)
sc <- spark_connect(config = "CS-DSX")
However, the above command hung. I stopped the command and checked the version of sparklyr which seems to be ok:
> ip <- installed.packages()
> ip[ rownames(ip) == "sparklyr", c(0,1,3) ]
Package Version
"sparklyr" "0.4.36"
You cannot use master parameter to connect to bluemix spark service if that is the intent since your kernels are defined in config.yml file, you should be using config parameter instead to connect.
config.yml is loaded up with your available kernel information(spark instances).
Apache Spark-ic:
method: "bluemix"
spark.master: "spark.bluemix.net"
spark.instance.id: "41a2e5e9xxxxxx47ef-97b4-b98406426c07"
tenant.id: "s7b4-b9xxxxxxxx7e8-2c631c8ff999"
tenant.secret: "XXXXXXXXXX"
hsui.url: "https://cdsx.ng.bluemix.net"
Please use config
sc <- spark_connect(config = "Apache Spark-ic")
as suggested in tutorial:-
http://datascience.ibm.com/blog/access-ibm-analytics-for-apache-spark-from-rstudio/
FYI,
By Default, you are connected to , i am working on finding how to change version with config parameter.
> version <- invoke(spark_context(sc), "version")
print(version)
[1] "2.0.2"
Thanks,
Charles.
I had the same issue and fix it as follows:
go to C:\Users\USER_NAME\AppData\Local/spark/ and delete everything you'll find in the directory
Then, in the R console run:
if (!require(shiny)) install.packages("shiny");
library(shiny)
if (!require(sparklyr)) install.packages("sparklyr");
library(sparklyr)
spark_install()
I am actually using wamp 2.5 with PHP 5.5.12 and when I try to create a phar file it returns me the following message :
Uncaught exception 'UnexpectedValueException' with message 'creating archive "..." disabled by the php.ini setting phar.readonly'
even if I turn to off the phar.readonly option in php.ini.
So how can I enable the creation of phar files ?
I had this same problem and pieced together from info on this thread, here's what I did in over-simplified explanation:
in my PHP code that's generating this error, I added echo phpinfo(); (which displays a large table with all sort of PHP info) and in the first few rows verify the path of the php.ini file to make sure you're editing the correct php.ini.
locate on the phpinfo() table where it says phar.readonly and note that it is On.
open the php.ini file from step 1 and search for phar.readonly. Mine is on line 995 and reads ;phar.readonly = On
Change this line to phar.readonly = Off. Be sure that there is no semi-colon at the beginning of the line.
Restart your server
Confirm that you're phar project is now working as expected, and/or search on the phpinfo()table again to see that the phar.readonly setting has changed.
phar.readonly can only be disabled in php.ini due to security reasons.
If you want to check that it's is really not done using other method than php.ini then in terminal type this:-
$ php -r "ini_set('phar.readonly',0);print(ini_get('phar.readonly'));"
If it will give you 1 means phar.readonly is On.
More on phar.configuration
Need to disable in php.ini file
Type which php
Gives a different output depending on machine e.g.
/c/Apps/php/php-7.2.11/php
Then open the path given not the php file.
E.g. /c/Apps/php/php-7.2.11
Edit the php.ini file
could do
vi C:\Apps\php\php-7.2.11\php.ini
code C:\Apps\php\php-7.2.11\php.ini
[Phar]
; http://php.net/phar.readonly
phar.readonly = Off
; http://php.net/phar.require-hash
phar.require_hash = Off
Save
Using php-cli and a hashbang, we can set it on the fly without messing with the ini file.
testphar.php
#!/usr/bin/php -d phar.readonly=0
<?php
print(ini_get('phar.readonly')); // Must return 0
// make sure it doesn't exist
#unlink('brandnewphar.phar');
try {
$p = new Phar(dirname(__FILE__) . '/brandnewphar.phar', 0, 'brandnewphar.phar');
} catch (Exception $e) {
echo 'Could not create phar:', $e;
}
echo 'The new phar has ' . $p->count() . " entries\n";
$p->startBuffering();
$p['file.txt'] = 'hi';
$p['file2.txt'] = 'there';
$p['file2.txt']->compress(Phar::GZ);
$p['file3.txt'] = 'babyface';
$p['file3.txt']->setMetadata(42);
$p->setStub('<?php
function __autoload($class)
{
include "phar://myphar.phar/" . str_replace("_", "/", $class) . ".php";
}
Phar::mapPhar("myphar.phar");
include "phar://myphar.phar/startup.php";
__HALT_COMPILER();');
$p->stopBuffering();
// Test
$m = file_get_contents("phar://brandnewphar.phar/file2.txt");
$m = explode("\n",$m);
var_dump($m);
/* Output:
* there
**/
✓ Must be set executable:
chmod +x testphar.php
✓ Must be called like this:
./testphar.php
// OUTPUT there
⚠️ Must not be called like this:
php testphar.php
// Exception, phar is read only...
⚠️ Won't work called from a CGI web server
php -S localhost:8785 testphar.php
// Exception, phar is read only...
For anyone who has changed the php.ini file, but just doesn't see any changes. Try to use the CLI version of the file. For me, it was in /etc/php/7.4/cli/php.ini
Quick Solution!
Check:
cat /etc/php/7.4/apache2/php.ini | grep phar.readonly
Fix:
sed -i 's/;phar.readonly = On/;phar.readonly = Off/g' /etc/php/7.4/apache2/php.ini
I have a list with a lot of page urls. I want to retrieve the unique websites.
"http://www.gadgetgiants.com/products/mica-8-inch-touchscreen-android-2-3-tablet-wifi-1-2ghz-cpu-flash10-3"
"http://www.malma.mx/products/pan-digital"
"http://www.gadgetgiants.com/products/snowpad-7-capacitive-multi-touch-screen-android-2-3-tabletwifi-samsung-cortex-a8-1-2ghz-cpu-camera-1080p-external-3g"
"http://www.spiritualityandwellness.com/products/internalized-motivation"
"http://www.spiritualityandwellness.com/products/evergreen-motivation"
Will result to:
www.gadgetgiants.com
www.malma.mx
www.spiritualityandwellness.com
egrep -o "www\.[a-zA-Z0-9.-]*\.[a-zA-Z]{2,4}" YOUR_FILE_NAME | sort -u
got the regex from here
(Edit) Example Usage and Output
$ cat ur.txt
"http://www.gadgetgiants.com/products/mica-8-inch-touchscreen-android-2-3"
"http://www.malma.mx/products/pan-digital"
"http://www.gadgetgiants.com/products/snowpad-7-capacitive-multi-touch"
"http://www.spiritualityandwellness.com/products/internalized-motivation"
"http://www.spiritualityandwellness.com/products/evergreen-motivation"
"http://www.swellness.com.au/products/evergreen-motivation"
$ egrep -o "www\.[a-zA-Z0-9.-]*\.[a-zA-Z]{2,4}" ur.txt | sort -u
www.gadgetgiants.com
www.malma.mx
www.spiritualityandwellness.com
www.swellness.com.au
Idea w/o regex:
Retrieve host from each address:
Uri uri = new Uri (yourLink);
string host = uri.Host;
Now you can just put all these hosts into HashSet or something.