I’m trying to use datomic cloud and using the
[datomic.client.api :as d]
api for it. This is what I’ve done so far:
Create a client:
(d/client {
:server-type :ion
:region "us-east-1"
:system "humboi-march-2021"
:creds-profile "humboi"
:endpoint "http://entry.humboi-march-2021.us-east-1.datomic.net:8182"
:proxy-port 8182
})
And I evidently have the humboi named-profile in my ~/.aws/credentials:
[humboi]
aws_access_key_id = foobar
aws_secret_access_key = foobarbaz
But when I run this:
(d/create-database
client
{:db-name "humboi-march-2021"})
Amongst a lot of logs, I get this exception:
Execution error (ExceptionInfo) at datomic.client.api.async/ares (async.clj:58).
profile file cannot be null
Why am I getting this exception? It seems that the app can't get the aws credentials. Why is this and how can I fix it?
I've also tried manually exporting the AWS_ACCESS_KEY_ID=xxx AWS_SECRET_ACCESS_KEY=yyy environment variables, but still get the same error.
Related
i am getting this while doing amplify init , so main agenda is to develop authentication through aws-cognito , which is using aws-amplify,
? Do you want to use an AWS profile? Yes
? Please choose the profile you want to use default
init failed
Error: read ECONNRESET
at TLSWrap.onStreamRead (internal/stream_base_commons.js:205:27) {
message: 'read ECONNRESET',
errno: 'ECONNRESET',
code: 'NetworkingError',
syscall: 'read',
region: 'us-east-1',
hostname: 'amplify.us-east-1.amazonaws.com',
retryable: true,
time: 2020-04-16T12:09:59.975Z
You may try the following strategies to eliminate the problem you are facing,
This more of looks like a Network problem as per the logs from your
Terminal, therefore if you have a jittery connection, I would
recommend that you try the same on a stable internet connection.
I will recommend to do an amplify delete in case there is some mis-configuration from the last time you did an amplify init, but the chances of this are very less.
Check your aws environment variables or configuration file maybe the credentials of your aws account are missing. Try doing an aws configure and reset the values of your key,secret, and region.
I hope the above suggestions help you somehow.
I upgraded to boto3 a few months ago. These operations always used to work properly. To my knowledge nothing changed, but recently this error has been occurring when I try to reach the aws servers.
client = boto3.client(
'mturk',
aws_access_key_id = key,
aws_secret_access_key = secret_key,
endpoint_url= r"https://mturk-requester.us-east-1.amazonaws.com/")
client.get_hit(HITId=hit.id)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL:
"https://mturk-requester.us-east-1.amazonaws.com/"
This now happens when posting hits, checking my balance, etc. All these operations were originally working as intended.
My awscli is configured to
[default]
region=us-east-1
Seems to work fine for me like this:
client = boto3.client('mturk',region_name='us-east-1')
aws --version
aws-cli/1.16.76 Python/2.7.10 Darwin/16.7.0 botocore/1.12.66
I'm trying to programmatically add an APNS_SANDBOX channel to a pinpoint app. I'm able to do this successfully via the pinpoint console, but not with aws cli or a lambda function which is the end goal. Changes to our Test/Prod environments can only be made via the CodePipeline, but for testing purposes I'm trying to achieve this with the aws cli.
I've tried both aws cli (using the root credentials) and a lambda function -- both result in the following error:
An error occurred (BadRequestException) when calling the UpdateApnsSandboxChannel operation: Missing credentials
I have tried setting the Certificate field in the UpdateApnsSandboxChannel json object as the path to the .p12 certificate file as well as using a string value retrieved from the openssl tool.
Today I worked with someone from aws support, and they were not able to figure out the issue after trying to debug for a couple of hours. They said they would send an email to the pinpoint team, but they did not have an ETA on when they might respond.
Thanks
I ended up getting this to work successfully -- This is why it was failing:
I was originally making the cli call with the following request object as this is what is including in the documentation:
aws pinpoint update-apns-sandbox-channel --application-id [physicalID] --cli-input-json file:///path-to-requestObject.json
{
"APNSSandboxChannelRequest": {
"BundleId": "com.bundleId.value",
"Certificate":"P12_FILE_PATH_OR_CERT_AS_STRING",
"DefaultAuthenticationMethod": "CERTIFICATE",
"Enabled": true,
"PrivateKey":"PRIVATEKEY_FILE_PATH_OR_AS_STRING",
"TeamId": "",
"TokenKey": "",
"TokenKeyId": ""
},
"ApplicationId": "Pinpoint_PhysicalId"
}
After playing around with it some more I got it to work by removing BundleId, TeamId, TokenKey, and TokenKeyId. I believe these fields are needed when using a p8 certificate.
{
"APNSSandboxChannelRequest": {
"Certificate":"P12_FILE_PATH_OR_CERT_AS_STRING",
"DefaultAuthenticationMethod": "CERTIFICATE",
"Enabled": true,
"PrivateKey":"PRIVATEKEY_FILE_PATH_OR_AS_STRING"
},
"ApplicationId": "Pinpoint_PhysicalId"
}
I have deployed a datomic transactor to an AWS cloud formation, connected to a DyanmoDB as storage. I'm able to connect to the transactor if I go to my local datomic directory (version 0.9.5.44) and use the repl:
=> (def uri "datomic:ddb://us-east-1/datemo/test-db?aws_access_key_id={xxx}&aws_secret_key={yyy}")
=> (def conn (d/connect uri))
#object[datomic.peer.Connection 0x5e5ddfbc "{:unsent-updates-queue 0, :pending-txes 0,:next-t 1000, :basis-t 63, :index-rev 0, :db-id \"test-db-ab4b8028-e25b-4431-b2a2-c7173b4c9918\"}"]
If, however, I do the same in my project's repl, it does not work, and instead I get the following error:
CompilerException java.lang.NoClassDefFoundError: Could not initialize
class datomic.ddb_cluster__init,
compiling:(form-init1336530578825920965.clj:1:11)
Incidentally, if I try to connect to my usual datomic dev transactor it works just fine. Does anyone know what I might be doing wrong here? The error seems to suggest that the class is not even present...
You need to include both the Datomic Peer library and the AWS SDK as dependencies in your project to connect to a DDB-backed instance.
This section of the documentation: http://docs.datomic.com/storage.html#sec-6-1 contains details of including the AWS SDK.
-Marshall
I'm trying to read a s3 bucket from Spark and up until today Spark always complain that the request return 403
hadoopConf = spark_context._jsc.hadoopConfiguration()
hadoopConf.set("fs.s3a.access.key", "ACCESSKEY")
hadoopConf.set("fs.s3a.secret.key", "SECRETKEY")
hadoopConf.set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
logs = spark_context.textFile("s3a://mybucket/logs/*)
Spark was saying .... Invalid Access key [ACCESSKEY]
However with the same ACCESSKEY and SECRETKEY this was working with aws-cli
aws s3 ls mybucket/logs/
and in python boto3 this was working
resource = boto3.resource("s3", region_name="us-east-1")
resource.Object("mybucket", "logs/text.py") \
.put(Body=open("text.py", "rb"),ContentType="text/x-py")
so my credentials ARE invalid and the problem is definitely something with Spark..
Today I decided to turn on the "DEBUG" log for the entire spark and to my suprise... Spark is NOT using the [SECRETKEY] I have provided but instead... add a random one???
17/03/08 10:40:04 DEBUG request: Sending Request: HEAD https://mybucket.s3.amazonaws.com / Headers: (Authorization: AWS ACCESSKEY:[RANDON-SECRET-KEY], User-Agent: aws-sdk-java/1.7.4 Mac_OS_X/10.11.6 Java_HotSpot(TM)_64-Bit_Server_VM/25.65-b01/1.8.0_65, Date: Wed, 08 Mar 2017 10:40:04 GMT, Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
This is why it still return 403! Spark is not using the key I provide with fs.s3a.secret.key but instead invent a random one??
For the record I'm running this locally on my machine (OSX) with this command
spark-submit --packages com.amazonaws:aws-java-sdk-pom:1.11.98,org.apache.hadoop:hadoop-aws:2.7.3 test.py
Could some one enlighten me on this?
(updated as my original one was downvoted as clearly considered unacceptable)
The AWS auth protocol doesn't send your secret over the wire. It signs the message. That's why what you see isn't what you passed in.
For further information, please reread.
I ran into a similar issue. Requests that were using valid AWS credentials returned a 403 Forbidden, but only on certain machines. Eventually I found out that the system time on those particular machines were 10 minutes behind. Synchronizing the system clock solved the problem.
Hope this helps!
It is very intriguing this random passkey. Maybe AWS SDK is getting the password from OS environment.
In hadoop 2.8, the default AWS provider chain shows the following list of providers:
BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider SharedInstanceProfileCredentialsProvider
Order, of course, matters! the AWSCredentialProviderChain, get the first keys from the first provider that provides that information.
if (credentials.getAWSAccessKeyId() != null &&
credentials.getAWSSecretKey() != null) {
log.debug("Loading credentials from " + provider.toString());
lastUsedProvider = provider;
return credentials;
}
See the code in "GrepCode for AWSCredentialProviderChain".
I face similar problem using profile credentials. SDK was ignoring the credentials inside ~/.aws/credentials (as good practice, I encourage you to not store credentials inside the program in any way).
My solution...
Set the credentials provider to use ProfileCredentialsProvider
sc._jsc.hadoopConfiguration().set("fs.s3a.endpoint", "s3.eu-central-1.amazonaws.com") # yes, I am using central eu server.
sc._jsc.hadoopConfiguration().set('fs.s3a.aws.credentials.provider', 'com.amazonaws.auth.profile.ProfileCredentialsProvider')
Folks, go for the IAM configuration based on Roles ... that will open up S3 access policies that should be added to the EMR default one.